IaaS Performance Benchmarks Part 6: Rackspace Cloud Servers

This is the sixth part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained my methodology for testing instance types across major IaaS providers. I've run benchmarks for Amazon Web Services, Google Compute Engine, and Microsoft Azure Virtual Machines. In this part, I look at Rackspace Cloud Servers, which have been around for a long time and been though a few revisions.

Rackspace focuses on support and its managed hosting options, and I believe it is the only major IaaS provider that offers a built-in plan for on-demand servers that includes managed hosting support. However, for the purposes of these benchmarks, I am only considering the features and prices of the compute resources without support services.

As of today, there are three U.S. regions available for Rackspace through RightScale (the cloud management service I’m using to run most of my benchmarks, as it gives me image parity): Chicago, Dallas, and Virginia. In each of those regions, I kicked off one of each “Performance” and “Standard” instance type that Rackspace offers, for a total of 48 VMs. Rackspace announced its Performance Cloud Servers in November, when it also renamed its "Next-Gen" instance types as Standard.

As with the previously-tested cloud providers, I ran on RightScale’s CentOS 6.4 images. Rackspace offers a somewhat-confusing discount program based on how long you’re willing to commit and whether you’ll pay up front. According to Rackspace, it is possible to get up to a 37% discount on its pricing with a three-year commitment and paying up front, so I have used a 37% discount as the “discounted pricing” amount.

Rackspace's Performance servers in theory should be competitive with Amazon’s newer instance types (e.g., M3, C3, I2) and Google Compute Engine. For example, the Performance instance types apparently run on Intel Xeon E5-2670 2.60GHz processors; AWS lists its M3 family as running on the E5-2670, the C3 family on the E5-2680, the I2 family on the E5-2670v2, and GCE says it uses 2.6GHz+ Xeon Sandy Bridge processors. However, on CentOS 6.4, I found that Rackspace's Performance instance types really didn’t perform significantly better than its Standard instance types.

Why? This description in a Rackspace developer blog post offers some clues: “So, terminology aside, today, when you boot any Linux image on Rackspace Cloud, that image uses standard paravirtualization (PV), for Windows and FreeBSD we use HVM. In the case of the new Performance Cloud Servers, we still default to PV images, and are currently testing PV on HVM images.”

But what exactly does this mean? Unfortunately, Rackspace’s own support portal has no useful information on PV vs. hardware-assisted virtualization (HVM), even though here, it means the difference between UnixBench scores that are competitive with AWS and GCE and scores that end up well below Azure. The short story is that you should see substantially higher UnixBench scores using HVM virtualization, at least on the processors being used. By not having HVM virtualization available yet for its Performance instance types for Linux, Rackspace is not offering a particularly competitive product today.

[Security concerns are driving organizations to consider private and hybrid clouds, according to a Rackspace survey last year. Read the report in "Private, Hybrid Cloud Interest Spurred By Security and Control."]

Based on conversations with several individuals at Rackspace, I believe (a) that Rackspace will have HVM support for Linux for its Performance instance types at some point in the first half of 2014 (note that this is my guess, not anything they said), and (b) that it will probably perform similarly to the benchmarks run on the beta version that Rackspace has made available, which gives numbers more than 3X better than what I got.

I will discuss my results by including both the “what if Rackspace supported HVM for Linux today” and the “what I actually got” perspectives. I should note that Rackspace has graciously given me access to a pre-release version of a CentOS image that can be launched with HVM on the Rackspace Cloud, which I plan to test before the conclusion of this IaaS benchmark series.

Rackspace Benchmark Results

Here are the main takeaways that I have from testing across Rackspace:

•For the first time in testing IaaS providers, I saw a regional difference: Rackspace Chicago delivered the three top single UnixBench scores (from the three most expensive instance types), and five of the top ten, but also five of the six worst scores (from the least expensive instance types). This requires further investigation to draw any significant conclusion, since I ran all these in a single day.

•The best price-per-performance for me was with the 512MB Standard, with the 1GB Performance a distant second. Of the more beefy instance types, the 8GB Performance stands out as having much better price-per-performance than the others (likely because it has eight vCPUs; you have to go to the 60GB Performance to get more than that).

•Performance in the single-threaded UnixBench results does appear to vary, at least a little bit, based on price, and the Performance types do better than the Standard types, even with everything running on PV virtualization. In particular, the 8GB, 15GB, 30GB, 60GB, 90GB, and 120GB Performance types all scored at least 544 (average) on the single-threaded UnixBench scores, whereas the rest of the instance types all scored 493 or less (average). However, under HVM virtualization, it appears that results over 1500 should be expected.

View Larger

View Larger

NEXT: Comparing Rackspace To Other IaaS Providers

So, how does Rackspace compare to AWS, GCE, and Azure, at least as far as these fairly simplistic benchmarks go?

•Rackspace’s cheapest instance type, the 512MB Standard, beats out Amazon pretty handily for price-per-performance, but is more expensive both Microsoft Azure’s Extra Small and Google Compute Engine’s f1-micro, from my experience. That said, Rackspace appears to guarantee the performance of the 512MB more than either GCE or Azure, so if you’re mostly interested in the low end, Rackspace may win out over time for you here.

•With HVM virtualization, Rackspace’s 1GB Performance will likely turn in better UnixBench scores than anything that any other major IaaS provider sells for the same price or less.

•With HVM virtualization, Rackspace’s Performance family will likely turn in UnixBench scores that are competitive with AWS and GCE at roughly the same price points, although there are also notable differences between the providers. At roughly less than $1,500/month/server on-demand, Rackspace offers significantly less RAM than either AWS or GCE (Rackspace only offers # of cores = GB of RAM until you hit 15GB RAM); however, Rackspace touts its SSDs and network bandwidth as superior.

•On the issue of I/O, it's interesting that AWS and GCE have focused on IOPS available to specific instances, whereas Rackspace seems to focus on the adjacent metrics of network bandwidth (both external and internal). The most-touted IOPS number that I found on Rackspace’s site is “Disk I/O up to ~35,000 4K random read IOPS and ~35,000 4K random write IOPS,” which comes with the caveat that this is “shared across the host” -- in other words, subject to noisy neighbors. This really requires further investigation with I/O benchmarking, but I think Rackspace would be in a better situation if it could offer guaranteed IOPS for specific instance/storage connections.

My overall impression of Rackspace is of a successful managed hosting company that has been trying to react to AWS for several years now, without significant success. Rackspace may have the best managed hosting and private/public/hybrid cloud consulting/hosting services, but HP, Datapipe, Dimension Data, SoftLayer, and CenturyLink -- among others -- look like solid competitors in this space.

Moreover, while those kind of services might have been what organizations wanted five years ago, I think AWS has made the industry re-examine exactly how we should think about infrastructure. In particular, services that provide developers with exceptional APIs and self-signup seem to be doing better than services that have focused on consulting and outstanding live support.

I think Rackspace is doing what it needs to do right now: get server options online that provide performance (both CPU and I/O) competitive with the top IaaS vendors. For the company's sake, I hope it can speed up its development/release cycles and that there remains a significant market that wants to pay Rackspace to manage above the infrastructure layer.