This is the sixth part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained my methodology for testing instance types across major IaaS providers. I've run benchmarks for Amazon Web Services, Google Compute Engine, and Microsoft Azure Virtual Machines. In this part, I look at Rackspace Cloud Servers, which have been around for a long time and been though a few revisions.
Rackspace focuses on support and its managed hosting options, and I believe it is the only major IaaS provider that offers a built-in plan for on-demand servers that includes managed hosting support. However, for the purposes of these benchmarks, I am only considering the features and prices of the compute resources without support services.
As of today, there are three U.S. regions available for Rackspace through RightScale (the cloud management service I’m using to run most of my benchmarks, as it gives me image parity): Chicago, Dallas, and Virginia. In each of those regions, I kicked off one of each “Performance” and “Standard” instance type that Rackspace offers, for a total of 48 VMs. Rackspace announced its Performance Cloud Servers in November, when it also renamed its "Next-Gen" instance types as Standard.
As with the previously-tested cloud providers, I ran on RightScale’s CentOS 6.4 images. Rackspace offers a somewhat-confusing discount program based on how long you’re willing to commit and whether you’ll pay up front. According to Rackspace, it is possible to get up to a 37% discount on its pricing with a three-year commitment and paying up front, so I have used a 37% discount as the “discounted pricing” amount.
Rackspace's Performance servers in theory should be competitive with Amazon’s newer instance types (e.g., M3, C3, I2) and Google Compute Engine. For example, the Performance instance types apparently run on Intel Xeon E5-2670 2.60GHz processors; AWS lists its M3 family as running on the E5-2670, the C3 family on the E5-2680, the I2 family on the E5-2670v2, and GCE says it uses 2.6GHz+ Xeon Sandy Bridge processors. However, on CentOS 6.4, I found that Rackspace's Performance instance types really didn’t perform significantly better than its Standard instance types.
Why? This description in a Rackspace developer blog post offers some clues: “So, terminology aside, today, when you boot any Linux image on Rackspace Cloud, that image uses standard paravirtualization (PV), for Windows and FreeBSD we use HVM. In the case of the new Performance Cloud Servers, we still default to PV images, and are currently testing PV on HVM images.”
But what exactly does this mean? Unfortunately, Rackspace’s own support portal has no useful information on PV vs. hardware-assisted virtualization (HVM), even though here, it means the difference between UnixBench scores that are competitive with AWS and GCE and scores that end up well below Azure. The short story is that you should see substantially higher UnixBench scores using HVM virtualization, at least on the processors being used. By not having HVM virtualization available yet for its Performance instance types for Linux, Rackspace is not offering a particularly competitive product today.
[Security concerns are driving organizations to consider private and hybrid clouds, according to a Rackspace survey last year. Read the report in "Private, Hybrid Cloud Interest Spurred By Security and Control."]
Based on conversations with several individuals at Rackspace, I believe (a) that Rackspace will have HVM support for Linux for its Performance instance types at some point in the first half of 2014 (note that this is my guess, not anything they said), and (b) that it will probably perform similarly to the benchmarks run on the beta version that Rackspace has made available, which gives numbers more than 3X better than what I got.
I will discuss my results by including both the “what if Rackspace supported HVM for Linux today” and the “what I actually got” perspectives. I should note that Rackspace has graciously given me access to a pre-release version of a CentOS image that can be launched with HVM on the Rackspace Cloud, which I plan to test before the conclusion of this IaaS benchmark series.
Rackspace Benchmark Results
Here are the main takeaways that I have from testing across Rackspace:
• For the first time in testing IaaS providers, I saw a regional difference: Rackspace Chicago delivered the three top single UnixBench scores (from the three most expensive instance types), and five of the top ten, but also five of the six worst scores (from the least expensive instance types). This requires further investigation to draw any significant conclusion, since I ran all these in a single day.
• The best price-per-performance for me was with the 512MB Standard, with the 1GB Performance a distant second. Of the more beefy instance types, the 8GB Performance stands out as having much better price-per-performance than the others (likely because it has eight vCPUs; you have to go to the 60GB Performance to get more than that).
• Performance in the single-threaded UnixBench results does appear to vary, at least a little bit, based on price, and the Performance types do better than the Standard types, even with everything running on PV virtualization. In particular, the 8GB, 15GB, 30GB, 60GB, 90GB, and 120GB Performance types all scored at least 544 (average) on the single-threaded UnixBench scores, whereas the rest of the instance types all scored 493 or less (average). However, under HVM virtualization, it appears that results over 1500 should be expected.
NEXT: Comparing Rackspace To Other IaaS Providers