IaaS Performance Benchmarks Part 8: HP Public Cloud

This is the eighth part in a series of articles about creating my own IaaS performance benchmarking project. After explaining my methodology for testing instance types across IaaS providers, I've run benchmarks for Amazon Web Services, Google Compute Engine, Microsoft Azure Virtual Machines, Rackspace Cloud Servers, and SoftLayer. In this part, I look at HP Public Cloud. HP has offered IaaS for a few years, but more recently has focused on providing an OpenStack-based public cloud.

While I had wanted to test HP Public Cloud through RightScale (as I had done in my other cloud tests), RightScale does not currently support all regions/zones that HP has in the U.S. So I verified that I got similar results testing RightScale CentOS 6.4 images through RightScale in HP's Availability Zone 1 and HP CentOS 6.4 images directly through HP, and then tested HP's CentOS 6.4 images in all of the availability zones. Thankfully, the OpenStack command-line interface (CLI) is well-documented and supported by HP (and the OpenStack community), and so testing was easy to automate.

As of today, there are two U.S. regions (East and West), each with three availability zones, and 11 instance types, although not every instance type is available in each availability zone. I ended up running 46 VMs across the six different availability zones on HP’s CentOS 6.4 image.

At this point in time, HP’s public pricing page does not offer any discounts for commitments or pre-payment. I was told by an HP salesperson that discounts are available, but are negotiated between customers and HP, so I can’t speak to how much those would be.

HP Public Cloud Benchmark Results

Here are the main takeaways that I have from testing across HP Cloud:

•I did not find HP Public Cloud to have significant variations between regions in terms of performance, although I did find that West-AZ2 and West-AZ3 were the most limited in terms of instance availability.

•Most IaaS providers either fall into the category of (a) having a single family that all runs on the same underlying hardware and naming instances Small, Medium, etc., based on the number of cores/RAM allocated or (b) having multiple families and naming instances HighCPU-Medium, HighMem-Medium, etc. HP does something different; it has a “regular” family and a HighMem family, but even within the regular family, there are clear variations in performance.

For example, by the single-core UnixBench scores, the Standard 8XLarge does better than the Standard 4XLarge, which is running on a better processor than the Standard 2XLarge, and on down the line. Most individual benchmarks in UnixBench showed these differences (e.g., Shell Scripts -- 1 concurrent -- ran almost four times faster on the Standard XLarge than on the Medium). Yet all of them get reported as “Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz” by the VM. So if you’re comparing with other providers, you should really consider that each instance type is its own family. (Note that HP Cloud does not address this in its FAQ or feature description).

•Like other clouds, the best price-per-performance for single-core UnixBench is the cheapest one; with HP, it's the Standard XSmall. HP has a gentle price-per-performance drop as you vertically scale instances, so unlike other providers, there aren’t a few small instances that have much better price-per-performance than the rest.

•HP’s price-per-performance for multi-threaded UnixBench scores matches pricing so well that there’s not much to say. If you go with HP Public Cloud, you don’t really need to think about instance-type selection as much as you would with other IaaS providers.

NEXT: Comparing HP To Other IaaS Providers

How does HP compare to the other IaaS Providers I’ve tested, far as these fairly basic benchmarks go?

•Because HP’s cheaper instances get lower single-core UnixBench scores than its more-expensive instances (in the “each instance type is its own family” model), it doesn’t set up well to win price-per-performance against other clouds, and so it really doesn’t register on the chart.

•HP Cloud beats Rackspace (at least under PV virtualization), and SoftLayer’s virtualized instances in terms of multi-threaded UnixBench, and probably edges out Azure, depending on your requirements. For example, here's a comparison:

HP’s Standard 2XLarge: $657/month, 30GB RAM, average multi-core UnixBench score of 3143.

Rackspace 30GB Performance: $979/month (as low as $617/month discounted), 30GB RAM, average multi-core UnixBench score of 2104.

SoftLayer Extra Large: $821/month (as low as $768/month discounted), 16GB RAM, average multi-core UnixBench score of 2301.

Azure A6: $576/month, 28GB RAM, average multi-core UnixBench score of 1936; and Extra Large $518/month (as low as $353/month), 14GB RAM, average multi-core UnixBench score of 2684.

•In general, HP Public Cloud’s multi-thread UnixBench scores are beaten by AWS, GCE, and SoftLayer Bare Metal: AWS’s c3.2xlarge is $432/month on demand (and as low as $172/month discounted), 28GB RAM, with a multi-core UnixBench score of 4876; GCE’s n1-highmem-4 is $440/month on demand, 26GB RAM, with a multi-core UnixBench score of 2790; and SoftLayer’s Bare Metal 8/8 is $720/month on demand (and as low as $259/month), 8GB RAM, with a multi-core UnixBench score of 6397.

•At the high end (see chart below) HP’s Standard 8XLarge multi-threaded UnixBench scores are only bested by AWS c3.4xlarge, c3.8xlarge, cr1.8xlarge, and SoftLayer Bare Metal, 16/16 although the HP Standard 8XLarge has more RAM (120GB) than all of those but the cr1.8xlarge (244GB). Google Compute Engine is beta-testing its instances that should reach these performance levels, but they’re not in stable release yet.

Among the major IaaS vendors today, only two base their services on OpenStack: HP and Rackspace. We have been running OpenStack at my company and I'm impressed with its progress and momentum. I certainly see the attraction of being able to run the same system and interfaces between a private cloud and public clouds, although I think it can be overblown.

For companies that are excited about OpenStack, it may make sense to work with a vendor like HP or Rackspace, both of which sell services around private, public, and hybrid clouds. And today, HP’s Linux offerings on its public cloud get higher UnixBench scores (and better UnixBench price-per-performance) than Rackspace’s. However, this will almost certainly change when Rackspace is able to get HVM virtualization up for its Linux instances.

In the course of testing, I also found HP’s support and interface gave me a better user experience than Rackspace’s. The HP engineers with whom I spoke in the Live Chat -- almost always outside of regular business hours -- were fast and extremely knowledgeable. The Rackspace engineers were slower to respond and less knowledgeable (for example, telling me incorrectly that Rackspace had HVM virtualization active on all of the company's instance types on Linux), although this could just be a factor of Rackspace having more users that need support and the difficulty in finding strong support personnel.

And while both providers have decent interfaces, HP’s Horizon interface was noticeably faster and served as a great complement to the CLI for power users, whereas Rackspace’s treats users like newbies all the time, no matter how many servers you want to launch.

In the end, though, I am more compelled by best-of-breed solutions than single-vendor solutions. I prefer to have my vendors laser-focused on a small number of core competencies than to have a one-stop-shop for all of my needs, as long as I have a good way to make them interoperate. I think there are enough good abstraction layers/cloud management platforms to make interaction between different clouds not-too-difficult to conquer, and so I don’t see the one-stop OpenStack shop as a compelling enough feature to determine my IaaS choices.