• 07/23/2014
    9:06 AM
  • Rating: 
    0 votes
    Vote up!
    Vote down!

Guide To IaaS Cloud Provider Performance

Choosing an Infrastructure-as-a-service cloud provider? Use these benchmarks to compare AWS, Google, Azure, Rackspace, and more.

I've been frustrated by a lack of comprehensive comparison benchmarks between Infrastructure-as-a-Service (IaaS) providers and within providers themselves. Most performance data shows only a small number of available "instance types" in a limited region, with no way to compare pricing or application performance.

As an Amazon Web Services customer, I know from my own experience that the c1.xlarge performs really well for many applications, but no one appears to have ever compared that to other cloud services.

So, I decided to run my own IaaS performance project and benchmarked every instance type across the major public IaaS provider regions on a consistent setup. My primary focus was to provide a broad, comprehensive view of most serious compute options available to IaaS customers.

Part 1: Test Methodology
First, I explained my goals for the benchmarks and how I used the tools UnixBench and SysBench’s MySQL to test different instance types in different data centers across different providers.

Part 2: Amazon Web Services
I launched more than 175 AWS VMs across instance types and compared performance by region and availability zone. Here you can read the detailed results.

Part 3: Amazon Web Services C3
After Amazon rolled out the C3 family of instances, I ran the benchmarking tests using C3 and updated my AWS charts accordingly.

Part 4: Google Compute Engine
I compared instance types across Google Compute Engine and against AWS, finding it a solid competitor to Amazon's offerings.

Part 5: Microsoft Azure Virtual Machines
I tested Windows Azure Virtual Machines and compared Azure to AWS and Google Compute Engine. Here you'll find the results.

Part 6: Rackspace Cloud Servers
I evaluated Rackspace Cloud Servers' compute resources and how they compare to AWS, Google Compute Engine, and Azure. See how they measure up.

Part 7: SoftLayer
In this part, I looked at SoftLayer, an IBM company, which offers bare-metal (not virtualized) pay-by-the-hour services.

Part 8: HP Public Cloud
I tested HP Public Cloud across regions and compared it to the other IaaS providers I've benchmarked. HP has recently focused on providing an OpenStack-based public cloud.

Do you have experience with these IaaS providers? Let us know how they fared in the comments.




I wonder how the uncertainty about Rackspace's future is affecting the IaaS landscape. The company announced in May that it was evaluating its options, but has yet to find a buyer.  

Useful for private cloud implementations?

I also wonder how applicable this strategy might be for those looking to provide similar services such as IT-as-a-Service or other IaaS offerings internally? I would think it is just as important to create benchmarks and/or measure internal services against external benchmarks. It's only then that you can assure users that you're giving them everything and then some when it comes to performance and offerings.




Re: Useful for private cloud implementations?

IT as a service - is getting very popular and helps keep certain level clients of cost down.  And at the same time these vendors can keep up with the multitude of technologies by dispersing across many people.

Alternative IaaS Providers

This is definitely a nice thing you put together, but I would be interested in seeing it expanded to see how a few more providers like Digital Ocean,, and Linode compare against the players you mentioned.

Re: Alternative IaaS Providers

KennethB, thanks for the feedback. We'll look into doing some expanded testing. Do you use any of the other providers you listed?