IaaS Performance Benchmarks Part 2: AWS: Page 2 of 2

In order to compare all three regions, I had to ignore the high-performance computing instance types (which aren’t available in US-West). I also did a straight comparison of US-East to US-Oregon, just of the high-performance instance types that are in both. Finally, I ignored a small number of my 175 runs in order to make sure I had the same number of the same instance types in each region, and kept diversity across AZs. This means that the three-region comparison was over 38 VM runs each, and the two-region, high-performance comparison was over 12 VM runs each.

My takeaway: It’s a wash. It's hard to argue that any region is substantially better than the rest, although if you had to pick one, US-East does better here than the rest.

Now, comparing US-East and US-Oregon (only the high-performance computing instances):

Here, US-East beats US-Oregon fairly convincingly for high-performance types, and US-East also is the exclusive region with the very good cg1.4xlarge instance type. So, I see US-East as the obvious choice for running high-performance instance types.

These cross-region analyses make a decent data-based rebuttal to recent complaints about Amazon having a “legacy” cloud in US-East.

[Find out what security issues to consider when signing a contract with an IaaS provider in "Top IaaS Security Requirements."]

I also looked at cross-AZ performance, but because everyone can have different availability zone assignments (for example, my US-East-1a could potentially be using data centers that are split across your US-East-1b and US-East-1e), charts aren’t particularly useful here. Suffice to say that I did see performance differences across AZs in all regions, so that you’re going to get the luck of the draw, or you’ll have to performance-test everything you launch to get the best performance. You can’t just pick a particular region or availability zone and expect it to be the best.

That said, the performance differences aren’t that huge, so luck-of-the-draw is probably fine for most use cases. It's probably cheaper to pay for a beefier instance than to performance-test everything on load.

Other Observations

I've always wondered whether there was a significant difference between running EBS-backed instances or instance-store-backed instances, especially with respect to I/O; I can imagine theoretical arguments going either way. The short answer is that there isn’t a big difference (at least as far as the benchmarks I ran). I launched 8 c1.xlarges, 8 m1.mediums and 8 m2.xlarges as both EBS-backed and instance-store-backed across all three U.S. regions, and did not find significant differences:

Finally, I think it’s worth acknowledging how good the AWS service is: Can you imagine telling someone 10 years ago that it would be possible to test the performance on more than 175 different VMs in more than nine different data centers across the United States with a few hours of programming, a few hours of actual machine runtime and about $150 in total costs?

I ran into zero problems with AWS. My only minor complaint is that some instance types weren’t runnable in some availability zones, but I was able to run every instance type in each region in which Amazon said it would work, on-demand, with launch times of a few minutes. And a tip of my hat to RightScale as well; its RightImages, ServerTemplates, RightScripts and API made this overall project possible.

During the next few weeks I will post results from other IaaS providers and compare them to AWS and each other. I will likely focus on just a handful of AWS instance types; I just don’t see the point of comparing any of the instance types in the m1 or the more specialized families that don’t show well in generalized benchmarks--like the m2, hi or hs--to other providers, considering that they don’t do well in these benchmarks and in my personal experience. By the end of this project, we’ll all be a bit more informed about the options across IaaS providers.