Cloud Infrastructure

03:40 PM
Connect Directly
LinkedIn
Google+
Twitter
RSS
E-Mail
50%
50%
Repost This

IaaS Performance Benchmarks Part 2: AWS

I conducted my own benchmarking project to compare IaaS cloud providers and their services. I tested Amazon Web Services first, and here are the results.

This is the second part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained the methodology I'm using to I'm test instance types across major IaaS providers. In this part, I post the results from my tests of Amazon Web Services instance types.

I launched more than 175 AWS VMs across current instance types, in all three U.S. regions (East, West and Oregon) and in three availability zones (AZs) per region. (The West and Oregon regions have only three AZs; East--for me--has only three AZs that allowed me to launch all of the instance types.) I ran instance-store-backed instances for every instance type I could; I ran EBS-backed for the t1.micro, and HVM-EBS-backed for the m3s and high-computing instance types.

In addition, I ran EBS-backed for a few of the instances that you can run either instance-store-backed or EBS-backed to see if there was a performance difference. The high-computing types aren’t allowed in US-West, and the cg type is only available in US-East. I was blocked from running a few instances in one of the US-West AZs. I ran a mix of spot and on-demand, and couldn’t find any performance difference between them.

I had to patch UnixBench to get it to run properly on the cg1 and cr1 instance types, as both have more than 16 cores. In my charts, I used on-demand pricing to compare relative differences in pricing. When I compare AWS to other providers in later parts of this series, I will discuss both reserved and on-demand pricing for AWS.

I also found that, in at least some circumstances, Ubuntu outperformed CentOS in benchmarks. However, I am sticking with CentOS for all benchmarks because there is more support for it across IaaS providers. (Specific circumstance: m3.2xlarge with HVM EBS AMIs had a single-core UnixBench score that was more than 20% higher on Ubuntu 12.04 than on CentOS 6.4).

Instance-Type Bake Off

Here are the main takeaways that I have from comparing instance types across the benchmarks:

• m1.small and t1.micro are both probably the best instance types as far as price/performance (at least if UnixBench is a reasonable measure of performance for you), but they are poor as far as performance-per-machine goes as compared with other options. If you’re using either, just know that if you pay more, you’ll get much better performance.

• The m3 family did really well in these benchmarks, but be warned that this is performance on HVM EBS AMIs. If you run m3s on regular EBS AMIs (which AWS will let you do, perhaps because HVM AMIs don’t appear to be available in US-West), the benchmark scores are significantly lower (well less than half in my benchmarks). Also, I would recommend testing Ubuntu (and perhaps other flavors of Linux) over CentOS for these, as I saw significantly better scores under Ubuntu with m3 instances in particular.

• Looking at the $1/hour or less (on-demand pricing) options: The m3.2xlarge ($1/hour), c1.xlarge ($0.68/hour) and m3.xlarge ($0.50/hour) stand out with significantly higher scores than the rest. One reason for favoring the c1.xlarge: You can run it backed with Amazon’s instance-store, instead of EBS, which means that you will not have the two single points of failure that exist with EBS-backed instances (the physical machine that the instance is on and the EBS volume).

• The cheapest instance that does fairly well in these benchmarks is the c1.medium ($0.145/hour). You have to pay more than five times as much to get double the performance in many of the benchmarks I ran, and the performance is close to much more expensive instance types for many uses.

• I think it would be hard to justify using the m1.xlarge in almost any circumstance. The m3.xlarge is only 2 cents/hour more than the m1.xlarge (on demand), has the same or better specifications, and gets significantly higher scores in these benchmarks.

• As you spend more money on AWS instances, you get generally better performance in these benchmarks. For example, the cg1.4xlarge costs about 3.5 times more than the c1.xlarge, and produced a multicore UnixBench score of roughly 3.5 times as much as the c1.xlarge.

• Amazon provides a good descriptive guide to picking instance types for particular uses, and these results should not be considered in any way a better general guide to selecting an appropriate instance-type family.

That said, I have met many AWS users who seem to select instance types based on either a “Hey, this one looks good and cheap” or a “Let’s spend as much money as possible so it goes fast” basis. If you know (or suspect) that your AWS deployment choices are based on this kind of criteria, these benchmarks should push you toward re-evaluating your selections.

UnixBench Single thread

UnixBench one thread per core



Note that in both SysBench charts above, I cut off the top for both the m1.small and t1.micro; their results were literally "off the chart."

Next Page: AWS Region Bake Off

Previous
1 of 2
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
PaulW464
50%
50%
PaulW464,
User Rank: Apprentice
11/6/2013 | 11:56:08 PM
re: IaaS Performance Benchmarks Part 2: AWS
Re: m1.xlarge v m3.xlarge - the m1 does have higher IO potential - 1000 v 500 for EBS optimised. Depending on your usage if that becomes significant.
Joe Masters Emison
50%
50%
Joe Masters Emison,
User Rank: Apprentice
10/17/2013 | 7:17:58 PM
re: IaaS Performance Benchmarks Part 2: AWS
I'm considering just focusing on the UnixBench results for the future parts, given that the SysBench MySQL test is running untuned, and just doesn't seem to be adding anything to the analysis. Any thoughts to the contrary would be helpful.
Joe Masters Emison
50%
50%
Joe Masters Emison,
User Rank: Apprentice
10/17/2013 | 4:08:54 PM
re: IaaS Performance Benchmarks Part 2: AWS
I think AWS has so many different instance types and options that almost any normal workload can be run on AWS and would be good from a cost/performance standpoint. I think the hardest thing on AWS right now is to figure out (a) which instance type is best for what you're doing, and (b) how best to architect your application and deployment so that you take advantage of the full benefits of the cloud and minimizing vendor lock-in.

The types of workload that I see as failing to make sense on AWS (and probably on any public IaaS provider) are those that (a) don't need the full set of availability and security benefits you get from AWS (e.g., some internal business tools), (b) are very expensive to migrate to AWS, and (c) have extremely specialized performance needs that cannot be attained by using virtualized hardware or by being in specific data centers (e.g., high-frequency trading).
lol3
50%
50%
lol3,
User Rank: Apprentice
10/17/2013 | 11:46:53 AM
re: IaaS Performance Benchmarks Part 2: AWS
lol.
Drew Conry-Murray
50%
50%
Drew Conry-Murray,
User Rank: Strategist
10/16/2013 | 8:33:10 PM
re: IaaS Performance Benchmarks Part 2: AWS
Hi Joe, any advice on what kinds of workloads make the most sense on AWS from a cost/performance standpoint?
More Blogs from Commentary
Infrastructure Challenge: Build Your Community
Network Computing provides the platform; help us make it your community.
Edge Devices Are The Brains Of The Network
In any type of network, the edge is where all the action takes place. Think of the edge as the brains of the network, while the core is just the dumb muscle.
Fight Software Piracy With SaaS
SaaS makes application deployment easy and effective. It could eliminate software piracy once and for all.
SDN: Waiting For The Trickle-Down Effect
Like server virtualization and 10 Gigabit Ethernet, SDN will eventually become a technology that small and midsized enterprises can use. But it's going to require some new packaging.
IT Certification Exam Success In 4 Steps
There are no shortcuts to obtaining passing scores, but focusing on key fundamentals of proper study and preparation will help you master the art of certification.
Hot Topics
6
Fight Software Piracy With SaaS
Andrew Froehlich, President & Lead Network Architect, West Gate Networks,  4/22/2014
6
Edge Devices Are The Brains Of The Network
Orhan Ergun, Network Architect,  4/23/2014
White Papers
Register for Network Computing Newsletters
Cartoon
Current Issue
2014 Private Cloud Survey
2014 Private Cloud Survey
Respondents are on a roll: 53% brought their private clouds from concept to production in less than one year, and 60% ­extend their clouds across multiple datacenters. But expertise is scarce, with 51% saying acquiring skilled employees is a roadblock.
Video
Slideshows
Twitter Feed