Joe Masters Emison

Upcoming Events

Where the Cloud Touches Down: Simplifying Data Center Infrastructure Management

Thursday, July 25, 2013
10:00 AM PT/1:00 PM ET

In most data centers, DCIM rests on a shaky foundation of manual record keeping and scattered documentation. OpManager replaces data center documentation with a single repository for data, QRCodes for asset tracking, accurate 3D mapping of asset locations, and a configuration management database (CMDB). In this webcast, sponsored by ManageEngine, you will see how a real-world datacenter mapping stored in racktables gets imported into OpManager, which then provides a 3D visualization of where assets actually are. You'll also see how the QR Code generator helps you make the link between real assets and the monitoring world, and how the layered CMDB provides a single point of view for all your configuration data.

Register Now!

A Network Computing Webinar:
SDN First Steps

Thursday, August 8, 2013
11:00 AM PT / 2:00 PM ET

This webinar will help attendees understand the overall concept of SDN and its benefits, describe the different conceptual approaches to SDN, and examine the various technologies, both proprietary and open source, that are emerging. It will also help users decide whether SDN makes sense in their environment, and outline the first steps IT can take for testing SDN technologies.

Register Now!

More Events »

Subscribe to Newsletter

  • Keep up with all of the latest news and analysis on the fast-moving IT industry with Network Computing newsletters.
Sign Up

See more from this blogger

IaaS Performance Benchmarks Part 2: AWS

This is the second part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained the methodology I'm using to I'm test instance types across major IaaS providers. In this part, I post the results from my tests of Amazon Web Services instance types.

I launched more than 175 AWS VMs across current instance types, in all three U.S. regions (East, West and Oregon) and in three availability zones (AZs) per region. (The West and Oregon regions have only three AZs; East--for me--has only three AZs that allowed me to launch all of the instance types.) I ran instance-store-backed instances for every instance type I could; I ran EBS-backed for the t1.micro, and HVM-EBS-backed for the m3s and high-computing instance types.

In addition, I ran EBS-backed for a few of the instances that you can run either instance-store-backed or EBS-backed to see if there was a performance difference. The high-computing types aren’t allowed in US-West, and the cg type is only available in US-East. I was blocked from running a few instances in one of the US-West AZs. I ran a mix of spot and on-demand, and couldn’t find any performance difference between them.

I had to patch UnixBench to get it to run properly on the cg1 and cr1 instance types, as both have more than 16 cores. In my charts, I used on-demand pricing to compare relative differences in pricing. When I compare AWS to other providers in later parts of this series, I will discuss both reserved and on-demand pricing for AWS.

I also found that, in at least some circumstances, Ubuntu outperformed CentOS in benchmarks. However, I am sticking with CentOS for all benchmarks because there is more support for it across IaaS providers. (Specific circumstance: m3.2xlarge with HVM EBS AMIs had a single-core UnixBench score that was more than 20% higher on Ubuntu 12.04 than on CentOS 6.4).

Instance-Type Bake Off

Here are the main takeaways that I have from comparing instance types across the benchmarks:

• m1.small and t1.micro are both probably the best instance types as far as price/performance (at least if UnixBench is a reasonable measure of performance for you), but they are poor as far as performance-per-machine goes as compared with other options. If you’re using either, just know that if you pay more, you’ll get much better performance.

• The m3 family did really well in these benchmarks, but be warned that this is performance on HVM EBS AMIs. If you run m3s on regular EBS AMIs (which AWS will let you do, perhaps because HVM AMIs don’t appear to be available in US-West), the benchmark scores are significantly lower (well less than half in my benchmarks). Also, I would recommend testing Ubuntu (and perhaps other flavors of Linux) over CentOS for these, as I saw significantly better scores under Ubuntu with m3 instances in particular.

• Looking at the $1/hour or less (on-demand pricing) options: The m3.2xlarge ($1/hour), c1.xlarge ($0.68/hour) and m3.xlarge ($0.50/hour) stand out with significantly higher scores than the rest. One reason for favoring the c1.xlarge: You can run it backed with Amazon’s instance-store, instead of EBS, which means that you will not have the two single points of failure that exist with EBS-backed instances (the physical machine that the instance is on and the EBS volume).

• The cheapest instance that does fairly well in these benchmarks is the c1.medium ($0.145/hour). You have to pay more than five times as much to get double the performance in many of the benchmarks I ran, and the performance is close to much more expensive instance types for many uses.

• I think it would be hard to justify using the m1.xlarge in almost any circumstance. The m3.xlarge is only 2 cents/hour more than the m1.xlarge (on demand), has the same or better specifications, and gets significantly higher scores in these benchmarks.

• As you spend more money on AWS instances, you get generally better performance in these benchmarks. For example, the cg1.4xlarge costs about 3.5 times more than the c1.xlarge, and produced a multicore UnixBench score of roughly 3.5 times as much as the c1.xlarge.

• Amazon provides a good descriptive guide to picking instance types for particular uses, and these results should not be considered in any way a better general guide to selecting an appropriate instance-type family.

That said, I have met many AWS users who seem to select instance types based on either a “Hey, this one looks good and cheap” or a “Let’s spend as much money as possible so it goes fast” basis. If you know (or suspect) that your AWS deployment choices are based on this kind of criteria, these benchmarks should push you toward re-evaluating your selections.

UnixBench Single thread

UnixBench one thread per core

Note that in both SysBench charts above, I cut off the top for both the m1.small and t1.micro; their results were literally "off the chart."

Next Page: AWS Region Bake Off

Page:  1 | 2  | Next Page »

Related Reading

More Insights

Network Computing encourages readers to engage in spirited, healthy debate, including taking us to task. However, Network Computing moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing/SPAM. Network Computing further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | Please read our commenting policy.
Vendor Comparisons
Network Computing’s Vendor Comparisons provide extensive details on products and services, including downloadable feature matrices. Our categories include:

Research and Reports

Network Computing: April 2013

TechWeb Careers