Cloud Infrastructure

06:22 PM
Connect Directly
LinkedIn
Google+
Twitter
RSS
E-Mail
50%
50%
Repost This

IaaS Performance Benchmarks Part 3: AWS C3

In this part of a continuing series comparing IaaS cloud providers and their services, I test Amazon Web Services' new C3 instance types.

Editor's note: This post was updated Monday with information from Amazon after initial publication.

This is the third part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained the methodology I'm using to test instance types across major IaaS providers. In the second part, I ran benchmarks across every instance type in every U.S. region run by Amazon Web Services (AWS), the most widely used IaaS provider. In this part, I look specifically at the new C3 family of instances that AWS launched at its re:Invent conference, and update my AWS charts accordingly.

Amazon Web Services has been around long enough to want to deprecate/retire some of its instance types, as seen both by the excellent M3 family (replacing the M1 instance types) and now the new C3 family (replacing the C1 instances).

While it appeared to me initially that Elastic Block Storage (EBS) backing was required for the new C3 instances, Amazon informed me that's not the case. In my testing, I used EBS-backed C3s.

Two important notes about these updated benchmarks. First, in order to get these results, you must run C3 instances on Hardware Virtual Machine (HVM) EBS volumes, not ordinary EBS volumes. AWS will let you run on regular EBS volumes, but there is absolutely no reason to do this, because it costs the same to run under HVM EBS volumes, and the performance is significantly better. (For more on this, take a look at my first set of AWS results and my comparison of EBS-backed M3 instances versus HVM EBS-backed M3 instances).

Second, I'm dropping the SysBench results from these and further benchmarks, because, after hearing significant feedback from benchmarking experts, I don’t think they’re adding anything more to these foot-deep, miles-wide analyses across IaaS vendors. Now onto the results.

[Read about a new initiative to develop a framework that provides secure connectivity from any device to a cloud application in "Cloud Security Alliance Launches Secure Network Effort."]

To test the C3 family, I launched 30 AWS VMs -- five new C3 instance types each in six different availability zones (AZs) in US-East and US-Oregon (you can't launch HVM EBS-backed instances in US-West). As I found with the other AWS instance types, the C3 instance types had consistent performance for each instance type.

My main takeaway: the C3 family is pretty fantastic, and really sets a new bar for affordable performance. Many people use the c1.medium and c1.xlarge instances for Web servers and databases, and if you do, switch to the C3 posthaste, as you get significantly better performance for essentially the same price.


View Larger

If you’re running single-threaded, general-purpose applications, you’re crazy if you’re not using the C3 or M3 families. The c3.large, in particular, is really fantastic, at 15 cents per hour on demand. And, in general, unless you can’t afford 15 cents/hour, it’s hard to make a case that you should be running any of the M1 or C1 instances any more.


View Larger

The one-thread-per-core UnixBench results encompass such a large range on AWS that it’s necessary to break them into two charts in order to understand the range across instance types. I have chosen to do this on price, as opposed to performance, but please note that there are instances on the cheaper chart that handily beat instances on the more expensive chart in these UnixBench scores.


View Larger

One interesting note is that the C3 series doesn’t quite scale linearly with respect to these UnixBench scores: The average score jumps from 1867 to 2966 to 4876 to 6924 to 8472, but the price doubles each time. These benchmarks are running on CentOS (we saw previously in this series that running on Ubuntu improved the scores significantly), and I’m using RightScale’s CentOS images to get parity across IaaS vendors, so take the specific scaling here with a grain of salt. However, I would strongly suggest that you test how well they scale for your application, because it may make sense to stick with the c3.large and scale horizontally.

All in all, the C3 family looks excellent in these benchmarks, and if you’re talking to other IaaS vendors that are showing you comparison benchmarks, you need to make sure that they’re doing a comparison against the C3s. Note, though, they almost certainly aren’t -- I haven’t seen a set of benchmarks touted by another IaaS vendor that ventured outside of the deprecated M1 family.

In the next few weeks, I’ll post my results from Google Compute Engine and Windows Azure, and comparing them to the best-performing AWS instance types. And again, please remember the aim of this benchmarking project: I’m trying to give all of us a high-level view on IaaS offerings. I would advise you to do further testing, based upon what you can afford.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Devops
50%
50%
Devops,
User Rank: Apprentice
4/23/2014 | 3:32:25 AM
re: IaaS Performance Benchmarks Part 3: AWS C3
AWS C3 instances are the new generation compute optimized instances. These results of the raw CoreMark score and the CoreMark score/$ per hour which gives us an idea of the CPU performance with respect to the cost.
Joe Masters Emison
50%
50%
Joe Masters Emison,
User Rank: Apprentice
11/25/2013 | 6:52:03 PM
re: IaaS Performance Benchmarks Part 3: AWS C3
The hardest part about migrating existing deployments will probably be getting an HVM EBS AMI you can use. There don't appear to be any Ubuntu ones for Ubuntu 9 or 10, so you'll have to upgrade, which will probably take some deal of effort (of course, less effort if you're using configuration management, but still some...)
More Blogs from Commentary
SDN: Waiting For The Trickle-Down Effect
Like server virtualization and 10 Gigabit Ethernet, SDN will eventually become a technology that small and midsized enterprises can use. But it's going to require some new packaging.
IT Certification Exam Success In 4 Steps
There are no shortcuts to obtaining passing scores, but focusing on key fundamentals of proper study and preparation will help you master the art of certification.
VMware's VSAN Benchmarks: Under The Hood
VMware touted flashy numbers in recently published performance benchmarks, but a closer examination of its VSAN testing shows why customers shouldn't expect the same results with their real-world applications.
Building an Information Security Policy Part 4: Addresses and Identifiers
Proper traffic identification through techniques such as IP addressing and VLANs are the foundation of a secure network.
SDN Strategies Part 4: Big Switch, Avaya, IBM,VMware
This series on SDN products concludes with a look at Big Switch's updated SDN strategy, VMware NSX, IBM's hybrid approach, and Avaya's focus on virtual network services.
Hot Topics
3
Converged Infrastructure: 3 Considerations
Bill Kleyman, National Director of Strategy & Innovation, MTM Technologies,  4/16/2014
White Papers
Register for Network Computing Newsletters
Cartoon
Current Issue
Video
Slideshows
Twitter Feed