Cloud Infrastructure

07:27 PM
Connect Directly
Repost This

IaaS Performance Benchmarks Part 5: Microsoft Azure Virtual Machines

As part of my project comparing IaaS services, I tested Windows Azure Virtual Machines and also compared Azure to AWS and Google Compute Engine. Here are the results.

This is the fifth part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained my methodology for testing instance types across major IaaS providers. In the second and third parts, I ran benchmarks across every instance type in every US region run by Amazon Web Services (AWS). In the fourth part, I evaluated Google Compute Engine (GCE), finding it a solid competitor to AWS. In this part, I look at Microsoft Windows Azure Virtual Machines, which are available for use with Linux.

There aren’t many companies large enough to take on Amazon, but Microsoft certainly qualifies, and so Windows Azure Virtual Machines is seen by many as a potential worthy competitor to AWS. But does it measure up?

As of today, there are two US regions available for Azure VMs through the RightScale cloud-management service: East and West. In each of those regions, I kicked off one of each Azure instance type that RightScale offers (at the moment, that does not include the A5 instance type), for a total of 14 VMs. As with AWS and GCE, I ran on RightScale’s CentOS 6.4 images. Azure offers discounted pricing that is somewhat similar to Amazon’s Reserved Instances, except it’s less painful -- you commit to pay for the server for six or 12 months and get reduced prices instead of having to pay an amount up front.

Azure Virtual Machines Comparisons

Here are the main takeaways that I have from comparing instance types across Azure:

• The best price-per-performance for me was with the Extra Small, which is subject to more serious fluctuations in performance; your mileage will vary at any given moment. Both the Small and Medium were distant seconds.

• Azure has fairly consistent performance across VMs as far as the single-threaded UnixBench results go, so if you’re just running a single thread and don’t need a massive amount of RAM, you should consider the Extra Small and Small.

• Like other IaaS vendors, Azure does offer some “high-memory” instance types. For about twice as much money, you can get four times the RAM, but no better performance (at least according to these benchmarks). By comparison, GCE offers more than seven times the RAM for twice as much money, and AWS offers more than twice as much RAM for less than twice as much money.

View Larger

View Larger

Comparing Azure To AWS And GCE

So how does Azure Virtual Machines stack up to AWS and GCE, at least as far as these fairly simplistic benchmarks go?

• If you get the same mileage I got out of the Extra Small instance (which shares CPU with other instances, so presumably if it gets popular, it’s going to suffer performance-wise), you should use it. For only around $14/month on demand, I got UnixBench single-core scores of over 800, which beats GCE’s f1-micro ($14/mo., UnixBench single-core average of 456) and AWS’s t1.micro ($14/mo. on demand, UnixBench single-core average of 187). You have to go to GCE’s g1-small, at about three times as much, to do better ($39/mo., UnixBench single-core average score of 891).

• Other Azure instance types don’t fare so well in the single-core price-for-performance comparison. That said, you may not be considering Azure because it’s the best performance option; you may be considering it because you’re running a .NET stack and want the ability to run some Linux VMs as well, using a single vendor. In that case, Azure’s single-core results are certainly in the same relative ballpark as AWS and GCE.

View Larger

View Larger

• The Small instance type is more expensive (even with the 12-month commitment) than GCE’s g1-small, and had slightly worse performance in my tests. Still, they’re pretty close, and AWS doesn’t really have a good option here, so if you’re thinking that you will only ever need a single-core instance with 1.7GB of RAM, then Azure is not a bad choice.

• The Medium instance type has similar specs to GCE’s n1-standard-1 (although GCE only offers one core here) and AWS’s c3.large, except both the n1-standard-1 and c3.large beat the Medium’s multi-core UnixBench score here by more than 40%, and cost roughly the same or less.

[Read how Microsoft is building and leasing data center space worldwide to bulk up its Azure and Xbox Live services in "Microsoft's Data Center Push."]

• The Large instance type has similar specs to GCE’s n1-standard-2 (although GCE only offers two cores here) and AWS’s c3.xlarge. The Large, if purchased on a 12-month commitment, is actually cheaper than the n1-standard-2, and returned slightly higher UnixBench scores. However, AWS’s c3.xlarge beat the Large’s multi-core UnixBench score by 57% and is cheaper than the Large.

• Perhaps the biggest problem with Azure’s offering at this point is that it caps out performance-wise at the Extra Large, available for $353-$518/month with 14GB RAM, which is outperformed handily by AWS’s c3.2xlarge ($260-432/mo., 15GB RAM) and also by GCE’s n1-standard-4 ($382/mo., 15GB RAM).

View Larger

It's no easy feat for Microsoft to put together such a solid Linux offering as Azure Virtual Machines. The idea that I can launch CentOS 6.4 VMs with Microsoft via API on both coasts within minutes is still somewhat shocking to me -- I certainly can’t imagine that such a thing would have been allowed to be spoken aloud in Redmond five years ago.

That said, I don’t really see Microsoft as a strong competitor for companies that mainly use Linux on their cloud servers. Both Amazon and Google seem like better choices to me from every angle. But for organizations that develop using a .NET stack and/or run primarily Windows-based VMs, I can see Azure’s Virtual Machines as a solid way to run some Linux servers alongside the Windows ones.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
12/16/2013 | 7:24:10 PM
re: IaaS Performance Benchmarks Part 5: Microsoft Azure Virtual Machines
Good stuff Joe! Now that you've had an opportunity to kick the tires on all three, do you have any feedback on usability, and tooling provided by the vendors?
Joe Masters Emison
Joe Masters Emison,
User Rank: Apprentice
12/11/2013 | 12:11:28 PM
re: IaaS Performance Benchmarks Part 5: Microsoft Azure Virtual Machines
I suspect that there are differences in terms of reliability, but they're not easy to identify from the type of testing I've been doing. All of the vendors I've looked at so far can spin up servers when I ask, and the servers come up properly (this is not something that was necessarily true 2-3 years ago), and none of the servers I've launched have died inexplicably (but I'm only running them for an hour or so). I would bet that AWS and Google are more reliable than Azure, simply because they've been doing it longer (Google internally for longer than AWS, I believe), but I have no hard facts to back that up with.
User Rank: Apprentice
12/10/2013 | 7:08:51 PM
re: IaaS Performance Benchmarks Part 5: Microsoft Azure Virtual Machines
Hi Joe -- Is there much difference in terms of reliability between IaaS providers, or does that have more to do with a customer's architectural decisions?
More Blogs from Commentary
Infrastructure Challenge: Build Your Community
Network Computing provides the platform; help us make it your community.
Edge Devices Are The Brains Of The Network
In any type of network, the edge is where all the action takes place. Think of the edge as the brains of the network, while the core is just the dumb muscle.
SDN: Waiting For The Trickle-Down Effect
Like server virtualization and 10 Gigabit Ethernet, SDN will eventually become a technology that small and midsized enterprises can use. But it's going to require some new packaging.
IT Certification Exam Success In 4 Steps
There are no shortcuts to obtaining passing scores, but focusing on key fundamentals of proper study and preparation will help you master the art of certification.
VMware's VSAN Benchmarks: Under The Hood
VMware touted flashy numbers in recently published performance benchmarks, but a closer examination of its VSAN testing shows why customers shouldn't expect the same results with their real-world applications.
Hot Topics
SDN Strategies Part 4: Big Switch, Avaya, IBM,VMware
Kurt Marko, Contributing Editor,  4/18/2014
White Papers
Register for Network Computing Newsletters
Current Issue
Twitter Feed