IaaS Performance Benchmarks Part 5: Microsoft Azure Virtual Machines

This is the fifth part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained my methodology for testing instance types across major IaaS providers. In the second and third parts, I ran benchmarks across every instance type in every US region run by Amazon Web Services (AWS). In the fourth part, I evaluated Google Compute Engine (GCE), finding it a solid competitor to AWS. In this part, I look at Microsoft Windows Azure Virtual Machines, which are available for use with Linux.

There aren’t many companies large enough to take on Amazon, but Microsoft certainly qualifies, and so Windows Azure Virtual Machines is seen by many as a potential worthy competitor to AWS. But does it measure up?

As of today, there are two US regions available for Azure VMs through the RightScale cloud-management service: East and West. In each of those regions, I kicked off one of each Azure instance type that RightScale offers (at the moment, that does not include the A5 instance type), for a total of 14 VMs. As with AWS and GCE, I ran on RightScale’s CentOS 6.4 images. Azure offers discounted pricing that is somewhat similar to Amazon’s Reserved Instances, except it’s less painful -- you commit to pay for the server for six or 12 months and get reduced prices instead of having to pay an amount up front.

Azure Virtual Machines Comparisons

Here are the main takeaways that I have from comparing instance types across Azure:

•The best price-per-performance for me was with the Extra Small, which is subject to more serious fluctuations in performance; your mileage will vary at any given moment. Both the Small and Medium were distant seconds.

•Azure has fairly consistent performance across VMs as far as the single-threaded UnixBench results go, so if you’re just running a single thread and don’t need a massive amount of RAM, you should consider the Extra Small and Small.

•Like other IaaS vendors, Azure does offer some “high-memory” instance types. For about twice as much money, you can get four times the RAM, but no better performance (at least according to these benchmarks). By comparison, GCE offers more than seven times the RAM for twice as much money, and AWS offers more than twice as much RAM for less than twice as much money.

View Larger

View Larger

Comparing Azure To AWS And GCE

So how does Azure Virtual Machines stack up to AWS and GCE, at least as far as these fairly simplistic benchmarks go?

•If you get the same mileage I got out of the Extra Small instance (which shares CPU with other instances, so presumably if it gets popular, it’s going to suffer performance-wise), you should use it. For only around $14/month on demand, I got UnixBench single-core scores of over 800, which beats GCE’s f1-micro ($14/mo., UnixBench single-core average of 456) and AWS’s t1.micro ($14/mo. on demand, UnixBench single-core average of 187). You have to go to GCE’s g1-small, at about three times as much, to do better ($39/mo., UnixBench single-core average score of 891).

•Other Azure instance types don’t fare so well in the single-core price-for-performance comparison. That said, you may not be considering Azure because it’s the best performance option; you may be considering it because you’re running a .NET stack and want the ability to run some Linux VMs as well, using a single vendor. In that case, Azure’s single-core results are certainly in the same relative ballpark as AWS and GCE.

View Larger

View Larger

•The Small instance type is more expensive (even with the 12-month commitment) than GCE’s g1-small, and had slightly worse performance in my tests. Still, they’re pretty close, and AWS doesn’t really have a good option here, so if you’re thinking that you will only ever need a single-core instance with 1.7GB of RAM, then Azure is not a bad choice.

•The Medium instance type has similar specs to GCE’s n1-standard-1 (although GCE only offers one core here) and AWS’s c3.large, except both the n1-standard-1 and c3.large beat the Medium’s multi-core UnixBench score here by more than 40%, and cost roughly the same or less.

[Read how Microsoft is building and leasing data center space worldwide to bulk up its Azure and Xbox Live services in "Microsoft's Data Center Push."]

•The Large instance type has similar specs to GCE’s n1-standard-2 (although GCE only offers two cores here) and AWS’s c3.xlarge. The Large, if purchased on a 12-month commitment, is actually cheaper than the n1-standard-2, and returned slightly higher UnixBench scores. However, AWS’s c3.xlarge beat the Large’s multi-core UnixBench score by 57% and is cheaper than the Large.

•Perhaps the biggest problem with Azure’s offering at this point is that it caps out performance-wise at the Extra Large, available for $353-$518/month with 14GB RAM, which is outperformed handily by AWS’s c3.2xlarge ($260-432/mo., 15GB RAM) and also by GCE’s n1-standard-4 ($382/mo., 15GB RAM).

View Larger

It's no easy feat for Microsoft to put together such a solid Linux offering as Azure Virtual Machines. The idea that I can launch CentOS 6.4 VMs with Microsoft via API on both coasts within minutes is still somewhat shocking to me -- I certainly can’t imagine that such a thing would have been allowed to be spoken aloud in Redmond five years ago.

That said, I don’t really see Microsoft as a strong competitor for companies that mainly use Linux on their cloud servers. Both Amazon and Google seem like better choices to me from every angle. But for organizations that develop using a .NET stack and/or run primarily Windows-based VMs, I can see Azure’s Virtual Machines as a solid way to run some Linux servers alongside the Windows ones.