IaaS Performance Benchmarks Part 4: Google Compute Engine

This is the fourth part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained the methodology I'm using to test instance types across major IaaS providers. In the second and third parts, I ran benchmarks across every instance type in every US region run by Amazon Web Services (AWS), the most widely used IaaS provider. In this part, I look at Google Compute Engine (GCE), which just came out of beta and is widely seen as one of the biggest potential competitors to AWS. I compared instance types across GCE as well as against AWS.

As of today, there is only one US region with two zones available for GCE in RightScale. In that region, in each of its zones, I kicked off one of each GCE instance type that comes with local disk storage, for a total of 24 VMs. As with AWS, I ran on RightScale’s CentOS 6.4 images. One note: GCE makes life simpler than AWS in terms of pricing, as you don’t have to deal with choosing and allocating reserved instances, and GCE bills in minute increments, not hour increments. That said, because AWS is the industry leader, when I compare GCE to AWS, I will calculate both the most expensive and cheapest pricing that AWS can provide.

GCE Instance Comparisons

Here are the main takeaways that I have from comparing instance types across GCE:

• Performance is consistent across both GCE zones, regardless of instance type (except that you should expect the performance of the f1-micro to fluctuate). This is good, although with only two US zones, it’s hard to really compare GCE to AWS.

• The best price-per-performance comes with the f1-micro (which is subject to more serious fluctuations in performance; your mileage will vary at any given moment), although both the g1-small and n1-standard-1 are fairly close.

• GCE also has consistent performance across VMs as far as the single-threaded UnixBench results go. If you’re just going to hit a single CPU (and don’t have particularly high memory requirements), you should probably look at either the g1-small or the n1-standard-1, both of which get the best price-per-performance in this particular comparison.

• Keeping with the consistency theme, when GCE gives you two, four, or eight cores, you get pretty similar performance across each of the types with that number of cores (“standard,” “high memory,” or “high cpu”), with “high cpu” doing a little bit better (2-8% better in my tests), but priced significantly less than the “standard” instances with that same number of cores. So n1-highcpu-8 only outperforms n1-standard-8 by 8%, but it’s around 60% of the price.

View Larger

View Larger

(Note: The prices here are based on the “-d” instance types, which are no longer available post-beta, but are decent approximations of what it would cost to buy both the instance type and persistent storage volumes that you’ll need with GCE).

View Larger

Next: GCE-AWS Comparison

So, how does GCE stack up to AWS, at least as far as these fairly simplistic benchmarks go?

• If you’re going to buy instances on demand, don’t need the capacity of Amazon’s high-compute instances, and only care about single-core performance, GCE beats AWS as far as these benchmarks go. Amazon really seems to be giving up the very cheap end of the market (meaning the < $50/month on-demand segment) by not offering an update to the previous-generation m1.small family. The GCE g1-small is around $39/month and returned a single-core score of 891, whereas Amazon’s m1.small is around $43/month on demand and returned a single-core score of 176.

Perhaps a better comparison going forward is the GCE n1-standard-1, which is around $95/month and returned a single-core score of 1872, whereas Amazon’s new c3.large (which has the same number of cores and memory as the n1-standard-1) is around $108/month on demand, and returned a single-core score of 1524.

• When you shift to AWS reserved pricing (and here, I’m looking at the absolute cheapest option that AWS offers --three-year, heavy utilization), many of the tables are turned. Now, the c3.large is $65/month (single-core score of 1524), which is better price-per-performance (on these benchmarks) than the n1-standard-1 ($95/mo for 1872) and the n1-highcpu-2 ($117/mo for 1687). Still, GCE wins at the < $50/mo end, though, with both the f1-micro and g1-small beating AWS’s cheapest options.

View Larger

View Larger

• GCE is just starting to compete against AWS’s high-compute instances with its recently announced instance types that are in limited preview (and aren’t tested here). For now, it's probably best to think about AWS v. GCE just in terms of the AWS m3 family against GCE highmem, and the AWS c3 family against GCE standard and highcpu (the c3 instances have half as much memory as GCE standard and twice as much memory as GCE highcpu).

• As with the single-core benchmarks, GCE generally offers better price-per-performance than AWS on-demand pricing across its instance types when looking at the one-thread-per-core UnixBench results, but it’s close. Perhaps the best example of this is looking at the c3.large ($108/mo on-demand, 1867) against the n1-standard-1 ( $95/mo, 1872, same amount of memory as the c3.large) and the n1-highcpu-2 (~ $117/mo, 1916, but half as much memory as the c3.large).

[Read about Canonical's new OpenStack Interoperability Lab, which will validate software and hardware from Cisco, EMC, Intel, Juniper and other partners in "Ubuntu Lab Tests OpenStack Interoperability."]

• As with the single-core benchmarks, if you’re willing to buy reserved instances at Amazon, AWS beats GCE at price-per-performance for similar instances. The c3.large, with a three-year heavy reserved instance, averages out to close to $65/mo (UnixBench multi-threaded score of 1867), and has essentially the same score as the n1-standard-1 (1867), which is close to 50% more expensive at around $95/mo. Even if you bought the cheapest reserved c3.large with AWS (one-year light utilization), you’d still pay less than GCE, at around $81/mo.

• Looking at the high end of what GCE offers on the CPU-heavy side, the n1-highcpu-8 ($470/mo, UnixBench multi-threaded score of 5567, 7.2GB RAM) and n1-standard-8 ($763/mo, 5146, 30GB RAM) do not offer quite as much price per performance as the c3.4xlarge with a three-year heavy utilization reserved instance ($521/mo, 6924, 30GB RAM) or even with a one-year light utilization reserved instance ($646/mo).

• The high-memory side is a bit harder to compare, mainly because AWS’s new m3 family caps out at the m3.2xlarge, which compares favorably to the n1-highmem-4 (fairly equivalent with on-demand pricing; AWS offers significantly better price for performance with reserved instances, and the m3.2xlarge comes with 4GB more RAM). But even though AWS doesn’t have an m3 instance type to compete with the n1-highmem-8 ($879/mo, 5052, 52GB RAM), the new c3.8xlarge ($1,042/mo, three-year heavy, 8472, 60GB RAM) has more RAM and significantly better performance.  

View Larger

Next: Final Observations

These benchmark results show how important Amazon’s m3 and c3 families are to its competition with Google Compute Engine. When GCE was initially announced, those instance types didn't exist, and any comparison similar to this one would have overwhelmingly favored GCE. I don’t expect GCE to stay static, either -- presumably Google will be rolling out more regions and more zones, and when it does, it may take that opportunity to leapfrog Amazon or just cut prices on existing services.

Also, it’s clear that AWS is currently giving up the very low end of the IaaS market (below 15 cents/hour on demand). And for the many people using AWS solely on-demand for compute -- or who run instances for less than an hour at a time on a regular basis and would benefit significantly from per-minute pricing -- GCE offers better price for performance, accordingly to the simple benchmark runs here.

I strongly favor a multi-cloud strategy built through abstraction layers (instead of lots of vendor-specific API calls), and, as such, Google Compute Engine is solid choice for your primary or secondary IaaS vendor, provided that you can get all of the functionality you need through GCE’s services.

Next up in my IaaS performance benchmark series: Azure.