IaaS Performance Benchmarks Part 7: SoftLayer

This is the seventh part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained my methodology for testing instance types across major IaaS providers. I've run benchmarks for Amazon Web Services, Google Compute Engine, Microsoft Azure Virtual Machines, and Rackspace Cloud Servers. In this part, I look at SoftLayer, which was acquired by IBM last year.

SoftLayer is unique among the IaaS providers I'm testing in that it has bare-metal (meaning not virtualized) pay-by-the-hour services that some industry observers think may give the company a significant edge.

As of today, there are six U.S. regions available for SoftLayer through RightScale (the cloud management service I’m using to run most of my benchmarks, as it gives me image parity): San Jose, Seattle, Washington, D.C., and three in Dallas (Dallas 1, Dallas 5, and Dallas 6). SoftLayer offers extremely configurable servers (far more so than the other major IaaS providers, which offer specific “instance types”), but RightScale has created five instance types for SoftLayer. I tested those, so I launched a total of 30 VMs through RightScale.

In addition, I also launched seven different bare-metal instances through SoftLayer’s own control panel since RightScale doesn't yet support SoftLayer's bare-metal servers. Consequently these aren’t as apples-to-apples as staying within RightScale, but I did run CentOS 6.4 on each of them, and I think the results are valid enough to help spur further investigation.

SoftLayer has what appears to be the simplest discounting system -- much better than AWS’s Reserved Instances or Rackspace’s commitment plus prepayment discounts. SoftLayer offers discounts from roughly 5% to 30% if you purchase servers by the month instead of by the hour.

Bare-Metal Launching

As I’ll detail below, SoftLayer’s bare-metal performance looks pretty solid. However, launching bare-metal instances was a different experience than launching virtualized instances. Most of the virtualized instances came up in under 10 minutes, but the bare-metal instances took at least an hour, and in most cases, closer to five hours from order to operational. At least two of my bare-metal instances required human intervention; they were apparently launched running CentOS 6.3, but required upgrading to CentOS 6.4 before they could be made operational. (Never mind that the “order form” for them only allowed me to choose CentOS 6.0).

Also, I was unable to get any bare-metal instance with fewer than eight cores and 8GB of RAM. I ordered 2-core/2GB RAM instances on five separate occasions in different data centers and got 8-core/8GB machines every time. However, SoftLayer assured me that I would be charged the 2-core/2GB price for these, and I was -- only 50 cents/hour.

On one hand, SoftLayer’s bare-metal instances can be launched via the same API as its virtualized instances, and you can pay for them by the hour, both of which make them quite cloud-like. On the other hand, I’m not exactly sure what all is required of SoftLayer to launch or terminate bare-metal instances, but my impression is that some kind of human intervention is required for them. You’ll have to decide for yourself whether they meet your definition of cloud computing.

Here are the specifications of the instances I launched with SoftLayer:

SoftLayer Benchmark Results

Here are the main takeaways that I have from testing across SoftLayer:

•SoftLayer has the biggest differences in performance between regions that I’ve seen. The Washington, D.C. region was underwhelming to say the least, and Dallas 5 and 6 were significantly better than the rest. Washington, D.C. had the worst or second-worst score for each instance type, and Dallas 5 and 6 were in the top three regions for every instance type, except for Extra Small.

•Like other clouds, the best price-per-performance for single-core UnixBench is the cheapest one; here, it's the Extra Small. However, unlike other clouds, the bare-metal instances actually have similar price-per-performance to the Extra Small. (And I would suspect that the bare-metal instance types with fewer cores would do even better, but as stated above, when I tried to order one with two cores, I ended up with an instance with eight cores).

[IBM says it's devised a method for avoiding cloud performance problems by dynamically managing network bandwidth in cloud computing environments. Get the details in "IBM Invention Aims To Fix Cloud Bottlenecks."]

•If the bare-metal instances are “cloud” enough for you, there are limited use cases for running virtual instances with SoftLayer (e.g. cheap instances, ephemeral instances, lots of RAM), since the performance difference is dramatic. The bare-metal instances are so much better in terms of price-per-performance that your need for the virtual instances will be limited. That said, if you buy SoftLayer for the bare metal, it will be nice to have the option to have the virtualized instances within the same networks/data centers within minutes.

NEXT: Comparing Soft layer to Other IaaS Providers

So, how does SoftLayer compare to other providers, based on these fairly simplistic benchmarks?

•SoftLayer doesn’t really have a competitive instance type at the low end. Every other IaaS provider has a cheaper instance that performs better than the Extra Small.

•In past parts of this IaaS benchmark series, I have looked at both on-demand and discounted pricing for providers because getting discounts can often require more commitment than customers are willing to pay (e.g., three-year commitments and pre-payment). However, for SoftLayer, I’m just going to look at the discounted pricing, for two reasons.

First, SoftLayer really isn’t competitive with its on-demand pricing. SoftLayer’s virtualized instances are underwhelming compared to the competition, and SoftLayer’s on-demand pricing for its bare-metal instances is so much higher than its discounted pricing as to make it an absurd choice for hourly payment (e.g., $720/month on-demand vs. $259/month discounted). That's especially true given the launch times -- you’re not exactly going to want to take the bare-metal servers up and down a lot. Second, SoftLayer’s discounted pricing is very easy -- just pay monthly instead of hourly; it’s a fairly minimal commitment compared with other providers.

•Looking at discounted pricing, SoftLayer’s Bare Metal instances perform fairly well as far as multi-threaded UnixBench price-per-performance go. Note that this is not really an apples-to-apples comparison with AWS and Rackspace (denoted as RS in the chart) because we’re looking at pre-payment and three-year commitments for both. AWS’s three-year heavy reserved pricing gives its C3 family an edge; GCE’s lack of discounted pricing hurts it here.

•Focusing on high UnixBench scores: SoftLayer’s 8-core bare metal (with 8GB of RAM) is $259/month with discounted pricing, whereas AWS’s 8-core c3.2xlarge (with 15GB of RAM) is $172/month, and turns in a close UnixBench single-core score (Bare Metal 8/8: 1848; c3.2xlarge: 1752). GCE, without discounted pricing, has the n1-highcpu-8 (with 7.2GB of RAM) at $470/month, also with a similar UnixBench single-core score (1801).

•That said, if you are actually able to get the 8-core, 8GB bare-metal machines I got and only pay $0.50/hour (or, even better, $159/month), then you’ll have a fabulous deal (assuming you don’t need more RAM). And I can imagine that the actual 2-core, 2GB bare-metal offering could be fairly competitive with AWS and GCE, but unfortunately it wasn't available.

•Looking at high multi-threaded UnixBench scores (over 5,000), SoftLayer’s bare-metal options score up with Amazon and GCE, although SoftLayer falls a bit short in apples-to-apples comparisons (or at least as close as we can get).

If you only care about CPU and don’t care about RAM much at all, then SoftLayer’s ability to configure machines specifically will give you better price-per-performance. SoftLayer’s Bare Metal 16/16, with a UnixBench multi-threaded score of 7406, is around half the cost of Amazon’s c3.8xlarge (even with three-year heavy reserved instances), which has a multi-threaded score of 8472 but comes with 108GB of RAM, not 16GB. And SoftLayer’s Bare Metal 8/8, with a multi-threaded UnixBench score of 6397, is around 75% of the cost of Amazon’s c3.4xlarge, which got a 6924,but also has 30GB of RAM.

•When you look more apples-to-apples, the c3.8xlarge beats SoftLayer’s Bare Metal 16/64 fairly handily in price-per-performance, although only Amazon has options that compare to SoftLayer here. Google’s higher-CPU offerings are still in beta, and no other provider I’ve tested has offerings with as high multi-threaded UnixBench scores as AWS.

It’s not surprising that most of the commentary about SoftLayer revolves around its bare-metal instances, and not its virtualized instances. While the ability to configure VM specifications may be nice in some cases, it’s not going to be enough to make me choose a provider when performance is behind the competition. And from my tests, SoftLayer’s VMs lag behind AWS, GCE, and Azure, and when Rackspace gets HVM virtualization for Linux live, it should also surpass SoftLayer’s VMs.

So I think the real question is: Are SoftLayer’s bare-metal instances really cloud? Interestingly, when I spoke with a SoftLayer employee to try and get my “server orders” moving (tip: once you’ve placed your order for a bare-metal instance, call up the main sales line, give them your order number, and you’ll get your servers faster), when she saw that I was ordering bare-metal instances, she said, “Oh, those take up to four hours ordinarily; it’s much faster if you get cloud ones.” This indicates to me that even within SoftLayer, there is some debate around whether the bare-metal instances really qualify as cloud. Certainly, the amount of time it takes to deploy them probably makes them inappropriate for some cloud architectures.

Perhaps the most surprising outcome to me after testing SoftLayer is that Amazon Web Services still holds the highest UnixBench multi-threaded scores and GCE still holds the highest UnixBench single-core scores I’ve gotten so far. It may be that the infrastructure and difficulty required to do “bare-metal cloud” is ultimately not worth it, as cloud providers can deliver similar performance through virtualizing beefier chips. Or perhaps my simplistic benchmarks here fail to capture enough of the benefits that customers get with bare-metal offerings (e.g., I/O, latency).

At any rate, SoftLayer does seem to be waging a slightly different battle than AWS or GCE. In particular, perhaps it's taking advantage of the fact that most cloud architectures likely consist of machines that aren’t ephemeral, and that auto-scaling with really powerful machines may be simply a nice-to-have feature for most customers.