• 01/06/2014
    2:07 PM
  • Rating: 
    0 votes
    Vote up!
    Vote down!

IaaS Performance Benchmarks Part 7: SoftLayer

As part of my project comparing IaaS services, I tested SoftLayer, an IBM company. I also compared SoftLayer to other IaaS providers. Here are my findings.

So, how does SoftLayer compare to other providers, based on these fairly simplistic benchmarks?

• SoftLayer doesn’t really have a competitive instance type at the low end. Every other IaaS provider has a cheaper instance that performs better than the Extra Small.

• In past parts of this IaaS benchmark series, I have looked at both on-demand and discounted pricing for providers because getting discounts can often require more commitment than customers are willing to pay (e.g., three-year commitments and pre-payment). However, for SoftLayer, I’m just going to look at the discounted pricing, for two reasons.

First, SoftLayer really isn’t competitive with its on-demand pricing. SoftLayer’s virtualized instances are underwhelming compared to the competition, and SoftLayer’s on-demand pricing for its bare-metal instances is so much higher than its discounted pricing as to make it an absurd choice for hourly payment (e.g., $720/month on-demand vs. $259/month discounted). That's especially true given the launch times -- you’re not exactly going to want to take the bare-metal servers up and down a lot. Second, SoftLayer’s discounted pricing is very easy -- just pay monthly instead of hourly; it’s a fairly minimal commitment compared with other providers.

View Larger

• Looking at discounted pricing, SoftLayer’s Bare Metal instances perform fairly well as far as multi-threaded UnixBench price-per-performance go. Note that this is not really an apples-to-apples comparison with AWS and Rackspace (denoted as RS in the chart) because we’re looking at pre-payment and three-year commitments for both. AWS’s three-year heavy reserved pricing gives its C3 family an edge; GCE’s lack of discounted pricing hurts it here.

View Larger

• Focusing on high UnixBench scores: SoftLayer’s 8-core bare metal (with 8GB of RAM) is $259/month with discounted pricing, whereas AWS’s 8-core c3.2xlarge (with 15GB of RAM) is $172/month, and turns in a close UnixBench single-core score (Bare Metal 8/8: 1848; c3.2xlarge: 1752). GCE, without discounted pricing, has the n1-highcpu-8 (with 7.2GB of RAM) at $470/month, also with a similar UnixBench single-core score (1801).

• That said, if you are actually able to get the 8-core, 8GB bare-metal machines I got and only pay $0.50/hour (or, even better, $159/month), then you’ll have a fabulous deal (assuming you don’t need more RAM). And I can imagine that the actual 2-core, 2GB bare-metal offering could be fairly competitive with AWS and GCE, but unfortunately it wasn't available.

View Larger

• Looking at high multi-threaded UnixBench scores (over 5,000), SoftLayer’s bare-metal options score up with Amazon and GCE, although SoftLayer falls a bit short in apples-to-apples comparisons (or at least as close as we can get).

If you only care about CPU and don’t care about RAM much at all, then SoftLayer’s ability to configure machines specifically will give you better price-per-performance. SoftLayer’s Bare Metal 16/16, with a UnixBench multi-threaded score of 7406, is around half the cost of Amazon’s c3.8xlarge (even with three-year heavy reserved instances), which has a multi-threaded score of 8472 but comes with 108GB of RAM, not 16GB. And SoftLayer’s Bare Metal 8/8, with a multi-threaded UnixBench score of 6397, is around 75% of the cost of Amazon’s c3.4xlarge, which got a 6924,but also has 30GB of RAM.

• When you look more apples-to-apples, the c3.8xlarge beats SoftLayer’s Bare Metal 16/64 fairly handily in price-per-performance, although only Amazon has options that compare to SoftLayer here. Google’s higher-CPU offerings are still in beta, and no other provider I’ve tested has offerings with as high multi-threaded UnixBench scores as AWS.

It’s not surprising that most of the commentary about SoftLayer revolves around its bare-metal instances, and not its virtualized instances. While the ability to configure VM specifications may be nice in some cases, it’s not going to be enough to make me choose a provider when performance is behind the competition. And from my tests, SoftLayer’s VMs lag behind AWS, GCE, and Azure, and when Rackspace gets HVM virtualization for Linux live, it should also surpass SoftLayer’s VMs.

So I think the real question is: Are SoftLayer’s bare-metal instances really cloud? Interestingly, when I spoke with a SoftLayer employee to try and get my “server orders” moving (tip: once you’ve placed your order for a bare-metal instance, call up the main sales line, give them your order number, and you’ll get your servers faster), when she saw that I was ordering bare-metal instances, she said, “Oh, those take up to four hours ordinarily; it’s much faster if you get cloud ones.” This indicates to me that even within SoftLayer, there is some debate around whether the bare-metal instances really qualify as cloud. Certainly, the amount of time it takes to deploy them probably makes them inappropriate for some cloud architectures.

Perhaps the most surprising outcome to me after testing SoftLayer is that Amazon Web Services still holds the highest UnixBench multi-threaded scores and GCE still holds the highest UnixBench single-core scores I’ve gotten so far. It may be that the infrastructure and difficulty required to do “bare-metal cloud” is ultimately not worth it, as cloud providers can deliver similar performance through virtualizing beefier chips. Or perhaps my simplistic benchmarks here fail to capture enough of the benefits that customers get with bare-metal offerings (e.g., I/O, latency).

At any rate, SoftLayer does seem to be waging a slightly different battle than AWS or GCE. In particular, perhaps it's taking advantage of the fact that most cloud architectures likely consist of machines that aren’t ephemeral, and that auto-scaling with really powerful machines may be simply a nice-to-have feature for most customers.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.