Cloud Infrastructure

07:00 AM
Howard Marks
Howard Marks
Commentary
Connect Directly
Twitter
RSS
E-Mail
100%
0%

Code Spaces: A Lesson In Cloud Backup

The SaaS provider is out of business after an AWS security breach, underscoring the need for proper cloud storage backup.

Over the past few years, I’ve been getting that foreboding feeling when I listen to the clouderati natter on about how the cloud is the solution to all your IT problems. If, they argue, you’d just put all your applications in the cloud you’d never have to back up a server again. Unfortunately, the folks that ran Code Spaces drank their own Flavor Aid and are now joining Monty Python’s Norwegian Blue parrot in pining for the fjords. 

The story of Code Spaces' death is a relatively simple one. It set up shop as a SaaS provider to software developers offering source code management tools like Git and Apache Subversion, all on Amazon Web Services (AWS). As long as things were good at AWS, things were OK at Code Spaces as well.

Then the hacker appeared. The hacker somehow gained access to Code Spaces' AWS control panel account, then got the folks at Code Spaces’ attention by mounting a denial-of-service attack and delivering a ransom note through the AWS control panel.

The Code Spaces folks responded as geeks everywhere would: They started nailing down their control panel, changing passwords and the like. However, the hacker was watching them via a backdoor account he created, and when he saw them locking doors, he set off the metaphorical bomb he’d hidden in the basement and started deleting AWS objects like EC2 machine definitions, S3 buckets, and, of course, all the backups. 

As you can see in the message at www.codespaces.com, that was enough to kill Codes Spaces as a going concern. At the moment, the Code Spaces crew is doing what I’m sure is yeomen’s work recovering what can be recovered for their customers. Then it’s resumé writing time.

What can we learn from this sad, sad story? Several things. For one thing, there was a failure of security. A company as dependent on AWS as Code Spaces should have instituted some basic security options that would have made deleting its primary data and backups a lot harder.

The first security option is multi-factor authentication. If you can afford the cloud service, you can afford the extra few dollars a month to require two-factor authentication with a key fob. Code Sources doesn’t know who the hacker was; if access to its accounts required a key fob, the company would know whose fob was being used.

Speaking of which, isn’t there, or shouldn’t there be, the equivalent of an Emergency Power Off (EPO) for AWS management? I mean a button that would prevent any object state changes, somewhat like ESX’s maintenance mode, until the cloud provider can really verify the identity of an administrator?

What Code Spaces apparently didn’t do was implement simple role-based management: Rather than give one user account full administration to the AWS infrastructure, create several different accounts. After all, deleting a backup is, or should be, a rare occurrence. If backup deletion is limited to a specific account, and the key fob for that account is kept in the office safe, the backups would have been safer.

However, I think it’s narrow minded to view this event as strictly a security failure. The Code Spaces folks were also guilty of violating the backup administrator’s rule of 3-2-1, which requires three copies of any piece of data, on two different media with one offsite.

By storing their primary and backup copies of data on AWS, they failed to be on two different types of media and be in two locations. Some could argue that S3 in multiple regions satisfies the requirement of two locations, but as long as it’s one console, I wouldn’t count it as offsite. That's because it remains vulnerable to this type of administrator identity attack, which is, frankly, more likely to be a disgruntled employee/ex-employee than a hacker.

The backup of your cloud environment should be at very least to cloud storage managed by another account that shares nothing, except possibly billing information, with the primary account. Sending those backups to a completely separate service would make me sleep even better at night. Corporate IT has to still be responsible for keeping data safe, even if it’s been sent to the cloud. 

Howard Marks is founder and chief scientist at Deepstorage LLC, a storage consultancy and independent test lab based in Santa Fe, N.M. and concentrating on storage and data center networking. In more than 25 years of consulting, Marks has designed and implemented storage ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
MarciaNWC
50%
50%
MarciaNWC,
User Rank: Strategist
7/3/2014 | 5:20:03 PM
Re: harsh lesson
Ars Technica found a cached copy of the Code Spaces website that shows the company actually boasted of its data recovery ability: http://arstechnica.com/security/2014/06/aws-console-breach-leads-to-demise-of-service-with-proven-backup-plan/

 

 

 
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Apprentice
7/3/2014 | 2:54:28 PM
Two different AWS regions don't count when used for security
Two different AWS regions may count as "offsite" for disaster recovery purposes, but it doesn't work for cloud security: both regions visible and accessible on the management console. That's an excellent point. Amazon probably doesn't want an Emergency Off Button; it doesn't really take responsibility for the operation of a customer workload. But, if left optional to the customer, it's a good idea and could be imposed as an automated service. 
MarciaNWC
50%
50%
MarciaNWC,
User Rank: Strategist
7/3/2014 | 1:19:28 PM
harsh lesson
That Code Spaces went out of business as a result of this hacker attack is rather sad, but putting all their eggs into the AWS basket seems like an obviously bad decision. Hopefully other organizations can learn from Code Spaces' mistakes.
Hot Topics
13
Fall IT Events: On The Road Again With 10 Top Picks
James M. Connolly, Editor in Chief, The Enterprise Cloud Site,  7/29/2014
5
25 GbE: A Big Deal That Will Arrive
Greg Ferro, Network Architect & Blogger,  7/29/2014
3
Cisco Ships APIC Controller, Reveals ACI Pricing
Marcia Savage, Managing Editor, Network Computing,  7/29/2014
White Papers
Register for Network Computing Newsletters
Cartoon
Current Issue
2014 Private Cloud Survey
2014 Private Cloud Survey
Respondents are on a roll: 53% brought their private clouds from concept to production in less than one year, and 60% ­extend their clouds across multiple datacenters. But expertise is scarce, with 51% saying acquiring skilled employees is a roadblock.
Video
Slideshows
Twitter Feed