Cloud Infrastructure

10:00 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

How DigitalGlobe Handles 2 Petabytes Of Satellite Data Yearly

DigitalGlobe's soon-to-be six satellites gather data at a freakish pace -- capturing images of everything from natural disasters to shipping containers in ports.

DigitalGlobe's soon-to-be six satellites orbiting the globe are amassing image data at a freakish pace. Its archive will be able to offer new services with that data, provided it can maintain the collection, storage, indexing, and service-delivery infrastructure needed to support it.

DigitalGlobe offers image services to forestry agencies, mining companies, government crop-estimating experts, and other organizations seeking information on everything from the number of cars in a competitor's parking lot to the number of shipping containers in port.

The Longmont, Colo., firm has different types of satellites in orbit, some collecting color images, some black and white. It applies different software to the resulting data -- for example, stretching upward when it's viewing the Himalayas and other mountain ranges, and chopping the shadows off tall buildings lest the image misrepresent their true height.

But one of its biggest tasks is data collection. Its five satellites (number six, WorldView-3, will be launched in early 2014) are scanning 3.3 million kilometers of earth imagery per day. The information is downloaded as the satellites pass over the equator, over the poles and points in between.

[DigitalGlobe was involved in the 2007 oceanic search for computer researcher Jim Gray. To learn more, see Remembering The Search For Jim Gray, A Year Later.]

Most collection points can be hardwired into a network. But the satellite collection dish at the South Pole has to beam the data to a communications satellite, which in turn relays it to the DigitalGlobe datacenter near its headquarters in Colorado. All in all, it collects an average of two petabytes of data annually. The firm maintains 40 PBs on spinning disks and 20 PBs on tape, with a copy in a second datacenter.

"Our archive goes back 7-8 years now. We can show changes on the earth's surface over time. We can show how things like the Peterman Glacier in Greenland or the Amazon rain forest looked years before, versus the way they look now," said Scott Hicar, CIO and senior VP of global technology services, in an interview.

This ability may lend itself to future projects in climate research or planning in the face of habitat changes in different parts of the planet. One satellite in DigitalGlobe's "constellation," QuickBird, can take color imagery, which is useful in agriculture, such as in assessing the damaging effects of drought on particular crops.

But to build modern services on its wealth of data, it's had to build out more of a private cloud infrastructure. Each datacenter has a high-performance, x86 server cluster consisting of "several thousand servers" running Red Hat Enterprise Linux, Hicar said. It has virtualized and manages the clusters with VMware vSphere software and uses IBM Moab software for job scheduling. It analyzes images and creates indexes of its vast library using Oracle. Services based on the clusters are monitored by CA Technologies Nimsoft, which identifies any hiccups or service slowdowns. The firm uses SAP ERP for all the information needed to keep satellites aloft.

DigitalGlobe recently established a FirstLook service that can zero in on a particular part of the world to generate up-to-date imagery to meet a customer's need. Take disaster first responders: DigitalGlobe offered images showing damage from the tsunami that washed over the Fukushima nuclear power plant in Japan and the Haiyan typhoon that hit the Philippines.

Emergency planners and disaster-relief organizations need such information to know how hard hit an area is, where they can stage relief efforts, and what supplies and equipment are needed. In addition to images, DigitalGlobe can also supply its Analysis Reports service, which provides detailed feedback on the image. When the Sakurajima volcano erupted in Japan in 2012, DigitalGlobe provided imagery and analysis of the size and rate of spread of its plume.

Satellite imaging can also meet more-routine needs. The Thio town council, a citizen body governing a remote area of Pacific island New Caledonia, contracted for satellite imagery and analysis to tell it where nickel mine waste and sediment were infiltrating its main drainage, the Thio River. In Africa, Namibia measures the effectiveness of mine rehabilitation efforts through satellite imagery.

Other governments want coastal mapping to check on changes to low-lying areas. Marine biologists want to map the expansion or regression of barely visible coral reefs.

Automotive geo-positioning systems and truck fleet management are among the more recent users of its services.

DigitalGlobe's new WorldView-3 satellite will launch in 2014.
DigitalGlobe's new WorldView-3 satellite will launch in 2014.

Hicar said DigitalGlobe will be able to keep expanding the types of services it offers as it increases the sophistication of analytics used in building its reports. The new WorldView-3 satellite will be more sensitive and better equipped than its black-and-white image-collecting predecessors. It will be a high-resolution, "super-spectral" data collector, able to see beyond the bands of visible light into infrared and other parts of the spectrum.

Web-based services such as FirstLook, accessible to any customer equipped with a desktop computer, are a sample of what's to come from satellite imaging services, Hicar said. It is already used by Microsoft's Bing search engine in supplying maps and as a data platform for the Esri geo-positioning system company. What is likely to follow will be increasingly sophisticated services for smartphones and other mobile devices, as smartphone suppliers and application builders realize what can be done with the satellite data, Hicar said.

The ability to highlight attractions of an area and specific landmarks would lend itself to travel applications. Direction-finding systems might benefit from showing what a destination looks like in an image. "Mining, forestry, natural resources, there are many uses," Hicar predicted.

Charles Babcock is an editor-at-large for InformationWeek, having joined the publication in 2003. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive Week.

Too many companies treat digital and mobile strategies as pet projects. Here are four ideas to shake up your company. Also in the Digital Disruption issue of InformationWeek: Six enduring truths about selecting enterprise software. (Free registration required.)

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
WhoaGuy
100%
0%
WhoaGuy,
User Rank: Apprentice
1/21/2014 | 4:03:58 PM
Re: Storing data cost more than revinue
They use plenty of other technology, and I don't know where your getting your operating cost from...?
RaghuK136
0%
100%
RaghuK136,
User Rank: Apprentice
1/5/2014 | 3:57:04 PM
Storing data cost more than revinue
Amazing every day digitalglobe stores 2000 TB which cost 1 million approx,Why dont they use other technology.
cbabcock
50%
50%
cbabcock,
User Rank: Apprentice
1/3/2014 | 4:10:56 PM
DigitalGlobe collects 2 PBs of image data per year
This story was corrected Dec. 31 to reflect that I missed in my discussionwith DigitalGlobe that the 2 PBs was collected annually, not daily. All that daily satellite imagery amounts to a lot of data, 3.3 million kilometers of the earth's surface. Nevetheless, the 2 PBs is an annual figure. 
WKash
50%
50%
WKash,
User Rank: Apprentice
12/31/2013 | 2:03:28 PM
Re: Lot's of raw data; more analysis in the future
It's becoming almost impossible to comprehend, but 2 Petabytes of data is equal to 40 million 4-drawer file cabinets filled with text.  If you put those file cabinets side by side, they'd span nearly 9,500 miles. One can only imagine what it takes to catalog and analyze all of it each day. 

And consider this:  That's still only one-tenth of what Google reportedly processes each day.
asksqn
50%
50%
asksqn,
User Rank: Apprentice
12/31/2013 | 2:23:29 AM
Here's proof of a changing climate
It is fascinating how even with petrabytes of data available to compare how much the icecaps have melted over the years that there are still those in denial of climate change.
Laurianne
50%
50%
Laurianne,
User Rank: Apprentice
12/30/2013 | 4:04:56 PM
Re: Lot's of raw data; more analysis in the future
DigitalGlobe sounds like yet another interesting customer private cloud story for Red Hat.
cbabcock
50%
50%
cbabcock,
User Rank: Apprentice
12/30/2013 | 12:29:18 PM
Lot's of raw data; more analysis in the future
That's right, they collect 2 PBs a day; they store something less than that through compression. DigitalGlobe has 4 billion square kilometers on archive, or 27X the surface of the earth. The raw data is one thing, what we will be able to do with it in the future is another. It will be addressed by increasingly sophisticated analysis systems.
RobPreston
50%
50%
RobPreston,
User Rank: Apprentice
12/30/2013 | 11:27:35 AM
Big Data?
It collects 2 petabytes of data a day. For those who say the big data movement is mostly hype, this is a reminder that it's absolutely core to some companies' business models. 
Slideshows
Cartoon
Audio Interviews
Archived Audio Interviews
Jeremy Schulman, founder of Schprockits, a network automation startup operating in stealth mode, joins us to explore whether networking professionals all need to learn programming in order to remain employed.
White Papers
Register for Network Computing Newsletters
Current Issue
2014 Private Cloud Survey
2014 Private Cloud Survey
Respondents are on a roll: 53% brought their private clouds from concept to production in less than one year, and 60% extend their clouds across multiple datacenters. But expertise is scarce, with 51% saying acquiring skilled employees is a roadblock.
Video
Twitter Feed