How DigitalGlobe Handles 2 Petabytes Of Satellite Data Yearly

DigitalGlobe's soon-to-be six satellites orbiting the globe are amassing image data at a freakish pace. Its archive will be able to offer new services with that data, provided it can maintain the collection, storage, indexing, and service-delivery infrastructure needed to support it.

DigitalGlobe offers image services to forestry agencies, mining companies, government crop-estimating experts, and other organizations seeking information on everything from the number of cars in a competitor's parking lot to the number of shipping containers in port.

The Longmont, Colo., firm has different types of satellites in orbit, some collecting color images, some black and white. It applies different software to the resulting data -- for example, stretching upward when it's viewing the Himalayas and other mountain ranges, and chopping the shadows off tall buildings lest the image misrepresent their true height.

But one of its biggest tasks is data collection. Its five satellites (number six, WorldView-3, will be launched in early 2014) are scanning 3.3 million kilometers of earth imagery per day. The information is downloaded as the satellites pass over the equator, over the poles and points in between.

[DigitalGlobe was involved in the 2007 oceanic search for computer researcher Jim Gray. To learn more, see Remembering The Search For Jim Gray, A Year Later.]

Most collection points can be hardwired into a network. But the satellite collection dish at the South Pole has to beam the data to a communications satellite, which in turn relays it to the DigitalGlobe datacenter near its headquarters in Colorado. All in all, it collects an average of two petabytes of data annually. The firm maintains 40 PBs on spinning disks and 20 PBs on tape, with a copy in a second datacenter.

"Our archive goes back 7-8 years now. We can show changes on the earth's surface over time. We can show how things like the Peterman Glacier in Greenland or the Amazon rain forest looked years before, versus the way they look now," said Scott Hicar, CIO and senior VP of global technology services, in an interview.

This ability may lend itself to future projects in climate research or planning in the face of habitat changes in different parts of the planet. One satellite in DigitalGlobe's "constellation," QuickBird, can take color imagery, which is useful in agriculture, such as in assessing the damaging effects of drought on particular crops.

But to build modern services on its wealth of data, it's had to build out more of a private cloud infrastructure. Each datacenter has a high-performance, x86 server cluster consisting of "several thousand servers" running Red Hat Enterprise Linux, Hicar said. It has virtualized and manages the clusters with VMware vSphere software and uses IBM Moab software for job scheduling. It analyzes images and creates indexes of its vast library using Oracle. Services based on the clusters are monitored by CA Technologies Nimsoft, which identifies any hiccups or service slowdowns. The firm uses SAP ERP for all the information needed to keep satellites aloft.

DigitalGlobe recently established a FirstLook service that can zero in on a particular part of the world to generate up-to-date imagery to meet a customer's need. Take disaster first responders: DigitalGlobe offered images showing damage from the tsunami that washed over the Fukushima nuclear power plant in Japan and the Haiyan typhoon that hit the Philippines.

Emergency planners and disaster-relief organizations need such information to know how hard hit an area is, where they can stage relief efforts, and what supplies and equipment are needed. In addition to images, DigitalGlobe can also supply its Analysis Reports service, which provides detailed feedback on the image. When the Sakurajima volcano erupted in Japan in 2012, DigitalGlobe provided imagery and analysis of the size and rate of spread of its plume.

Satellite imaging can also meet more-routine needs. The Thio town council, a citizen body governing a remote area of Pacific island New Caledonia, contracted for satellite imagery and analysis to tell it where nickel mine waste and sediment were infiltrating its main drainage, the Thio River. In Africa, Namibia measures the effectiveness of mine rehabilitation efforts through satellite imagery.

Other governments want coastal mapping to check on changes to low-lying areas. Marine biologists want to map the expansion or regression of barely visible coral reefs.

Automotive geo-positioning systems and truck fleet management are among the more recent users of its services.

Figure 1: DigitalGlobe's new WorldView-3 satellite will launch in 2014.

Hicar said DigitalGlobe will be able to keep expanding the types of services it offers as it increases the sophistication of analytics used in building its reports. The new WorldView-3 satellite will be more sensitive and better equipped than its black-and-white image-collecting predecessors. It will be a high-resolution, "super-spectral" data collector, able to see beyond the bands of visible light into infrared and other parts of the spectrum.

Web-based services such as FirstLook, accessible to any customer equipped with a desktop computer, are a sample of what's to come from satellite imaging services, Hicar said. It is already used by Microsoft's Bing search engine in supplying maps and as a data platform for the Esri geo-positioning system company. What is likely to follow will be increasingly sophisticated services for smartphones and other mobile devices, as smartphone suppliers and application builders realize what can be done with the satellite data, Hicar said.

The ability to highlight attractions of an area and specific landmarks would lend itself to travel applications. Direction-finding systems might benefit from showing what a destination looks like in an image. "Mining, forestry, natural resources, there are many uses," Hicar predicted.

Charles Babcock is an editor-at-large for InformationWeek, having joined the publication in 2003. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive Week.

Too many companies treat digital and mobile strategies as pet projects. Here are four ideas to shake up your company. Also in the Digital Disruption issue of InformationWeek: Six enduring truths about selecting enterprise software. (Free registration required.)