The Rise of Data-Driven Intelligence

Posted by David Hill
December 05, 2012

One of the earliest names for information technology was "data processing," which encompassed the need for both data and processing power. However, the glamour of IT for many years was in application development, where a processing- or computing-centric focus ruled the roost. From birth (creation) to death (deletion), most data remained within the control of applications. Of course, applications that analyze data after it has been created have long existed (such as business intelligence and seismic processing), but these applications were a small fraction of practical IT uses. Not any more.

Application-Driven Vs. Data-Driven Intelligence

In his book "Reinventing Discovery" (which I recommend, by the way), the author Michael Nielsen discusses data-driven intelligence and contrasts it with artificial intelligence and human intelligence. He defines data-driven intelligence as the ability of computers to extract meaning from data. He differentiates it from artificial intelligence, which he says takes tasks that humans are good at and aims to mimic or improve human performance (such as chess playing) and human intelligence (such as our ability to process visual information). According to Nielsen, data-driven intelligence complements human intelligence by solving different kinds of problems. (Big data, anyone?)

Let's examine what it means from an IT perspective. Application-driven intelligence tends to create, read, update and delete data to fulfill an initial purpose, such as a workflow process to manage order processing, shipping and payment collection. By contrast, data-driven intelligence takes existing data (human- or machine-generated) and uses it for a secondary or additional purpose, such as performing e-discovery on email files or a big data analysis that uses external information gleaned from the Web for upselling or cross-selling customers. Sensory information (such as meter reading) or machine/computer-generated information (such as logs) are created first and then analyzed by a downstream process (which may be in real-time) as appropriate.

From an IT perspective, the application development methodologies (as well as the skill sets of the developers) may be different. From an operational perspective, the service level agreements (SLAs), such as for performance and recoverability of the data, may have to be planned differently. The resources (servers, networks, storage) have to be planned differently as well. IT is familiar with application-driven intelligence-based applications, but has to learn more how to deal with data-driven intelligence applications, such as big data.

Application-Driven Vs. Data-Driven Intelligence
	Application-Driven Intelligence	Data-Driven Intelligence
Primary Goal	Substitute application intelligence for human intelligence in managing a process	Extract meaning and knowledge from data
Description	Data is created and managed to fit the needs of the application; typically, the creation of data is part of a process using the application.	The application is created and managed to fit the needs of the data, which may be (and likely are) created independent of the application
Example	Supply chain management Customer relationship management Content management Online transaction processing	Big data Data warehousing Search engine Sensor-based analysis IBM's Watson or Apple's Siri
Source: the Mesabi Group, November 2012

There is nothing new under the sun. Data-driven intelligence (such as statistical analysis using techniques like regression analysis, linear programming and simulation modeling) have been around for a long time. More recently, new concepts have emerged, including data warehousing, online analytical processing and data mining. The problem is that terms such as advanced analytics, business intelligence and big data are regarded as valuable by businesses, but existed as isolated IT islands. However, viewing these siloed (or at best overlapping) efforts and thinking of them in terms of data-driven intelligence provides a way of bringing them together to emphasize the importance of a data-centric focus.

Yes, there are hybrids. Data-driven intelligence can be inserted in an operational system, such as retail sale to check a credit card to see if it is fraudulent, or at points within a supply chain.

Data-driven intelligence is an additive view that broadens our understanding and does not replace application-driven intelligence. Let software intelligence continue to multiply and add to our understanding and the value that we derive from IT.

Next page: Thinking About Data-Driven Intelligence Applications

Related Reading

---