The Rise of Data-Driven Intelligence

One of the earliest names for information technology was "data processing," which encompassed the need for both data and processing power. However, the glamour of IT for many years was in application development, where a processing- or computing-centric focus ruled the roost. From birth (creation) to death (deletion), most data remained within the control of applications. Of course, applications that analyze data after it has been created have long existed (such as business intelligence and seismic processing), but these applications were a small fraction of practical IT uses. Not any more.

Application-Driven Vs. Data-Driven Intelligence

In his book "Reinventing Discovery" (which I recommend, by the way), the author Michael Nielsen discusses data-driven intelligence and contrasts it with artificial intelligence and human intelligence. He defines data-driven intelligence as the ability of computers to extract meaning from data. He differentiates it from artificial intelligence, which he says takes tasks that humans are good at and aims to mimic or improve human performance (such as chess playing) and human intelligence (such as our ability to process visual information). According to Nielsen, data-driven intelligence complements human intelligence by solving different kinds of problems. (Big data, anyone?)

Let's examine what it means from an IT perspective. Application-driven intelligence tends to create, read, update and delete data to fulfill an initial purpose, such as a workflow process to manage order processing, shipping and payment collection. By contrast, data-driven intelligence takes existing data (human- or machine-generated) and uses it for a secondary or additional purpose, such as performing e-discovery on email files or a big data analysis that uses external information gleaned from the Web for upselling or cross-selling customers. Sensory information (such as meter reading) or machine/computer-generated information (such as logs) are created first and then analyzed by a downstream process (which may be in real-time) as appropriate.

From an IT perspective, the application development methodologies (as well as the skill sets of the developers) may be different. From an operational perspective, the service level agreements (SLAs), such as for performance and recoverability of the data, may have to be planned differently. The resources (servers, networks, storage) have to be planned differently as well. IT is familiar with application-driven intelligence-based applications, but has to learn more how to deal with data-driven intelligence applications, such as big data.

There is nothing new under the sun. Data-driven intelligence (such as statistical analysis using techniques like regression analysis, linear programming and simulation modeling) have been around for a long time. More recently, new concepts have emerged, including data warehousing, online analytical processing and data mining. The problem is that terms such as advanced analytics, business intelligence and big data are regarded as valuable by businesses, but existed as isolated IT islands. However, viewing these siloed (or at best overlapping) efforts and thinking of them in terms of data-driven intelligence provides a way of bringing them together to emphasize the importance of a data-centric focus.

Yes, there are hybrids. Data-driven intelligence can be inserted in an operational system, such as retail sale to check a credit card to see if it is fraudulent, or at points within a supply chain.

Data-driven intelligence is an additive view that broadens our understanding and does not replace application-driven intelligence. Let software intelligence continue to multiply and add to our understanding and the value that we derive from IT.

Next page: Thinking About Data-Driven Intelligence ApplicationsBeing able to distinguish between an application-driven intelligence solution and a data-driven intelligence application is important because the development methodology is different. Although both can use agile development methods, there are a number of key differences. For example, Ken Collier in his book Agile Analytics discusses the difference between agile development for data warehousing and business intelligence and that of traditional applications. All in all, you have to know what is different (skill sets, methodology, resources, time frame) for building or using an application-driven intelligence solution vs. a data-driven intelligence application.

Operationally, SLA decisions and IT infrastructure decisions for data-driven intelligence applications probably differ from application-driven intelligence applications. For example, current customer orders must be protected at all cost, so significant data protection, including strong disaster recovery, needs to be in place. However, where external Web data is ingested internally for a big data analysis, it may not make any sense to provide a high level of data protection as the data could simply be re-ingested from its original sources.

You don't have to worry about trying to fit a project into a particular definition. Big data is a hot topic, but what is it exactly? Doug Laney (now of Gartner) introduced the popular concept of volume, variety and velocity. This is a very powerful and useful idea, but it does not precisely define what is big and what is little. Moreover, size alone does not determine value. Using a data-driven intelligence approach causes you to think about its overall value and the software technology that needs to be applied. If the project seems to fit the bill of big data, then call it that. If it does not but still delivers value, go ahead. Be benefit-driven, not label-driven.

Recognize that collectively data-driven intelligence is the engine that makes more data-centric IT possible. This collective perspective encompasses all the pieces and gives a better sense of the total value that results when viewing the world through a data-driven intelligence lens.

Mesabi Musings

This is a short introduction to a broad subject and will require further discussion both from a general perspective as well as using specific product illustrations. Now, application-driven intelligence tends to focus on operational business processes. Data-driven intelligence tends to fulfill the needs of management information systems.

The big difference is that the proper use of data-driven intelligence will help organizations switch from intuition-based to data-based decisions--a transformation that should have a positive effect at all levels of an organization.