Log Analysis: Untapped Network Intelligence

Don't overlook log data as a resource for detecting network problems and improved network planning.

Vess Bakalov

August 7, 2015

3 Min Read
Network Computing logo

More and more, I'm seeing networking and IT teams latching onto log data as a way to proactively monitor their complex digital networks and infrastructures. These groups are operationalizing log data for advanced warning of potential application or service disruption, as opposed to after-the-fact forensic analysis. The main reason that continues to pop up -- they are tired of having to know exactly what problems to look for in order to frame search queries.

I recently sat down with Eric Sharpsten, CTO of a large government contracting group, to discuss his thoughts on integrating log data and performance metrics.

"If I can get [log] data and marry it up with other data that drives the performance metrics, that's really powerful," Sharpsten said. He noted one of the biggest advances in log data use is the ability to get performance information quickly, allowing a swift recovery process after an outage or event.

Shifting views

Leveraging log data has received mixed reviews in the past, mainly because of the specialized nature of system logs, which were "owned" by separate groups, like security or administration. This isolation made it difficult to quickly access different data sets and correlate various types of log data with each other and with higher-level performance metrics.

These days, the implementation of log data is yielding better performance results for networking and IT groups for a couple reasons. There's a shift toward focusing more on a subset of specific log data to store and retain long term, rather than just indiscriminately storing all the data all the time. Networking and IT staff are also recognizing the importance of leveraging log data as a tool for network troubleshooting and proactive monitoring and planning.

In fact, a recent Enterprise Management Associates report highlighted this fundamental shift. According to the firm's survey, 47% of participants said log data is one of the most important tools in network management.

The obstacles

The popularity of log tools is a testament to their value. But search remains an after-the-fact process, used most often in troubleshooting to find a misconfiguration or a protocol mismatch, or some failure that has already occurred. Splunk, for instance, is primarily a forensics tool designed to search, monitor and analyze machine data after a performance event has already disrupted application or service delivery.

Perhaps that is why just over 50% of EMA's survey respondents said that "knowing what to look for" has hindered their decision to rely on log data. Another 36% said they found "writing new scripts or filters to find what is important" most daunting.

The reality is that tools, technologies and thinking based on legacy models still cultivate anxiety and delays in utilizing log data. Three things are needed to ensure log analytics are viable and valuable:

  • Networking and IT staff need to be able to have an understanding, first, of normal log behavior through baselining.

  • They need the ability to send near real-time alerts when abnormalities appear.

  • And they need to correlate real-time network and infrastructure performance metrics, flows and logs at scale. 

New technologies can solidify log data's leap from a reactive tool to a proactive remedy. Advanced indexing, metadata tagging, searching, analytics and reporting dramatically simplify the work of navigating a wealth of current data in historical context. Formerly isolated event data can be used as a real-time operations resource to uncover the causes of problems, and track how device changes affect digital network and infrastructure performance. 

About the Author(s)

Vess Bakalov

CTO & Co-Founder, SevOneA co-founder of SevOne, Vess Bakalov leads the company's product development, design, and overall vision of the technology architecture. Vess created SevOne with the future of infrastructures in mind after working with legacy solutions during his time as a network architect at BankOne. Vess's favorite word is "basically" because it forces people to listen, knowing what he's about to say at that very moment is the simplest way possible and with no jargon. He has a true passion for optimization, which is reflected in both his technological and business decisions. Vess graduated from the University of Delaware with a degree in Computer Science. He spends his spare time gaming, snowboarding, and traveling with his family.

Stay informed! Sign up to get expert advice and insight delivered direct to your inbox

You May Also Like

More Insights