Rollout: Splunk's Splunk Server 2.1 Beta

UPDATED November 8, 2006

You've Probably Turned To Log Files to identify a problem's source before. But which log file do you look in, and what format is that log file in? Manually searching millions of lines of data just isn't feasible. Splunk 2.1 speeds the task by using keywords to let you quickly search through all your enterprise's logs.

Splunk is available for download for just about any platform except Microsoft Windows. Currently, the Splunk server can be installed on FreeBSD, Linux, MacOS X and Sun Solaris machines. The vendor is working on a Windows version, but does not have a timetable for delivery. Competing products lock you into a specific OS. LogLogic, for example, is available only as a turnkey device and Tenable's Log Correlation Engine only runs in Red Hat Enterprise.Splunk integrates itself into almost any environment. Like its competition, the Splunk server reads logs from a wide range of sources, including sendmail, Linux authentication, Web servers, Snort, Cisco firewall and Windows event logs (with the help of Snare). However, the product stands out, for better or worse, when it encounters a log source without a predefined processor. At that point, Splunk attempts to figure out the log source and generate a processor on the fly. If this doesn't work--and it doesn't always--you can write a processor for undefined log files. A command-line utility attempts to process the unknown log file and map it to Splunk's standard format. If the log isn't too strange, it does a good job. However, if you need to write your own processor, you'll be at it awhile.

Splunk lets you find obscure events in your logs, using a clickable interface to search for a general term followed by other terms related to that query. You can then exclude specific terms to trim out unwanted events. A bar graph showing the number of events over time also can show specific periods of time that have more events than they should. Splunk makes searching for events even easier by normalizing the time stamps on all your data sources. You need to query for only one type of time stamp and all the different types will also show up in the proper time sequence.

Even though Splunk greatly assists you when exploring the depths of your logs, you still must know how those logs work. You may need to use wild cards to make sure you see all the pertinent logs. For example, Cisco firewall's log IP addresses in at least two different formats--say, "from 192.168.10.2/80" and "outside:192.168.10.2/80." We searched our logs for "192.168.10.2," and Splunk returned log lines in the first format only, but if we used a wild card (*192.168.10.2), all the log lines were returned. The vendor is aware of this problem and plans to fix it.

Splunk runs a wiki called "SplunkBase," which lets you look up a log that you don't know and see what someone else has to say about it. If you can't find enough help there, you can anonymize your data and share the event by posting your unknown log to SplunkBase. Other users can then comment on your entry.Splunk Saves The Day

To test this product, we installed the 30-day free trial of the Beta 1 version of Splunk 2.1 on a dual-core 3.4-GHz Intel Pentium 4 processor-based system with 2 GB of memory running SuSE Linux 10. Installation took just 10 minutes. The Splunk server uses unprivileged ports, so you can install it as a non-root user. For security, the Splunk server's interface is run in an encrypted SSL Web server.

We learned more about our network than we would like to admit. Splunk quickly located the source of an LDAP error. Upon looking at the graph of events by time, we spotted the same error message every 10 minutes. It took just a few minutes to match that error message with a scheduled job that changes users' passwords. We saved that Splunk search, and now use the Live Splunk feature (think "cron") to run the query hourly. Splunk sends an e-mail when it finds something that matches the criteria we set. Competing vendors do some alerting, but Splunk's are more customizable.

Splunk was useful for tracking mail through our convoluted e-mail system, and even more handy when we decided to feed Splunk our IDS logs. Like any good log search tool, we could poke at questionable IDS entries, then search across all of our logs for events that occurred in the same time frame. We expanded a search for an attacker's IP address and discovered that attacker was attempting several different attacks on many of our systems at once. By using Splunk to search the millions of log lines in our authentication logs, Web server and mail logs at once, we determined that we weren't looking at a targeted attack, but rather a broad sweep across all our systems.

Although Splunk server is great as a search engine for log data, where its strengths lie, it does have some shortcomings. Splunk generates good technical reports for the systems administrators, but those reports won't impress management. By comparison, LogLogic LX 2000's easily customizable product has built-in templates, and Tenable's Log Correlation Engine can generate executive summaries or technical reports.It's also awkward to expire logs from Splunk's database after a given period of time. By default, Splunk retains 180 days' worth of logs in its databases, but this can be changed to whatever your logging policy states. To make this change, you have to mess with an XML file deep inside the server's inner workings. The vendor is working on making this more convenient in their next feature release.

Splunk server 2.1 will set you back $25,000 for a system that can handle up to 40 GB of log data per day--plenty of capacity for a midsize enterprise. The beta version, which we tested, is free for the first 30 days; Splunk then charges the full price to monitor more than 500 MB of logs per day. In contrast LogLogic's LX 2000 (a device that can handle 3,000 messages per second) and Tenable's Log Correlation Engine (unlimited license) are priced at about $25,000 and $50,000, respectively, and are suitable for a midsize enterprise.

Dave Decoster is the network security administrator for the Computer-aided Engineering Center at the University Of Wisconsin-madison. Write to him at ddecoster@ nwc.com.