Can IBM Bring the Semantic Web to Notes and Outlook?

I've been trying out Omnifind Personal Email Search, a tool for Outlook and Notes that IBM posted for free download on its Alphaworks site last week. While email search itself isn't new (Google's Desktop Search will happily index your inbox along with the rest of your hard drive), the IBM software is slightly different.

Rather than finding a specific email message or thread, Omnifind is aimed at searching for unstructured data: the information buried within an inbox. And it looks like one of the first genuinely useful desktop applications based on the Semantic Web – an idea that has been somewhat eclipsed by Web services and Web 2.0, but which could eventually unite them with SOA.
The key technology in the tool is UIMA (Unstructured Information Management), an IBM-led open-source framework for analyzing text and other unstructured data. This is essentially pattern recognition: a series of ten digits with hyphens, brackets or spaces in the right places is a phone number, two letters followed by five numbers is a zip code, etc. The tool uses this to generate semantic XML tags automatically, overcoming what has been the biggest barrier to the Semantic Web: that people don't have the time or inclination to add metadata to documents manually.
Automated pattern-recognition isn't new to users of Web-based email, so OmniFind could be viewed as yet another case of corporates slowly catching up with consumers. (Gmail won't just automatically recognize a street address; it will offer to display a map and give directions.) The same kinds of algorithms are used by search engines internally, both for ad-serving on the fly and for building the main search index itself.

IBM goes beyond Google in a couple of areas. First, the tags are fully exposed so that people can use them in searches. Type "John address" and the system will show every street address in an email from (or mentioning) someone called John – even if the word "address" doesn't appear. A large email archive could include hundreds of messages from people called John, of course, so there are lots of duplicates. But that's a good thing if the point is to find and map the street address buried in his email signature, as the frequency of messages helps the program guess which John you're looking for. Who needs a separate contact database when everyone's info is already in your inbox?
The same principle applies to email addresses, URLs, and phone numbers – with the latter presenting an option to call the number using softphones like Skype if installed. By default, OmniFind also tags parts of each message as a date, time or "person", representing a human name . All of them can be used in the same kind of searches. For example, "person InformationWeek" will attempt to find all the people in an inbox who work at InformationWeek.
The algorithms underlying UIMA aren't perfect. Just as the ads on Google and Yahoo often miss their targets, Omnifind can throw up false positives: It thought that the person mentioned most in my inbox was named Colleague, presumably because so much spam begins "Dear Colleague." But this is mostly because the tags (and their corresponding patterns) are fairly generic.
Omnifind also lets users edit the default tags or create their own, using regular expressions to represent search patterns. IBM suggests that these be used to customize the search to a specific organization, finding information like employee IDs or package tracking numbers. It could also be used to weed out irrelevant search results, most of which are caused by the one-size-fits-all approach that public search engines must take.

The main drawback is that devising and hone regular expressions can take a lot of work. IBM could help a lot by including optional tags for fields that aren't organization-specific, such as ISBNs, international phone numbers and credit card details (which probably shouldn't be flying around in emails at all, but that's a separate problem.) IBM's SOA strategy is very focused on customizations for specific industries, so it will probably do something similar here if the technology makes it into a commercial product.
Another problem is that the indexing process needs to be repeated every time a new tag is added, a fairly processor-intensive task. The tool has five different settings that trade off indexing speed against impact on other apps, but even at the fastest it took three hours to apply the default tags to a 1 GB Notes archive (about 30,000 messages) on a dual-core machine. It has to run on a client PC, meaning it's actually better suited for Microsoft Outlook than IBM's own Lotus Notes.
The tool is entirely browser-based, which at first seems a little awkward: Actually reading a searched-for email requires launching a separate application (Notes or Outlook.) However, the point is to avoid needing to reading the email at all. When it works as intended, it simply extracts the necessary information and bypasses the email client.
The long-term goal of UIMA is to apply the same automated pattern recognition to other kinds of data, which will likely be harder. Email is in some senses the low-hanging fruit, as it isn't entirely unstructured: There are the formal fields like "To", plus the informal structure of salutations and signatures that it inherited from regular mail.