Search-engine designers are working with linguists to understand and act on the meanings in text--a field known as natural-language processing, or NLP. The designers use algorithms that rely on phonetics, morphology, syntax, semantics, discourse and pragmatic context. Search engines can use these linguistic aspects to improve searching by expanding search terms--for example, finding France, Germany and Belgium when searching for Europe.

In the most complex form of natural-language search, the system will analyze and distill each query and document to a formal representation, which can be compared to locate the best matches. Such systems are designed to be better at answering questions and dealing with vague information needs--a question like "Why is the sky blue?" for example--than are traditional engines that require exact matches of words, such as "What articles have been published in Network Computing about search engines?"

Most search engines simply use NLP to interpret queries written in question form and to regularize singular and plural forms of words in the index. Excalibur RetrievalWare, for example, sets up a knowledge base for synonyms and provides conceptual searching. Ask Jeeves employs human subject experts to invent questions and locate their associated answer pages; this engine's designers use NLP mainly to match user questions to predefined questions. The Web search engine Northern Light uses NLP to recognize and separate kinds of documents (press releases or FAQs, for example), which provides helpful clustering of results.

But even as NLP promises to make searches better, there's still the issue of retraining users. Analysis of search-engine logs shows that very few users type whole questions. Instead, they tend to enter two or three key words and hope for the best. As a result, the search engines miss contextual clues as to the nature of the questions, which makes NLP less useful. Still, in the long-term, NLP for document analysis will locate more useful pages and improve results' relevance ranking and categorization tremendously.