Three search engines promise to serve up direct answers to so-called natural-language queries: natural-language veteran AskJeeves; GuruNet’s Answers.com, a relative newcomer to Web search; and MSN Search, which recently beefed up its search results with information culled from Microsoft’s Encarta encyclopedia. In our sample searches, however, the three services did much better at answering our general-information questions than they did at replying to technical, topical, or geo-specific questions.
We expected the answers to our questions to appear at the top of each site’s initial results page, without our having to follow a link. We gave the sites some credit when the correct answer was on the first page in the search results one click away, or was within the top ten links returned two or more clicks away.
We posed dozens of questions and reduced our results to a representative 15. AskJeeves produced four on-the-money answers; MSN Search and Answers.com answered three of our questions spot-on. We included Google as a control; its results were almost identical to those of Answers.com.
All three sites were on target with the question ‘What was John Dillinger famous for?’ though we had to click the top link at Answers.com, and they also aced ‘Who was the 17th president of the United States?’ Yet none of the three could immediately answer a question about the speed of USB 2.0 versus ethernet.
Of course, Web information is constantly changing. Our latest results varied considerably from those we got when we asked the same questions days earlier.
Extra words–ironically, the natural language–often got in the way of good results. For example, entering the keywords ‘high tide Santa Monica California’ returned better results than ‘When is high tide on October 15, 2005, in Santa Monica, California?’
Direct Answers: No Search Engine Aced Our Test
Web search champ Google was as good at answering our natural-language questions as the three sites that promote their ability to respond to questions rather than keyword phrases. Our goal was to see the correct answer without having to follow a link on the results page no clicks. We gave the sites partial credit for providing the answer on the top link’s page one click, and for having the answer in its top ten links two or more clicks. Following are 15 questions that represent the dozens we posed to each service, along with the correct answers. Also listed are two keyword phrases that we submitted to the sites as a control.
Concluding Results:
Answers.com’s results are nearly identical to those of Google, but with several general-interest questions leading to an "answer" page rather than to a list of Web results. AskJeeves lists more sponsored links than the other search services; in fact, there were often more sponsored links returned than non-sponsored links. MSN Search also returns many sponsored links, but it was the fastest of the bunch. On some searches such as "Where is Algerie?" MSN corrected misspellings on the fly with no user intervention. On Google’s results page, unobtrusive sponsored links are placed on the right side. On some searches such as "Where is Algerie?" Google corrected misspellings on the fly with no user intervention, and sometimes suggested the correct spelling.