MC² Market & Competitive Convergence

Top search engines (in history now)

According to a study done by Steve Lawrence and C. Lee Giles of the NEC Research Institute, Princeton, New Jersey USA and published in the July 8, 1999 journal of Nature, the Web now has some 800 million pages consisting of more than 6 trillion bits.

So, how do we find what we need or make sure our pages can be found in this ever expanding Web? Lucky for us, they studied 11 search engines and ranked them by how much of the Web they cover. There are some real surprises here. First, a lot of the Web isn't covered by any one search engine. And who is Northern Light, anyway? In the last survey, in 1997, the Web was only 320 pages big and they weren't around. Well, they are now, and I surfed over and registered this site. And apparently some meta search engines haven't heard of them either: Even Metacrawler didn't have Northern Light on their list of search engines that they simultaneously query.

And if you haven't registered your site, you had better start now because the study said it takes, on average, six months for a search engine to send out its spider, have it crawl on your site, report back and list your site. Spiders or bots (short for robots) follow the Web's link from site to site, indexing the pages as they crawl about. This leads to another tip: Have others link to your site from their page. This is like spider food. So, let's exchange links. Also, see the list of 25 search engines at the bottom of the page.

Search Engine Coverage (%)
Northern Light 16.0
Snap 15.5
Altavista 15.5
HotBot 11.3
Microsoft 8.5
Infoseek 8.0
Google 7.8
Yahoo 7.4
Excite 5.6
Lycos 2.5
EuroSeek 2.2

Now this all seems like ancient history. For instance, Northern Light is no longer even a search engine, having morphed into something else altogether. Still, Dr. Lawrence has continued his research at NEC, and it's worthwhile to view his work to gain insight about how to better promote your site.

Search tips

Online b to b biz ballooning

According to Forrester Research Inc., Cambridge Mass., intercompany trade of hard goods over the Internet hit $43 billion in 1998 and could reach $1.3 trillion by 2003, an annual growth rate of 99%.

For Industrial equipment and supplies, Forrester Research forecasts:

$ (millions)

1997 1999 2001 2003
$88 $1,266 4,520 15,699

The search engine information below is from Robert W. Bly, a freelance copywriter specializing in conventional and Internet direct mail. His latest book, Internet Direct Mail: The Complete Guide to Successful e-mail Marketing Campaigns (coauthored with Steve Roberts and Michelle Feit), will be published in October, 2000 by NTC Business Books.

Search Engines (25), in Alphabetical Order:

America Online
AOL Search allows its members to search across the Web and AOL's own content from one place. The "external" version, listed above, does not list AOL content. The main listings for categories and Web sites come from the Open Directory (see below).

AltaVista is consistently one of the largest search engines on the Web in terms of pages indexed. Its comprehensive coverage and wide range of power searching commands makes it a particular favorite among researchers. It also offers a number of features designed to appeal to basic users. These include "Ask AltaVista" results, which come from Ask Jeeves (see below), and directory listings from the Open Directory and LookSmart.

Ask Jeeves
Ask Jeeves is a human-powered search service that aims to direct you to the exact page that answers your question. If it fails to find a match within its own database, then it will provide matching Web pages from various search engines. Some results from Ask Jeeves also appear within AltaVista.

Direct Hit
Direct Hit measures what people click on in the search results presented at its own site and at its partner sites, such as HotBot. Sites that get clicked on more than others rise higher in Direct Hit's rankings. Thus, the service dubs itself a "popularity engine." Aside from running its own Web site, Direct Hit provides the main results that appear on HotBot (see below). It is also available as an option to searchers at MSN Search. Direct Hit is owned by Ask Jeeves (above).

dot com directory
The dot com directory is the largest listing of on-line businesses you'll find anywhere on the Web. It provides the URL, street address, maps, and some company information for each business listing and allows users to search by company name, business type, or URL by city, state or Canadian region.

Excite is one of the more popular search services on the Web. It offers a fairly large index and integrates non-Web material such as company information and sports scores into its results, when appropriate.

FAST Search
Formerly called All The Web, FAST Search aims to index the entire Web. It was the first search engine to break the 200 million Web page index milestone and consistently has one of the largest indexes of the Web. The Norwegian company behind FAST Search also powers some of the results that appear at Lycos (see below).

Go / Infoseek
Go is a portal site produced by Infoseek and Disney. It offers portal features such as personalization and free e-mail, plus the search capabilities of the former Infoseek search service, which has now been folded into Go. Searchers will find that Go consistently provides quality results in response to many general and broad searches. It also has an impressive human-compiled directory of Web sites. Go is not related to GoTo (below).

Unlike the other major search engines, GoTo sells its main listings. Companies can pay money to be placed higher in the search results, which GoTo feels improves relevancy. Non-paid results come from Inktomi. GoTo launched in 1997. In February 1998, GoTo shifted to its current pay-for-placement model and soon after replaced the WWW Worm with Inktomi for its non-paid listings.

Google is a search engine that makes heavy use of link popularity as a primary way to rank Web sites. This can be especially helpful in finding good sites in response to general searches such as "cars" and "travel," because users across the Web have in essence voted for good sites by linking to them. The system works so well that Google has gained widespread praise for its high relevancy. Google also has a huge index of the Web and provides some results to Yahoo and Netscape Search.

HotBot is a favorite among researchers because of its many powerful searching features. In most cases, HotBot's first page of results comes from the Direct Hit service (see above), and then secondary results come from the Inktomi search engine, which is also used by
other services. It gets its directory information from the Open Directory project (see below). HotBot launched in May 1996 as Wired Digital's entry into the search engine market. Lycos purchased Wired Digital in October 1998 and continues to run HotBot as a separate search service.

Backed by US television network CBS, iWon has a directory of Web sites generated automatically by Inktomi, which also provides its more traditional crawler-based results. iWon gives away daily, weekly, and monthly prizes in a marketing model unique among the major services. It launched in Fall 1999.

Originally, there was an Inktomi search engine at UC Berkeley. The creators then formed their own company with the same name and created a new Inktomi index, which was first used to power HotBot. Now the Inktomi index also powers several other services. All of them tap into the same index, though results may be slightly different due to variations in filtering and ranking methods.

LookSmart is a human-compiled directory of Web sites. In addition to being a stand-alone service, LookSmart provides directory results to MSN Search, Excite, and many other partners. Inktomi provides LookSmart with search results when a search fails to find a match from among LookSmart's reviews.

Lycos started out as a search engine, depending on listings that came from spidering the Web. In April 1999, it shifted to a directory model similar to Yahoo. Its main listings come from the Open Directory project, and then secondary results come from the FAST Search engine. Some Direct Hit results are also used. In October 1998, Lycos acquired the competing HotBot search service, which continues to be run separately.

MSN Search
Microsoft's MSN Search service is a LookSmart-powered directory of Web sites, with secondary results that come from Inktomi. RealNames and Direct Hit data are also made available. MSN Search offers a unique way for Internet Explorer 5 users to save past searches.

Netscape Search
Netscape Search's results come primarily from the Open Directory and Netscape's own "Smart Browsing" database, which does an excellent job of listing "official" Web sites. Secondary results come from Google. At the Netscape Netcenter portal site, other search engines are also featured.

Northern Light
Northern Light is another favorite search engine among researchers. It features a large index of the Web, along with the ability to cluster documents by topic.

Open Directory
The Open Directory uses volunteer editors to catalog the Web. Formerly known as NewHoo, it was launched in June 1998. When Netscape acquired Open Directory in November 1998, they pledged that anyone would be able to use information from the directory through an open license arrangement. Netscape itself was the first licensee. Lycos and AOL Search make heavy use of Open Directory data. AltaVista and HotBot prominently feature Open Directory categories within their results pages.

Raging Search
Operated by AltaVista, Raging Search uses the same core index as AltaVista and virtually the same ranking algorithms. Why use it? AltaVista offers it for those who want fast search results, with no portal features getting in the way.

The RealNames system is meant to be an easier-to-use alternative to the current Web site addressing system. For instance, those with RealNames-enabled browsers can enter a word like "Nike" to reach the Nike Web site.

Snap is a human-compiled directory of Web sites, supplemented by search results from Inktomi. Like LookSmart, it aims to challenge Yahoo as the champion of categorizing the Web. Snap launched in late 1997 and is backed by Cnet and NBC.

WebCrawler has the smallest index of any major search engine on the Web -- think of it as Excite Lite. The small index meansWebCrawler is not the place to go when seeking obscure or unusual material. However, some people may feel that by having indexed fewer pages, WebCrawler provides less overwhelming results in response to general searches. In 1996, Excite acquired WebCrawler, which it continues to run as an independent search engine.

Yahoo is the Web's oldest and most popular search service and has a well-deserved reputation for helping people find information easily. The secret to Yahoo's success is human beings. It is the largest human-compiled guide to the Web, employing about 150 editors who have categorized over 1 million sites.

WebTop is a crawler-based search engine that claims an extremely large index. In addition to listing Web pages, WebTop also provides information from news sources in its search results.