MC² Market & Competitive Convergence
Top search engines (in history now)
According to a study done by Steve Lawrence and C. Lee Giles of the NEC
Research Institute, Princeton, New Jersey USA and published in the
July 8, 1999 journal of
Web now has some 800 million pages consisting of more than 6 trillion bits.
So, how do we find what we need or make sure our pages can be found
in this ever expanding Web? Lucky for us, they studied 11 search engines
and ranked them by how much of the Web they cover. There are some real
surprises here. First, a lot of the Web isn't covered by any one search
engine. And who is Northern Light, anyway? In the last survey, in 1997,
the Web was only 320 pages big and they weren't around. Well, they are
now, and I surfed over and registered this site. And apparently some meta
search engines haven't heard of them either: Even
didn't have Northern Light on their list of search engines that they simultaneously
And if you haven't registered your site, you had better start now because
the study said it takes, on average, six months for a search engine to
send out its spider, have it crawl on your site, report back and list your
site. Spiders or bots (short for robots) follow the Web's link from site
to site, indexing the pages as they crawl about. This leads to another
tip: Have others link to your site from their page. This is like spider
food. So, let's exchange links. Also, see the list of 25 search engines at the bottom of the page.
Now this all seems like ancient history. For instance, Northern Light is no longer even a search engine,
having morphed into something else altogether. Still, Dr. Lawrence has continued his research at NEC, and it's worthwhile
to view his work to gain insight about how to better promote your site.
Online b to b biz ballooning
According to Forrester Research Inc., Cambridge Mass., intercompany trade of hard goods over
the Internet hit $43 billion in 1998 and could reach $1.3 trillion by 2003,
an annual growth rate of 99%.
For Industrial equipment and supplies, Forrester
The search engine information below is from
Robert W. Bly, a freelance copywriter specializing in conventional and
Internet direct mail. His latest book, Internet Direct Mail: The Complete
Guide to Successful e-mail Marketing Campaigns (coauthored with Steve Roberts
and Michelle Feit), will be published in October, 2000 by NTC Business
Search Engines (25), in Alphabetical Order:
AOL Search allows its members to search across the Web and AOL's own
content from one place. The "external" version, listed above, does not
list AOL content. The main listings for categories and Web sites come from
the Open Directory (see below).
AltaVista is consistently one of the largest search engines on the
Web in terms of pages indexed. Its comprehensive coverage and wide range
of power searching commands makes it a particular favorite among researchers.
It also offers a number of features designed to appeal to basic users.
These include "Ask AltaVista" results, which come from Ask Jeeves (see
below), and directory listings from the Open Directory and LookSmart.
Ask Jeeves is a human-powered search service that aims to direct you
to the exact page that answers your question. If it fails to find a match
within its own database, then it will provide matching Web pages from various
search engines. Some results from Ask Jeeves also appear within AltaVista.
Direct Hit measures what people click on in the search results presented
at its own site and at its partner sites, such as HotBot. Sites that get
clicked on more than others rise higher in Direct Hit's rankings. Thus,
the service dubs itself a "popularity engine." Aside from running its own
Web site, Direct Hit provides the main results that appear on HotBot (see
below). It is also available as an option to searchers at MSN Search. Direct
Hit is owned by Ask Jeeves (above).
dot com directory
The dot com directory is the largest listing of on-line businesses
you'll find anywhere on the Web. It provides the URL, street address, maps,
and some company information for each business listing and allows users
to search by company name, business type, or URL by city, state or Canadian
Excite is one of the more popular search services on the Web. It offers
a fairly large index and integrates non-Web material such as company information
and sports scores into its results, when appropriate.
Formerly called All The Web, FAST Search aims to index the entire Web.
It was the first search engine to break the 200 million Web page index
milestone and consistently has one of the largest indexes of the Web. The
Norwegian company behind FAST Search also powers some of the results that
appear at Lycos (see below).
Go / Infoseek
Go is a portal site produced by Infoseek and Disney. It offers portal
features such as personalization and free e-mail, plus the search capabilities
of the former Infoseek search service, which has now been folded into Go.
Searchers will find that Go consistently provides quality results in response
to many general and broad searches. It also has an impressive human-compiled
directory of Web sites. Go is not related to GoTo (below).
Unlike the other major search engines, GoTo sells its main listings.
Companies can pay money to be placed higher in the search results, which
GoTo feels improves relevancy. Non-paid results come from Inktomi. GoTo
launched in 1997. In February 1998, GoTo shifted to its current pay-for-placement
model and soon after replaced the WWW Worm with Inktomi for its non-paid
Google is a search engine that makes heavy use of link popularity as
a primary way to rank Web sites. This can be especially helpful in finding
good sites in response to general searches such as "cars" and "travel,"
because users across the Web have in essence voted for good sites by linking
to them. The system works so well that Google has gained widespread praise
for its high relevancy. Google also has a huge index of the Web and provides
some results to Yahoo and Netscape Search.
HotBot is a favorite among researchers because of its many powerful
searching features. In most cases, HotBot's first page of results comes
from the Direct Hit service (see above), and then secondary results come
from the Inktomi search engine, which is also used by
other services. It gets its directory information from the Open Directory
project (see below). HotBot launched in May 1996 as Wired Digital's entry
into the search engine market. Lycos purchased Wired Digital in October
1998 and continues to run HotBot as a separate search service.
Backed by US television network CBS, iWon has a directory of Web sites
generated automatically by Inktomi, which also provides its more traditional
crawler-based results. iWon gives away daily, weekly, and monthly prizes
in a marketing model unique among the major services. It launched in Fall
Originally, there was an Inktomi search engine at UC Berkeley. The
creators then formed their own company with the same name and created a
new Inktomi index, which was first used to power HotBot. Now the Inktomi
index also powers several other services. All of them tap into the same
index, though results may be slightly different due to variations in filtering
and ranking methods.
LookSmart is a human-compiled directory of Web sites. In addition to
being a stand-alone service, LookSmart provides directory results to MSN
Search, Excite, and many other partners. Inktomi provides LookSmart with
search results when a search fails to find a match from among LookSmart's
Lycos started out as a search engine, depending on listings that came
from spidering the Web. In April 1999, it shifted to a directory model
similar to Yahoo. Its main listings come from the Open Directory project,
and then secondary results come from the FAST Search engine. Some Direct
Hit results are also used. In October 1998, Lycos acquired the competing
HotBot search service, which continues to be run separately.
Microsoft's MSN Search service is a LookSmart-powered directory of
Web sites, with secondary results that come from Inktomi. RealNames and
Direct Hit data are also made available. MSN Search offers a unique way
for Internet Explorer 5 users to save past searches.
Netscape Search's results come primarily from the Open Directory and
Netscape's own "Smart Browsing" database, which does an excellent job of
listing "official" Web sites. Secondary results come from Google. At the
Netscape Netcenter portal site, other search engines are also featured.
Northern Light is another favorite search engine among researchers.
It features a large index of the Web, along with the ability to cluster
documents by topic.
The Open Directory uses volunteer editors to catalog the Web. Formerly
known as NewHoo, it was launched in June 1998. When Netscape acquired Open
Directory in November 1998, they pledged that anyone would be able to use
information from the directory through an open license arrangement. Netscape
itself was the first licensee. Lycos and AOL Search make heavy use of Open
Directory data. AltaVista and HotBot prominently feature Open Directory
categories within their results pages.
Operated by AltaVista, Raging Search uses the same core index as AltaVista
and virtually the same ranking algorithms. Why use it? AltaVista offers
it for those who want fast search results, with no portal features getting
in the way.
The RealNames system is meant to be an easier-to-use alternative to
the current Web site addressing system. For instance, those with RealNames-enabled
browsers can enter a word like "Nike" to reach the Nike Web site.
Snap is a human-compiled directory of Web sites, supplemented by search
results from Inktomi. Like LookSmart, it aims to challenge Yahoo as the
champion of categorizing the Web. Snap launched in late 1997 and is backed
by Cnet and NBC.
WebCrawler has the smallest index of any major search engine on the
Web -- think of it as Excite Lite. The small index meansWebCrawler is not
the place to go when seeking obscure or unusual material. However, some
people may feel that by having indexed fewer pages, WebCrawler provides
less overwhelming results in response to general searches. In 1996, Excite
acquired WebCrawler, which it continues to run as an independent search
Yahoo is the Web's oldest and most popular search service and has a
well-deserved reputation for helping people find information easily. The
secret to Yahoo's success is human beings. It is the largest human-compiled
guide to the Web, employing about 150 editors who have categorized over
1 million sites.
WebTop is a crawler-based search engine that claims an extremely large
index. In addition to listing Web pages, WebTop also provides information
from news sources in its search results.