It's very simple, and I'm sure the people who run the search engines have sufficient grey matter to fathom this concept-- More Comprehensive Listings Mean Better Search Results. See? Very simple.
But Infoseek, along with several other major search engines, can't seem
to get this through their thick heads. They have taken it upon themselves
to ban anyone on free Web hosts from the listings. Effectually, you have
to pay to be listed on these engines. The reasoning behind these moves,
according to Infoseek, is that all members of all free hosts "attempt to
subvert the index" by spamming the engine (pages with word lists or invisible
text, misleading META tag, etc.). Granted, some freebie members do try
to spam the indexes, as do many other, nonfree sites. Some quasi-legit
companies use tactics that many consider subversive "cheating" on rankings;
State Farm Insurance recently raised the eyebrows of engines and
Engine Watch alike for its rampant and excessive use of "bridge pages"--essentially
hundreds of separate pages containing lists of specific queries people
were likely to type into search engines. "Car insurance", "boat insurance",
etc. When a user clicked into the bridge page, they were redirected by
a huge link to State Farm's home page. Others (including SF?) compile lists
of domain names of search spiders, and dynamically serve the spider a different
(keyword-packed) page than is served to normal visitors. Cheater alert!!
The number of sites in the index has a very direct and measurable impact on the quality of the results returned. In searching for a very specific tidbit, fewer pages means less likelyhood of finding what you need! Furthermore, my experience has been that personal websites (banned) are infinitely more useful than big-business and advertiser websites. In fact, PC Magazine actually printed a letter I wrote them to this effect (I think the Nov. '97 issue, but don't quote me on that). Corporate websites seldom carry specific, useful information; they just want to sell sell sell, and push their crap on you in any and every way possible. And some materials just plain impossible to find on corporate sites--pirate radio, witty and creative Qbasic games and some program source codes, poetry, rant pages, dirty jokes, witty writings by someone who writes from the heart and for the fun of it (not because he's being paid to do so), or the personal experiences of an Everyday Joe. You will not find this on any corporate site. Due to the banning of personal pages, these valuable resources have been lost by the Internet community.
This selective banning based on where a resource "lives" is a form of
discrimination plain and simple, scarcely different from that which has
brought about affirmative action and all sorts of other laws that we, as
a society of moral and intelligent beings, shouldn't even need.
1940's - "You're from Africa; you can't come into our store."
1990's - "You're from Angelfire; you can't be in our search index."
Wouldn't it be interesting to imagine a future Internet where personal, "freebie" websites get priority placement over commercial content in search engine listings; a kind of "affirmative action" to repay the discrimination of today.
I have a simple method of determining the quality of a search engine.
I submit my own freebie site (you're reading it now) to the engine, wait
the specified amount of time for the index to be updated, then search for
it. If my site isn't listed, I don't use that engine. It's that simple.
My site, as a representative sample of personal website content, is an indication of how much of the "good stuff" is available though a particular search engine. If your engine is withholding the "good stuff" (personal homepage content) then I must regard its results as lower quality, and seek better elsewhere.