$ for searching

Chuck Murcko chuck@topsail.org
Tue, 15 Jan 2002 15:42:01 -0500


Start with your robots.txt file.

http://www.robotstxt.org/wc/norobots.html

would be a start. There's a list of known robots there under

http://www.robotstxt.org/wc/robots.html

as well as a mailing list for the issue. This is by no means the total 
solution but it helps.

Chuck

On Tuesday, January 15, 2002, at 03:19 PM, Allan Doyle wrote:

> Good spammish subject line, eh?
>
> I just saw that Northern Light is about to stop allowing free
> searches. This doesn't cause me too much pain, because I never used
> them for searching anyway. But it does bring up an issue. Do I want to
> provide content for them to index if they are not going to let people
> find it for free?
>
> What other search engines out there have this policy? Is there a
> central list? Are people building robot.txt files to block non-free
> engines?
>
> I'm going to have to think about this for a little while. They hold 25
> records that came from my sites. That's a drop in the bucket, but hey,
> someone has to stand up for the free engines. If I pull my content
> from non-free engines, that makes them just a little less desirable and
> will swing the pendulum an infinitesimal amount away from them. Thus
> market forces may prevail, driving traffic to the free sites up,
> enabling them to generate more revenue from advertising.
>
> But we all know it's not so black and white. A lot of sites that let
> you search for free also take money for good placement. So what's a
> socially conscious web page owner to do?
>