Search engine for FoRK

Rohit Khare (khare@mci.net)
Wed, 02 Jul 1997 18:14:17 -0400

Dan and Keith have both brought it up now, so I looked into the situation.
First, AltaVista only claims to have indexed 40 pages from XeNT. So, the
hunch is that AV is coping with the load by capping the number of pages
from a site. W3.org, for example, has only 600 reported hits.

Excite for Web Servers doesn't support NeXTSTEP, nor Win95. If I have to
adminster another unix box, I'd rather do glimpse or harvest. My ideal
choice would be a standalone, spider-indexer that works on a separate
server. Seems like a fine isolation, instead of a tool which indexes the
file-space and maps them together later.

Any solution? Any volunteers? The entire site will fit on a Zip disk... :-)

RK

-------------

Also came across some reverse-ego-surfing: ALan Cooper was looking for his
name and pointed to me; I was looking for mine, so now I point to him:
http://www.cooper.com/aboutface/about_about_face.html

-------------

How often do you update the index?
The index is update once a day at least, typically at night. All
pages manually submitted to AltaVista through the Add URL
feature as well as all changes brought in by our robot Scooter are
then processed.

How often does scooter visit sites?
When we introduced AltaVista, we used to create a completely new
index every four to six weeks. Now we update our index
continuously to reflect changes on the Web.
But we don't use a fixed schedule. Instead Scooter visits pages
according to the frequency at which they appear to change: a
page which has been stable for months will be revisited less often
than a page which is different every time we check it. 

---
Rohit Khare /// MCI Internet Architecture (BOS) /// khare@mci.net
Voice+Pager: (617) 960-5131  VNet: 370-5131   Fax: (617) 960-1009