Web crawler engine.

There are lot of web search engines around, i mean like altavista's google's infoseek's etc.. What does one do when she wants to build one herself. I mean what do you people use when you build local search engine ( city, locality, my needs are about 100 000 pages?webs? for the beggining). I've be

Web crawler engine.

There are lot of web search engines around, i mean like altavista's google's infoseek's etc.. What does one do when she wants to build one herself. I mean what do you people use when you build local search engine ( city, locality, my needs are about 100 000 pages?webs? for the beggining). I've b