I'm working on writing a web robot (for searching web pages) in Python so I've been browsing around the web looking for data about building web robots.
I'm surprised to find that I didn't find much data. Seems like robots would be kind of a common thing to build. I know there is this: "The Web Robots Pages" (http://www.robotstxt.org/wc/robots.html) but that doesn't have a lot of data and seems pretty out of date. I checked amazon.com for books on building web robots and I didn't find very much. Seems kind of weird that I don't find much data on this subject. Anybody know some good places to go to get information on building a web robots? -- http://mail.python.org/mailman/listinfo/python-list
