Hi,
Does anyone know where I can find a perl program that will spider small
sections of the internet? I'm using wget right now to spider my own
sites but it just doesn't have enough options. The best thing would be
a stand alone perl program which is not too complicated and can be
configured relatively easily. Otherwise, any hints on using some of the
perl modules to accomplish this?
What I would like program to do:
input: URL
actions:
1-download URL and make a list of links
2-make a list of domains that are linked to from the URL
3-delete Page (but output the URL and links to a log file)
4-spider all links from page
5-update list of domains, not replicating ones already listed
6-repeat steps 1 to 5
7-create a table that shows who links to who
output:file with link relationships between domains (but not specific
pages)
Any tips on the different modules I might use and how I might use them
would be great.
Thanks,
Bryan
