On Mar 4, 2009, at 4:44 PM, bruce wrote:
Hi...
Sorry that this is a bit off track. Ok, maybe way off track!
But I don't have anyone to bounce this off of..
I'm working on a crawling project, crawling a college website, to
extract
course/class information. I've built a quick test app in python to
crawl the
site. I crawl at the top level, and work my way down to getting the
required
course/class schedule. The app works. I can consistently run it and
extract
the information. The required information is based upon an XPath
analysis of
the DOM for the given pages that I'm parsing.
My issue is now that I have a "basic" app that works, I need to
figure out
how I guarantee that I'm correctly crawling the site. How do I know
when
I've got an error at a given node/branch, so that the app knows that
it's
not going to fetch the underlying branch/nodes of the tree..
When running the app, I can get 5000 classes on one run, 4700 on
antoher,
etc... So I need some method of determining when I get a "complete"
tree...
How do I know when I have a complete "tree"!
hi Bruce,
To put this another way, you're trying to convince yourself that your
program is correct, yes? For instance, you're worried that you might
be doing something like discovering a URL on a site but failing to
pursue that URL, yes?
The standard way of testing any program is to test known input and
look for expected output. Repeat as necessary. In your case that would
mean crawling a site where you know all of the URLs and to see if your
program finds them all. And that, of course, isn't proof of
correctness, it just means that that particular site didn't trigger
any error conditions that would cause your program to misbehave.
I think every modern OS makes it easy to run a Web server on your
local machine. You might want to set up suite of test sites on your
machine and point your program at localhost. That way you can build a
site to test your application in areas you fear it may be weak.
I'm unclear on what you're using to parse the pages, but (X)HTML is
very often invalid in the strict sense of validity. If the tools
you're using expect/insist on well-formed XML or valid HTML, they'll
be disappointed on most sites and you'll probably be missing URLs. The
canonical solution for parsing real-world Web pages with Python is
BeautifulSoup.
HTH
Philip
--
http://mail.python.org/mailman/listinfo/python-list
