Re: webspider getting stuck

2008-05-25 Thread John Nagle
[EMAIL PROTECTED] wrote: i am writing a simple webspider . how do i avoid getting stuck at something like this: Enter username for W3CACL at www.w3.org: ? It's a silly feature of urllib. See http://docs.python.org/lib/module-urllib.html where it says: "Note: When performing basic authe

webspider getting stuck

2008-05-24 Thread notnorwegian
i am writing a simple webspider . how do i avoid getting stuck at something like this: Enter username for W3CACL at www.w3.org: ? i can obv add an if-clause for the specific site but since i guess there will be more of the same thats ov not a viable approach in the long run. -- http://mail.pytho