a few questions about scrapy
I've installed scrapy and gotten a basic set-up working, and I have a few odd questions that I haven't been able to find in the documentation. I plan to run it occasionally from the command line or as a cron job, to scrape new content from a few sites. To avoid duplication, I have in memory two sets of long with the md5 hashes of the URLs and files crawled, and the spider ignores any that it has seen before. I need to load them from two disk files when the scrapy job starts, and save them to disk when it ends. Are there hooks or something similar for start-up and shut-down tasks? How can I put a random waiting interval between HTTP GET calls? Is there any way to set the proxy configuration in my Python code, or do I have so set the environment variables http_proxy and https_proxy before running scrapy? thanks -- http://mail.python.org/mailman/listinfo/python-list
A rule for your twitlist/mailing list
Where path includes "google.com" and subject includes "solutions" or "test", delete. 99 percent of the junk just . gone . feels so good I wish Google would still let you subscribe to the newsgroup and recieve updates in your inbox - that way I can mark the testbanks as "spam" Is the mailing list for comp.lang.python still open? -- https://mail.python.org/mailman/listinfo/python-list
Re: Finding Return Code From GPG
On Sat, 08 Jul 2006 00:26:06 +0200, Piet van Oostrum wrote: > >>NN> What is still not working is the display from gpg. It shows up on the >>NN> monitor screen like this: > >>NN> gpg: Signature made Tue 04 Jul 2006 08:04:10 AM CET using DSA key ID >>NN> 06B09BA4 >>NN> gpg: Good signature from "Boz Scraggs <[EMAIL PROTECTED]>" > >>NN> I need to get that into a file so I can parse it. Any further suggestions? > > That output is passed to stderr. So you can get it in a string: > > output = Popen(["gpg", "--output", "outfile", "--verify", "sigtest"], > stderr=PIPE).communicate()[1] That worked, thanks very much; and my thanks to all who replied. -- http://mail.python.org/mailman/listinfo/python-list
Finding Return Code From GPG
I'm running gpg in python to verify a signature. I can see that it is working, because gpg is displaying the results. But I need a way to let the python script know this. According to the gpg manual there is a return code from gpg of 0 if the verify is good and 1 if it is bad. Can anyone tell me how I can get the python script to 'see' this return code? -- http://mail.python.org/mailman/listinfo/python-list
Re: Finding Return Code From GPG
On Tue, 04 Jul 2006 19:00:23 +0200, Dennis Benzinger wrote: > Nomen Nescio wrote: >> I'm running gpg in python to verify a signature. I can see that it is >> working, because gpg is displaying the results. >> >> But I need a way to let the python script know this. According to the >> gpg manual there is a return code from gpg of 0 if the verify is good >> and 1 if it is bad. >> >> Can anyone tell me how I can get the python script to 'see' this return >> code? >> >> > How do you run GPG? I suggest using the subprocess module > <http://docs.python.org/lib/module-subprocess.html>. If you just need > something simple then you can use its call function > <http://docs.python.org/lib/node236.html>. Thanks, I used the popen function which did some of what I want. Here is the code I used: from subprocess import * output = Popen(["gpg", "--output", "--verify", "sigtest"], stdout=PIPE).communicate()[0] That worked, sort of - it did the verification at least and I can see the return code. What is still not working is the display from gpg. It shows up on the monitor screen like this: gpg: Signature made Tue 04 Jul 2006 08:04:10 AM CET using DSA key ID 06B09BA4 gpg: Good signature from "Boz Scraggs <[EMAIL PROTECTED]>" I need to get that into a file so I can parse it. Any further suggestions? -- http://mail.python.org/mailman/listinfo/python-list
python is great
python is great. -- http://mail.python.org/mailman/listinfo/python-list
scope of generators, class variables, resulting in global na
Hello, Can someone help me understand what is wrong with this example? class T: A = range(2) B = range(4) s = sum(i*j for i in A for j in B) It produces the exception: : global name 'j' is not defined The exception above is especially confusing since the following similar example (I just replaced the generator by an explicit array) works: class T: A = range(2) B = range(4) s = sum([(i*j) for i in A for j in B]) (BTW, the class scope declarations are intentional). Thanks, Leo. -- http://mail.python.org/mailman/listinfo/python-list
