Inconsistent SMTP/Gmail connection drop
I have a laptop that wakes up and then runs a script that collects
info and sends an email with a spreadsheet (zipped and about 350KB)
report to a number of people. However, every time I get the following
error:
File "/usr/lib/python2.7/smtplib.py", line 343, in getreply
raise SMTPServerDisconnected("Connection unexpectedly closed")
- When I check the laptop and manually run the function that sends the
email and report, it works fine. I have never been able to replicate
the error when doing it manually.
- It can't be a problem with Gmail at that time in the morning or my
connection, because the script also sends me a copy of the log file
straight after, and that works.
- The same code has been working for years, it just that I have
recently rebuilt the machine so now it is using Python 2.7. I have
also change the references to use absolute file references, rather
than relative. I also have changed the code to run as root.
- I've checked the smtp debug report and everything seems ok. I just
never get the confirmation code (250) to say that it's completed.
- Even stranger ... it ran ok once. But I didn't change a thing and
the next day it wouldn't work.
The only thing I can think of is that when the file is transferred to
the reports directory, it somehow isn't been released quickly enough
and perhaps this is tripping up SMTP. But ... this is a bit of a guess
given I've tried to exhaust all the other options.
Any ideas?
--
http://mail.python.org/mailman/listinfo/python-list
Re: Inconsistent SMTP/Gmail connection drop
Not sure how I'd test or determine that. I've set smtp debug to true and the output looks all good. I suppose I could try sending an email before the report is sent, as well as the email that is sent after it. On Thu, Aug 4, 2011 at 12:17 AM, David Stanek wrote: > On Wed, Aug 3, 2011 at 5:46 PM, Astley Le Jasper > wrote: >> >> Any ideas? >> > > Is it possible that the first email is sent before the network connection > has been properly established? > > -- > David > blog: http://www.traceback.org > twitter: http://twitter.com/dstanek > www: http://dstanek.com > -- http://mail.python.org/mailman/listinfo/python-list
Re: Inconsistent SMTP/Gmail connection drop
Thanks for those suggestions. I tried something last night before I got your ideas. 1. I added a line to send a copy of the report just to me, 2 lines before the line where it emails the report to all the recipients. 2. I also added a timer.sleep(5) pause just before the line that emails the reports to all the recipients. So the order was: a) Send the full report just to me FAILED. b) Pause for 5 seconds. c) Send the full report to all 4 recipients ... WORKED. d) Send the log file just to me ... WORKED. So ... what looks like may be happening is that something might still be locking the file. In which case i'll try again without the test email to me. And/or perhaps send a test email without the attachment before the main report to see if that gets through, to see if the connection with gmail is actually really connected. See also below... > 1) You might see if there's something about the size of the message - is it > bigger after collecting data all night? Is google disconnecting after a > maximum amount of data is transferred? I don't think so. The test message I mentioned above got through and the attachment was identical (320kb or abouts) > 2) You might try sending a tiny test message at the beginning, just to > yourself, and seeing if that has the problem As above. > 3) You might try running your script under a system call tracer: > http://stromberg.dnsalias.org/~dstromberg/debugging-with-syscall-tracers.html > - and inspecting the list of syscalls and their return codes near when the > process terminates Ouch. Looks complicated! ;-) > 4) I get disconnects from gmail once in a while too, especially when > downloading a large list of new message headers; I've been just coding > around it by making my code idempotent and otherwise restartable. This > seems to happen with both 2.x and 3.x, so I assume it's not really related > to the Python version, but I suppose I should try it on PyPy or Jython > someday. Note that the error message in 3.x was better than in 2.x. I was thinking that perhaps I could put a loop in that tests if the email goes through, and if not, pauses for a few seconds and then tries again. -- http://mail.python.org/mailman/listinfo/python-list
Re: Inconsistent SMTP/Gmail connection drop
Just to add a little bit to the mix, I have started having these problems since I rebuilt my machine (Xubuntu 11) and I changed the disc formats to ext4 without even thinking about it ... I was wondering if maybe that had something to do with it? -- http://mail.python.org/mailman/listinfo/python-list
Re: National grid to lat long conversion
Hi, I was a bit limited with time so in the end used an online service with an API which is great for the limited number of look-ups I needed. It can be found at: http://www.nearby.org.uk/ It was also very useful for site for other conversions and lookups. I manage to find a couple of other examples in other languages which I might have used as the basis for some pythonification. PHP - http://www.megalithia.com/search/llfuncshighlight.php - http://www.jstott.me.uk/phpcoord/ Javascript - http://www.movable-type.co.uk/scripts/latlong-gridref.html Cheers Neil -- http://mail.python.org/mailman/listinfo/python-list
Running a Python script from crontab
I need help ... I've been looking at this every evening for over a week now. I'd like to see my kids again! I have script that runs fine in the terminal but when I try to run it in a crontab for either myself or root, it bails out. The trouble is that obviously I get no console when using crontab so can't see any traceback. I use logging which shows 'info' messages as the script is working, but no error messages. I've peppered it with debug messages to try to track it down to a line, but it stops it the middle of appending data to a list. I'd expect it to bail out when calling a module or reading/writing to a different file. Is there any way of getting more info from the app, like popping up a console while its running? my crontab is: 30 15 * * * cd /home/myusername/src && python myscript.py ALJ -- http://mail.python.org/mailman/listinfo/python-list
Re: Running a Python script from crontab
James ... thanks for the suggestion. I have done this and the error logging usually catches all my errors and logs them. I wondered if logging itself was failing! Philip ... thanks also. I did wonder about making the everything explicit. I've seen that mentioned elsewhere. Writing out the stdout & stderr to another file is a great idea. I'll have ago. But I think our dinner guests are starting to make comments so I'd better go! -- http://mail.python.org/mailman/listinfo/python-list
Re: Running a Python script from crontab
Ok ... this is odd.
I tried gregory's suggestion of redirecting the stdout & stderr to a
text file. This worked. I could see all the logging information.
However, there was no error to see this time ... the application
worked completely without any problems.
I also then tried Jon's suggestion of attaching to the console ...
this also allowed me to see all the logging info as it went through.
Again, there were no problems as the application worked without any
problems.
Which suggests that it's the logging module or something with the
output streams.
Here is my logging setup in the primary module. It directs info to the
main console and also to a logging file.
>>>
#Set up logging
log = logging.getLogger('scrape_manager')
log.setLevel(logging.INFO)
my_filename = ss.compact_date() + '_' + time.strftime("%H%M")
ensure_dir('logs/')
#Checks to see if the dir is there and creates it if not
log_file = 'logs/systemlog_' + my_filename + ".log"
console_h = logging.StreamHandler()
file_h = logging.FileHandler(log_file,'w')
log.addHandler(console_h)
log.addHandler(file_h)
console_fmt = logging.Formatter('%(levelname)s (%(threadName)-10s) %
(module)s %(message)s',)
file_fmt = logging.Formatter('%(asctime)s\t%(levelname)s\t(%
(threadName)-10s)\t%(module)s\t%(message)s',)
console_h.setFormatter(console_fmt)
file_h.setFormatter(file_fmt)
>
I also call this from an auxiliary module that is use my the primary
module:
log = logging.getLogger('scrape_manager')
If I change the name of the logger in the auxilary module then I loose
all the loggiing information but the application works.
What am I doing wrong with the logging setup? I can't see how I would
change it.
ALJ
--
http://mail.python.org/mailman/listinfo/python-list
Re: Running a Python script from crontab
I've included a switch to include or exclude the logging to console. When logging only to file, the script runs fine. Of course, I still don't understand why dual logging, and specifically to the console, causes a problem and if anyone has any comments about the dual output logging code above then I'd still be happy to hear about it. ALJ -- http://mail.python.org/mailman/listinfo/python-list
Re: Running a Python script from crontab
On 3 Dec, 16:41, Philip Semanchuk <[EMAIL PROTECTED]> wrote: > On Dec 3, 2008, at 10:29 AM, Astley Le Jasper wrote: > > > I've included a switch to include or exclude theloggingto console. > > Whenloggingonly to file, the script runs fine. > > > Of course, I still don't understand whyduallogging, and specifically > > to the console, causes a problem and if anyone has any comments about > > thedualoutputloggingcode above then I'd still be happy to hear > > about it. > > Trying to write non-ASCII characters perchance? Errmmm ... that's kind of spoookey. I using UTF-8 encoding as I have a lot of European language characters. But would that cause a problem when running from crontab but not in the terminal? Go on then ... spill the beans. -- http://mail.python.org/mailman/listinfo/python-list
Re: Running a Python script from crontab
On 3 Dec, 19:49, Philip Semanchuk <[EMAIL PROTECTED]> wrote: > On Dec 3, 2008, at 1:33 PM, Astley Le Jasper wrote: > > > > > On 3 Dec, 16:41, Philip Semanchuk <[EMAIL PROTECTED]> wrote: > >> On Dec 3, 2008, at 10:29 AM, Astley Le Jasper wrote: > > >>> I've included a switch to include or exclude theloggingto console. > >>> Whenloggingonly to file, the script runs fine. > > >>> Of course, I still don't understand whyduallogging, and specifically > >>> to the console, causes a problem and if anyone has any comments > >>> about > >>> thedualoutputloggingcode above then I'd still be happy to hear > >>> about it. > > >> Trying to write non-ASCII characters perchance? > > > Errmmm ... that's kind of spoookey. I using UTF-8 encoding as I have a > > lot of European language characters. But would that cause a problem > > when running from crontab but not in the terminal? > > > Go on then ... spill the beans. > > Oh, I don't know exactly. It's just what I thought of when you said > that the problem occurs when logging to the console but not to files. > I don't have a deep Unix background so I can't give you the details on > what "the console" means to a cron job. > > In my experience, the environment in which a cron job runs is > different from the environment in which some command line scripts run > (remember my earlier suggestion about needing to explicitly set the > PATH?) and so if the console for a cron job differed from the console > that a Python program sees when run in a terminal, that would not > surprise me. > > Hope it's a useful suggestion > Philip I tried using the path variables but it didn't help. There again, I didn't really understand what was being passed in the path variables and, rather than just copying and pasting, I wanted to look up further info to get to grips with it. I've got something that works, so now the pressure is off I can spend some time looking it up. [sigh ... my experiences with linux haven't been that good so far. It is, if nothing else, interesting] Thanks ALJ -- http://mail.python.org/mailman/listinfo/python-list
Re: Running a Python script from crontab
On Dec 4, 12:34 am, Lawrence D'Oliveiro <[EMAIL PROTECTED] central.gen.new_zealand> wrote: > In message <[EMAIL PROTECTED]>, Philip > > Semanchuk wrote: > > In my experience, the environment in which a cron job runs is > > different from the environment in which some command line scripts run... > > Which is true, but again, cron should report the environment in the mail > message. For example, here are some headers from a recent run of the > maildir backup task I have scheduled twice a day: > > Subject: Cron <[EMAIL PROTECTED]> $HOME/bin/backupdir $HOME/.maildir > X-Cron-Env: > X-Cron-Env: > X-Cron-Env: > X-Cron-Env: > X-Cron-Env: Where do you get the emails from? -- http://mail.python.org/mailman/listinfo/python-list
Finding the instance reference of an object
Sorry for the numpty question ... How do you find the reference name of an object? So if i have this bob = modulename.objectname() how do i find that the name is 'bob' -- http://mail.python.org/mailman/listinfo/python-list
Re: Finding the instance reference of an object
On 16 Oct, 16:52, Carsten Haese <[EMAIL PROTECTED]> wrote:
> Astley Le Jasper wrote:
> > Sorry for the numpty question ...
>
> > How do you find the reference name of an object?
>
> > So if i have this
>
> > bob = modulename.objectname()
>
> > how do i find that the name is 'bob'
>
> Why do you need to find that? You know that its name is 'bob'.
>
> --
> Carsten Haesehttp://informixdb.sourceforge.net
I'm creating mulitple instances, putting them in a list, iterating
through the list to send them to some functions where process them
with some instance specific parameters. Something along the lines of:
bob = someobject()
harry = someobject()
fred = someobject()
parameterdict = {'bob':(0,1,2),'harry':(3,4,5),'fred':(6,7,8)}
people_list = (bob, harry, fred)
for person in people_list:
add_parameters(person)
def add_parameters(person)
mytuple = parameterdict[??instance.name]
person.x = mytuple[0]
person.y = mytuple[1]
person.z = mytuple[2]
... alternatively there is probably a very much easier way of doing
it.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Finding the instance reference of an object
On 16 Oct, 18:53, "Diez B. Roggisch" <[EMAIL PROTECTED]> wrote:
> Astley Le Jasper schrieb:
>
>
>
> > On 16 Oct, 16:52, Carsten Haese <[EMAIL PROTECTED]> wrote:
> >> Astley Le Jasper wrote:
> >>> Sorry for the numpty question ...
> >>> How do you find the reference name of an object?
> >>> So if i have this
> >>> bob = modulename.objectname()
> >>> how do i find that the name is 'bob'
> >> Why do you need to find that? You know that its name is 'bob'.
>
> >> --
> >> Carsten Haesehttp://informixdb.sourceforge.net
>
> > I'm creating mulitple instances, putting them in a list, iterating
> > through the list to send them to some functions where process them
> > with some instance specific parameters. Something along the lines of:
>
> > bob = someobject()
> > harry = someobject()
> > fred = someobject()
>
> > parameterdict = {'bob':(0,1,2),'harry':(3,4,5),'fred':(6,7,8)}
> > people_list = (bob, harry, fred)
>
> > for person in people_list:
> > add_parameters(person)
>
> > def add_parameters(person)
> > mytuple = parameterdict[??instance.name]
> > person.x = mytuple[0]
> > person.y = mytuple[1]
> > person.z = mytuple[2]
>
> > ... alternatively there is probably a very much easier way of doing
> > it.
>
> Why not simply do
>
> bob = someobject(0, 1, 2)
>
> ?
>
> Diez
Because that was pseudo code to demonstrate what I was trying to
achieve. The parameters are more complicated than that.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Finding the instance reference of an object
Thanks for all the responses. That helps. Ta ALJ -- http://mail.python.org/mailman/listinfo/python-list
PySqlite - division of real numbers without decimal fractions
I've been getting errors recently when using pysqlite. I've declared the table columns as real numbers to 2 decimal places (I'm dealing with money), but when doing division on two numbers that happen to have no decimal fractions, the results through pysqlite are coming through as integers. The funny thing is that when looking at the database using SQLite Manager or SQLite Pro the results there are displayed correctly. As a temporary fix I've had to multiply one of the numbers with 1.0 which has fixed it but I wonder if there is something else I'm doing wrong. -- http://mail.python.org/mailman/listinfo/python-list
Re: PySqlite - division of real numbers without decimal fractions
On Nov 7, 6:36 am, Dan Bishop <[EMAIL PROTECTED]> wrote: > On Nov 6, 3:46 pm, Astley Le Jasper <[EMAIL PROTECTED]> wrote: > > > I've been getting errors recently when using pysqlite. I've declared > > the table columns as real numbers to 2 decimal places (I'm dealing > > with money), but when doing division on two numbers that happen to > > have no decimal fractions, the results through pysqlite are coming > > through as integers. The funny thing is that when looking at the > > database using SQLite Manager or SQLite Pro the results there are > > displayed correctly. As a temporary fix I've had to multiply one of > > the numbers with 1.0 which has fixed it but I wonder if there is > > something else I'm doing wrong. > > You're using old-style division. Put the line "from __future__ import > division" in your script. Hi, But the calculations are being done within a sql statement within SQLite? ([actual_price]-[recommended_price])/[recommended_price] -- http://mail.python.org/mailman/listinfo/python-list
Re: Threading on an old machine
Sorry ... that should be:
for sitename in mysites:
log.info("define thread")
thread_list[sitename]=threading.Thread(name=sitename,target=myproceedure,
args=(sitename,))
log.info("done")
thread_list[sitename].start()
log.info("Started")
--
http://mail.python.org/mailman/listinfo/python-list
Re: Threading on an old machine
On 10 Nov, 11:07, Astley Le Jasper <[EMAIL PROTECTED]> wrote:
> Sorry ... that should be:
>
> for sitename in mysites:
> log.info("define thread")
>
> thread_list[sitename]=threading.Thread(name=sitename,target=myproceedure,
> args=(sitename,))
> log.info("done")
> thread_list[sitename].start()
> log.info("Started")
Ok ... I just tried running it in the terminal and it works ... so I
presume there must be an issue with IDLE?
--
http://mail.python.org/mailman/listinfo/python-list
Re: PySqlite - division of real numbers without decimal fractions
On 8 Nov, 05:39, Dennis Lee Bieber <[EMAIL PROTECTED]> wrote: > On Fri, 07 Nov 2008 14:36:52 +0100, Gerhard Häring <[EMAIL PROTECTED]> > declaimed the following in comp.lang.python: > > > Astley Le Jasper wrote: > > > I've been getting errors recently when using pysqlite. I've declared > > > the table columns as real numbers to 2 decimal places (I'm dealing > > > with money), > > > MySQL doesn't have any MONEY type. All it has is INTEGER, REAL, TEXT, > > BLOB and NULL types. > > Did you mean SQLite? > > > > > Perhaps using SQLite's column affinity would help? Let the type be named > > "real" instead of anything fancy: > > If dealing with monetary computations, it might be better to define > converters/adapters for Python's decimal type... Though that may mean > that doing simple SQL arithmetic may not be possible -- might need to > supply a Python function to work the arithmetic with conversion of the > data... > -- > Wulfraed Dennis Lee Bieber KD6MOG > [EMAIL PROTECTED] [EMAIL PROTECTED] > HTTP://wlfraed.home.netcom.com/ > (Bestiaria Support Staff: [EMAIL PROTECTED]) > HTTP://www.bestiaria.com/ Hi, Sorry. I don't get this. I am using numbers to 2dp (it doesn't really matter that it's money or not) and importing them into SQLite where all the views are held. One of the columns is doing the following calculations: ([actual_price]-[recommended_price]) AS [difference] ([actual_price]-[recommended_price])/[recommended_price] AS [difference_proportion] When using a SQLite gui like SQLiteManager I can see the imported data is stored correctly and the column has been calculated correctly. So I'll have something like: [actual_price],[recommended_price],[difference], [difference_proportion] 199.99,299.99,-100.00,-0.34335 100.00,120.00,-100.00,-0.1667 However, when calling the view from pysqlite I get the following results 199.99,299.99,-100.00,-0.34 100.00,120.00,-100.00,0 So the row where both numbers have no decimal fraction are changing to an integer. I looks like there is something going on in between sqlite and pysqlite. -- http://mail.python.org/mailman/listinfo/python-list
Threading on an old machine
I have an application that put on an old machine with a fresh Xubuntu
installation (with Python 2.5). But I can't get the threading to work
The application was written on a combination of Windows XP and
OpenSuse and has been running without any problems using Eclipse/
Pydev. However, now that I try to run the application using IDLE it
just hangs. I have managed to track the point at which it hangs to
the following line:
for sitename in mysites:
log.info("define thread")
thread_list[search_engine]=threading.Thread(name=sitename,
target=myproceedure, args=(sitename,))
log.info("done")
thread_list[search_engine].start()
log.info("Started")
It gets to the "done" and then hangs. It doesn't appear to get to
'myproceedure'.
The machine is an old Pentium II, with 256mb and 3gb spare on the HD.
It doesn't appear to be having any problems with the other scripts
I've ran.
ALJ
--
http://mail.python.org/mailman/listinfo/python-list
Re: Threading on an old machine
On 10 Nov, 16:20, Terry Reedy <[EMAIL PROTECTED]> wrote:
> Astley Le Jasper wrote:
> > I have an application that put on an old machine with a fresh Xubuntu
> > installation (with Python 2.5). But I can't get the threading to work
>
> > The application was written on a combination of Windows XP and
> > OpenSuse and has been running without any problems using Eclipse/
> > Pydev. However, now that I try to run the application using IDLE it
> > just hangs. I have managed to track the point at which it hangs to
> > the following line:
>
> > for sitename in mysites:
> > log.info("define thread")
> > thread_list[search_engine]=threading.Thread(name=sitename,
> > target=myproceedure, args=(sitename,))
> > log.info("done")
> > thread_list[search_engine].start()
> > log.info("Started")
>
> > It gets to the "done" and then hangs. It doesn't appear to get to
> > 'myproceedure'.
>
> Since 'myproceedure' is before 'done', the above is unclear.
I included a log.info in myprocedure. However, it doesn't trigger
because the thread doesn't actually get to the myprocedure ... at
least with Idle anyway.
--
http://mail.python.org/mailman/listinfo/python-list
Extracted files form zip corrupted
I want to batch extract files from a directory of zips. The thing is
that the files are excel spreadsheets. I don't want to read them in
python, just dump them as extracted files in another directory.
However, when I do a test the excel file becomes corrupted. Any clues?
>>
import zipfile
zf = zipfile.ZipFile('C:\Temp\mytest.zip')
filename = 'spreadsheet.xls'
data = zf.read(filename)
uz_file = 'C:\Temp\' + filename
f = open(uz_file, "w")
f.write(data)
f.close()
>>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Extracted files form zip corrupted
Thanks MRAB. What do you mean about backslashes? -- http://mail.python.org/mailman/listinfo/python-list
What do you think of ShowMeDo
Hi, I've just stumbled over this (http://showmedo.com/) and being the very visual person I am, it seems like it could be a good way to learn about python. However, before I smack down $60, I wondered if anyone had any opinions on it. My gut feel is that it could be pretty good. ALJ -- http://mail.python.org/mailman/listinfo/python-list
Re: What do you think of ShowMeDo
Gosh ... it's all gone quite busy about logging in, gui etc. Certainly, I would try to make it clearer what is free and what isn't. But flash ... using that doesn't bother me. Loggin in ... fine ... I don't care as long as it's quick and there is something I might want. i just wanted to know if the content was of a suitable quality that's worth paying $60 for. You'll never get anywhere without also having a good collection of books or online references, but there is something really appealing about watching someone else walk you through. It get you started. It's another perspective on the same topic. I do have a suggestion though. Consider micro-payments. If there is only one video that I might be interested in, then I think I'd be happy to pay $1 (or whatever) for. Lower risk. Lower barrier to entry. I think that the site might be loosing out on a lot of users that may only want one or a few tutorials. And then once you've used it once, you're more likely to use it again. Micropayments could also be the basis of rating the videos. It'll perhaps give an indication of demand, which could guide further development and publication of videos. ALJ -- http://mail.python.org/mailman/listinfo/python-list
Xpath for HTML processing
Can anyone suggest something inthat can process an XPath like the following: "/html/body/table[2]/tbody/tr/td[5]/table/tbody/tr[3]/td/table[3]/ tbody/tr[5]/td" Cheers -- http://mail.python.org/mailman/listinfo/python-list
Re: Good programming style
On 12 Sep, 12:44, Bruno Desthuilliers wrote:
> Astley Le Jasper a écrit :
>
> > I'm still learning python and would like to know what's a good way of
> > organizing code.
>
> > I am writing some scripts to scrape a number of different website that
> > hold similar information and then collating it all together. Obviously
> > each site needs to be handled differently, but once the information is
> > collected then more generic functions can be used.
>
> > Is it best to have it all in one script or split it into per site
> > scripts that can then be called by a manager script?
> > If everything is
> > in one script would you have per site functions to extract the data or
> > generic function that contain vary slightly depending on the site,
>
> As far as I'm concerned, I'd choose the first solution. Decoupling
> what's varying (here, site-specific stuff) from "invariants" is so far
> the best way I know to keep complexity manageable.
>
> > for
> > example
>
> > import firstSiteScript
> > import secondSiteScript
>
> > firstsitedata = firstSiteScript.getData('search_str)
> > secondsitedata = secondSiteScript.getData('search_str)
> > etc etc
>
> Even better :
>
> - put generic functions in a 'generic' module
> - put all site-specific stuff each in it's own module in a specific
> 'site_scripts' directory
> - in your 'main' script, scan the site_scripts directory to loop over
> site-specific modules, import them and run them (look for the __import__
> function).
>
> This is kind of a Q&D lightweight plugin system, that avoids having to
> hard-code imports and calls in the main script, so you just have to
> add/remove site-specific script to/from the site_scripts directory .
>
> Also, imported modules are not recompiled on each import - only when
> they change - while the 'main' script get recompiled on each invocation.
>
> (snip)
>
> > OR
>
> > def getdata(search_str, website):
> > if website == 'firstsite':
> >
> > elif website =='secondsite':
>
> This one is IMHO the very worst thing to do.
>
> My 2 cents...
Excellent, thanks for that.
--
http://mail.python.org/mailman/listinfo/python-list
Noob thread lock question
I have a number of threads that write to a database. I have created a thread lock, but my question is this: - If one thread hits a lock, do a) all the other threads stop, or b) just the ones that come to the same lock? - I presume that the answer is b. In which case do the threads stop only if they come to the same instance of a lock. For example, you could have a lock instance for one database and another instance for another database (first_db_thread_lock = threading.RLock() second_db_thread_lock = threading.RLock()). I appreciate this is a bit of a noob question, but I didn't want to assume anything. ALJ -- http://mail.python.org/mailman/listinfo/python-list
Re: Noob thread lock question
Cheers for the responses. ALJ -- http://mail.python.org/mailman/listinfo/python-list
Re: Noob thread lock question
When you say don't forget about the GIL, what should I not be forgetting? I'm using sqlite and the following: <<> thread_lock = threading.RLock() def db_execute(sql): thread_lock.acquire() try: connection = sqlite3.connect(database_name) cursor = connection.cursor() cursor.execute(sql) connection.commit() except: log.error(formatExceptionInfo()) finally: thread_lock.release() <<> -- http://mail.python.org/mailman/listinfo/python-list
Python, Reportlabs, Pil and Windows 7 (64bit)
I have a Windows 7 (64bit AMD) machine and am having quite a lot of problems installing Reportlabs and Pil. I wondered if anyone else has had the same issues and what the best way of dealing with it. So far I've tried: 1. Reportlabs / Pil 32 installers - I've tried using these but they can't find python. I also tried registering Python (http://effbot.org/ zone/python-register.htm) but this also fails. 2. Reportlabs / Pil Source - I downloaded each of these and tried to do a "python setup.py install". However, both complain that they can't find "vcvarsall.bat". I've done some checking and it's because the compiler isn't present. Everyone is suggesting downloading Visual Studio Express C++, but this only comes with the 32bit compiler. There seems to be quite a lot of work to get 64bit VSE working on a 64bit machine (http://jenshuebel.wordpress.com/2009/02/12/visual-c-2008- express-edition-and-64-bit-targets/). But before I start down that path, I wondered if anyone had any advice ( and no I don't mean suggesting I swap to Linux). ALJ -- http://mail.python.org/mailman/listinfo/python-list
Re: Python, Reportlabs, Pil and Windows 7 (64bit)
@Robin Thanks. I thought that this seemed to be a general python thing because it was effecting both installs. However, after also reading Martin's comments ... @Martin > This is somewhat imprecise: is it > a) that your CPU is AMD64, and thus supports 64-bit mode, or > b) that *in addition*, your Windows 7 installation is a 64-bit > installation, or > c) that *in addition*, your Python installation is also a 64-bit >installation. > > Unless you have a specific need for 64-bit mode, I recommend that you > use the 32-bit version of Windows (unless you have more than 4GB of > main memory), and (even if you have a 64-bit Windows) you install the > 32-bit version of Python on it (unless you have the need to access more > than 2GB of objects in your Python applications. Sorry. I have Windows 7 (64-bit) installed on a machine with an AMD cpu (which supports 64-bit mode), with a 64-bit version of (Activestate) python 2.6 although I didn't realise the later until I looked just now. > Install the 32-bit version of Python, and these installers should work fine. Well, I uninstalled the 64-bit version and the installers did indeed work. I’m sorry everyone. I didn’t realise I had installed the 64-bit version of Python. Well, at least someone else might find have the same problem. But I think that there is going to be a bit of a rough patch as everyone moves over to 64-bit. ALJ -- http://mail.python.org/mailman/listinfo/python-list
Re: Python, Reportlabs, Pil and Windows 7 (64bit)
Hi Robin, It looks like you've been busy. I'm sorry but you are well over my head at the moment! :-) If you need me to test an install then I'd be happy to help. However, I just received an email from Christoph Gohlke saying: " ... There are 64 bit versions of Reportlab and PIL for Python 2.6 for Windows available at http://www.lfd.uci.edu/~gohlke/pythonlibs/ ..." Regards Neil -- http://mail.python.org/mailman/listinfo/python-list
Default if none
I realise I could roll my own here, but I wondered if there was an inbuilt version of this? >. def default_if_none(*args): for arg in args: if arg: return arg return None x = None y = 5 z = 6 print default_if_none(x,y,z) >> 5 -- http://mail.python.org/mailman/listinfo/python-list
Re: Default if none
... oh ... that simple. Now I feel dumb. Thanks! ALJ -- http://mail.python.org/mailman/listinfo/python-list
Creating a single list
This is probably a really silly question but, given the example code
at the bottom, how would I get a single list?
What I currently get is:
('id', 20, 'integer')
('companyname', 50, 'text')
[('focus', 30, 'text'), ('fiesta', 30, 'text'), ('mondeo', 30,
'text'), ('puma', 30, 'text')]
('contact', 50, 'text')
('email', 50, 'text')
what I would like is:
('id', 20, 'integer')
('companyname', 50, 'text')
('focus', 30, 'text'),
('fiesta', 30, 'text'),
('mondeo', 30, 'text'),
('puma', 30, 'text'),
('contact', 50, 'text')
('email', 50, 'text')
SAMPLE CODE>>>
def getproducts():
temp_list=[]
product_list=['focus','fiesta','mondeo','puma'] #usually this
would come from a db
for p in product_list:
temp_list.append((p,30,'text'))
return temp_list
def createlist():
column_title_list = (
("id",20,"integer"),
("companyname",50,"text"),
getproducts(),
("contact",50,"text"),
("email",50,"text"),
)
return column_title_list
for item in createlist():
print item
>>>
--
http://mail.python.org/mailman/listinfo/python-list
Re: Creating a single list
On May 29, 4:10 pm, superpollo wrote:
> Astley Le Jasper ha scritto:
>
>
>
> > This is probably a really silly question but, given the example code
> > at the bottom, how would I get a single list?
>
> > What I currently get is:
>
> > ('id', 20, 'integer')
> > ('companyname', 50, 'text')
> > [('focus', 30, 'text'), ('fiesta', 30, 'text'), ('mondeo', 30,
> > 'text'), ('puma', 30, 'text')]
> > ('contact', 50, 'text')
> > ('email', 50, 'text')
>
> > what I would like is:
>
> > ('id', 20, 'integer')
> > ('companyname', 50, 'text')
> > ('focus', 30, 'text'),
> > ('fiesta', 30, 'text'),
> > ('mondeo', 30, 'text'),
> > ('puma', 30, 'text'),
> > ('contact', 50, 'text')
> > ('email', 50, 'text')
>
> > SAMPLE CODE>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > def getproducts():
> > temp_list=[]
> > product_list=['focus','fiesta','mondeo','puma'] #usually this
> > would come from a db
> > for p in product_list:
> > temp_list.append((p,30,'text'))
> > return temp_list
>
> > def createlist():
> > column_title_list = (
> > ("id",20,"integer"),
> > ("companyname",50,"text"),
> > getproducts(),
> > ("contact",50,"text"),
> > ("email",50,"text"),
> > )
> > return column_title_list
>
> > for item in createlist():
> > print item
>
> >>> def createlist():
> ... column_title_list = [
> ... ("id",20,"integer"),
> ... ("companyname",50,"text")]
> ... column_title_list += getproducts()
> ... column_title_list += [
> ... ("contact",50,"text"),
> ... ("email",50,"text")]
> ... return column_title_list
> ...
> >>> for item in createlist():
> ... print item
> ...
> ('id', 20, 'integer')
> ('companyname', 50, 'text')
> ('focus', 30, 'text')
> ('fiesta', 30, 'text')
> ('mondeo', 30, 'text')
> ('puma', 30, 'text')
> ('contact', 50, 'text')
> ('email', 50, 'text')
> >>>
>
> bye
Cheers for that. I was thinking there might be something clever, but
as usual, simple is better.
--
http://mail.python.org/mailman/listinfo/python-list
Does MS Office need to be installed to use pywin32 for editing Excel docs?
I'm creating excel docs on the fly using XLWT. However, I need to include filters which I believe can only be done with pywin32. Does pywin32 use elements from Windows itself, or excel when dispatching? -- http://mail.python.org/mailman/listinfo/python-list
Re: Does MS Office need to be installed to use pywin32 for editing Excel docs?
On Jun 15, 10:44 am, Tim Golden wrote:
> On 15/06/2010 09:32, Astley Le Jasper wrote:
>
> > Does pywin32 use elements from Windows itself, or excel when
> > dispatching?
>
> Yes: it's simply exposing to the Python user the API provided
> by MS Office (or whatever other app) via the IDispatch COM
> mechanism.
>
> IOW, if you don't have Microsoft Excel installed, this won't work:
>
> import win32com.client
> win32com.client.Dispatch ("Excel.Application")
>
> TJG
Hi Tim,
Thanks for that.
ALJ
--
http://mail.python.org/mailman/listinfo/python-list
Grouping pairs - suggested tools
I have a list of tuples that indicate a relationship, ie a is related to b, b is related to c etc etc. What I want to do is cluster these relationships into groups. An item will only be associated with a single cluster. Before I started, I wondered if there was any particular tool within Python I should be looking at. I don't expect anyone to code this for me, just say ... "you need to look at using x". I was going to use populate a dictionary and Sorry for being so vague. Example Data: [(a,b) (a,c) (a,d) (b,c) (b,d) (c,d) (e,f) (e,g) (f,g) (h,i)] Output (grouping id, item ref) (1,a), (1,b), (1,c), (1,d), (2,e), (2,f), (2,g), (3,h), (3,i) -- http://mail.python.org/mailman/listinfo/python-list
Re: Grouping pairs - suggested tools
Thanks for the feedback both. I'll have a look at these. One of the frustrating things I find about python is that there are so many modules. There have been times when I've spend ages doing something and then some has said, "Oh yeah, there is a module for that". Anyway. Nearly finished. Cheers ALJ -- http://mail.python.org/mailman/listinfo/python-list
Re: Grouping pairs - suggested tools
Thanks all. I was playing around with this and came up with another
solution using dictionaries (... comfort zone ...!!)
<<< Code
>>
from operator import itemgetter
def group_stuff(data):
group_dic = {}
group_id = 0
for line in data:
i = 0
for ref in line:
if group_dic.get(ref) is None:
if group_dic.get(line[1-i]) is not None:
group_id = group_dic[line[1-i]]
else:
group_id +=1
group_dic[ref] = group_id
i+=1
group_list = []
for id, group in sorted(group_dic.items(), key=itemgetter(1,0)):
group_list.append((group, id))
return group_list
if __name__ == '__main__':
data = [('a','b'),('a','c'),('a','d'),('b','c'),('b','d'),
('c','d'),('e','f'),('e','g'),('f','g'),('h','i')]
grouped = group_stuff(data)
print grouped
<<< Code
>>
Output: [(1, 'a'), (1, 'b'), (1, 'c'), (1, 'd'), (2, 'e'), (2, 'f'),
(2, 'g'), (3, 'h'), (3, 'i')]
--
http://mail.python.org/mailman/listinfo/python-list
Re: Grouping pairs - suggested tools
> I think you have the same bug as Alf's code, you never merge existing
> groups. Have you tried Arnaud's counterexample?
>
> By the way, are ('a', 'b') and ('b', 'a') to be considered equivalent for
> your problem?
>
> Peter
Hi Peter,
Yes. I realise that this doesn't take into account existing
relationships/groupings. For the particular situation that I am
looking at this is unlikely to happen and/or not critical. However, I
understand that generally you would want to potentially assign an item
to one or more groups.
In my case the ('a','b') and ('b','a') are equivalent. Perhaps for
anyone else looking at this, I can elaborate on the problem to make it
a bit more concrete.
I have a very long listing of customer details and am trying to clean
the data. In particular I am looking for duplicates:
<< Core Data >>
id, Name, Address
a, Acme Ltd, 1 Main Street
b, Acme Limited, 1 Main St
c, Acme L'td, 1 Main Street
d, Smiths, 22 Upper Road
e, Smyths, 22 Upper Rd
f, Smiths ltd, 22 Upperrd
g, Apple Empire, 222 Lower Way
h, Apple Emp, 222 Lower Way
Obviously this is oversimplified. The actual dataset has thousands of
records. I am using the difflib module and comparing each item against
all those below it, and where the items are similar they are stored in
a paired table
<< Paired Data >>
id1, id2, relationship_strength
a, b, 0.8
a, c, 0.88
b, c, 0.8
d, e, 0.75
d, f, 0.88
e, f, 0.87
g, h, 0.77
However, these pairing aren't so easy to read and I want to include
the full address and account information so it's easier for the lucky
person who is going to clean the data. And this is where the grouping
cluster comes in.
<< Grouped Data >>
group_id, id
1, a
1, b
1, c
2, d
2, e
2, f
3, g
3, h
So in my situation those records that are very similar to each other
will be clustered together.
Thanks again.
ALJ
--
http://mail.python.org/mailman/listinfo/python-list
Multilingual documentation solutions
My query isn't specific to Python, but some of you might have been in a similar position and if there are any technical solutions, I'd prefer to do it in Python. I have developed an application using Django and the supporting documentation needs to be multilingual. This isn't at code level, rather for staff and administrators. I wondered if there a best practice out there or any tools people are using. At the moment I'm producing a word document with screenshots that gets translated, but this is getting very difficult to control, especially tracking small content changes and translations. -- http://mail.python.org/mailman/listinfo/python-list
Re: Multilingual documentation solutions
Thanks for the pointer. I'll have a look into it. -- http://mail.python.org/mailman/listinfo/python-list
