You might try running your Python process with: strace -f -s 1024 -o /tmp/script.strace python /path/to/script.py
Then you (perhaps with a C programmer) can likely track down what happened right before the crash by examining the system call tracer near the end of the file. http://stromberg.dnsalias.org/~strombrg/debugging-with-syscall-tracers.html On Mon, Apr 9, 2012 at 3:01 AM, Janis <janis.vik...@gmail.com> wrote: > Hello! > > I have this problem with my script exiting randomly with Linux OS > status code -9 (most often) or -15 (also sometimes, but much more > rarely). As far as I understand -9 corresponds to Bad file descriptor > and -15 Block device required. > > 1) Is there a way how I could find out what exactly causes Python > process to exit? > 2) What could be the reason of Python exiting with these status code? > > The script is a crawler that crawls several web sites to download web > pages and extract information. Most often it exits after having run > for 2 hours and having downloaded ~24 000 files. Some specific web > sites are more affected than others, i.e., there are other instances > of the script running in parallel that download more pages and > complete normally. That could be related to the speed each page is > returned etc. > > I have a try-catch block in the root of the script which works very > well to catch any kind of exceptions, but in these cases either Python > does not catch the exception or fails to log it into MySQL error log > table. > > I can not use debugger because the code exits after it has run for an > hour or more. In order to try to catch the exact place I have put > logging after each line in some places. The logging prints to stdout, > to a file and logs the line into MySQL DB. This has revealed that code > may exit upon a random simple lines such as: time.sleep(0.1) - most > often, f = opener.open(request) - also often, but sometimes also such > simple statements as list.add('string'). > > It can also be that the problem occues and then the script exits upon > attempt to do any output - stdout (i.e. regular print '1'), writing to > file and logging into MySQL. I have changed either of these to be the > first ones in the debug log function, but each time the script did not > fail in between these lines. > > The main libraries that are in use: MySQLDB, urllib and urllib2, but > none of them seems to be the direct cause of the problem, there is no > direct call of any of these upon exit. I suspect that this could be > related to some garbadge collecting process. > > Versions tried with: Python 2.6, Python 2.7 (and I think it happened > also in Python 2.5, but I'm not sure) > > Thanks, > Janis > -- > http://mail.python.org/mailman/listinfo/python-list >
-- http://mail.python.org/mailman/listinfo/python-list
