Python randomly exits with Linux OS error -9 or -15
Hello!
I have this problem with my script exiting randomly with Linux OS
status code -9 (most often) or -15 (also sometimes, but much more
rarely). As far as I understand -9 corresponds to Bad file descriptor
and -15 Block device required.
1) Is there a way how I could find out what exactly causes Python
process to exit?
2) What could be the reason of Python exiting with these status code?
The script is a crawler that crawls several web sites to download web
pages and extract information. Most often it exits after having run
for 2 hours and having downloaded ~24 000 files. Some specific web
sites are more affected than others, i.e., there are other instances
of the script running in parallel that download more pages and
complete normally. That could be related to the speed each page is
returned etc.
I have a try-catch block in the root of the script which works very
well to catch any kind of exceptions, but in these cases either Python
does not catch the exception or fails to log it into MySQL error log
table.
I can not use debugger because the code exits after it has run for an
hour or more. In order to try to catch the exact place I have put
logging after each line in some places. The logging prints to stdout,
to a file and logs the line into MySQL DB. This has revealed that code
may exit upon a random simple lines such as: time.sleep(0.1) - most
often, f = opener.open(request) - also often, but sometimes also such
simple statements as list.add('string').
It can also be that the problem occues and then the script exits upon
attempt to do any output - stdout (i.e. regular print '1'), writing to
file and logging into MySQL. I have changed either of these to be the
first ones in the debug log function, but each time the script did not
fail in between these lines.
The main libraries that are in use: MySQLDB, urllib and urllib2, but
none of them seems to be the direct cause of the problem, there is no
direct call of any of these upon exit. I suspect that this could be
related to some garbadge collecting process.
Versions tried with: Python 2.6, Python 2.7 (and I think it happened
also in Python 2.5, but I'm not sure)
Thanks,
Janis
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python randomly exits with Linux OS error -9 or -15
Thank you all for the help! I will need to think a bit about the other suggestions. But, Alan, as to this: > > How do you get -9 and -15? Exit status is supposed to be between 0 and > > 127. I have the following code that has caught these: p = subprocess.Popen([Config.PYTHON_EXE,'Load.py',"%s" % (row[1],)], bufsize=0, executable=None, stdin=None, stdout=None, stderr=subprocess.PIPE, preexec_fn=None, close_fds=False, shell=False, cwd='../Loader/') stdout, stderr = p.communicate() if p.returncode != 0: ... So you say it's SIGKILL and SIGTERM? Then I guess these could be misleading statuses from those cases when I have terminated the sessions myself, and when there is the real problem apparently the caller detected nothing here. SIGKILL and SIGTERM would probably also explain why there was nothing in stderr. Thanks, Janis P.S. the problem seems to occure also when I call the process directly and not via Popen from another script. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python randomly exits with Linux OS error -9 or -15
I have confirmed that the signal involved is SIGKILL and, yes, apparently OS is simply running out of memory. Thank you all, again! Best Regards, Janis -- http://mail.python.org/mailman/listinfo/python-list
Re: using subprocess.Popen does not suppress terminal window on Windows
Hi, and I'm sorry for using this old thread, but I'm experiencing the same
problem, except, that I wan't to execute any shell script or exe without blank
terminal window.
Is there any way in python to supress blank console screen while script is
executing? Multiplatform solution would be nice.
With best regards, Jānis.
On Friday, June 18, 2010 7:23:19 PM UTC+3, Steven wrote:
> I am calling a ruby program from a python gui and using
> subprocess.Popen in Windows XP using python 2.6. Unfortunately,
> whenever the ruby program is called a blank command window appears on
> screen, annoying my users. Is there a way to suppress this behaviour?
>
> Below is a minimal program that demonstrates the problem. The problem
> does not manifest if the python program is launched via the command
> line. To duplicate launch from Windows Explorer by double-clicking on
> the python file.
>
> --- call_double.pyw ---
> from subprocess import *
> import time
>
> time.sleep(3) # to show that command window is result of call to Popen
> p = Popen(['ruby.exe', 'double.rb'], stdin=PIPE, stdout=PIPE,
> stderr=PIPE)
> results = open('results.txt', 'w')
> for n in range(10):
> p.stdin.write("%d\n" % n)
> result = p.stdout.readline().strip()
> results.write('double(%s) => %2s\n' % (n, result))
> results.close()
>
> --- end of call_double.pyw ---
>
> --- double.rb ---
> while true
> puts $stdin.gets().strip!.to_i * 2
> STDOUT.flush
> end
> --- end of double.rb ---
>
> thanks for any help,
> Steven Rumbalski
I
--
http://mail.python.org/mailman/listinfo/python-list
Re: using subprocess.Popen does not suppress terminal window on Windows
Thanks for answer, but that's not helping. I'm making a little embedded system programming IDE so I need to run .exe(windows only), make commands, perl & python scripts etc(multiplatform). I'm using subprocess.Popen for all of them and it works fine except that blank console window and btw it pop's out under linux too. Maybe the problem is that original python script has .pyw extension, so it hides his own console, but I don't need thatone too. P.S. If it makes a diffrence I'm using wxPython 2.9. & Python 2.7.2. -- http://mail.python.org/mailman/listinfo/python-list
Re: using subprocess.Popen does not suppress terminal window on Windows
It looks like normal terminal to me, could You define normal? Looks like it appears only when target script prints something, but it shouldn't cus I'm using pipes on stdout and stderr. If anyone is interested I'm using function doPopen from here: http://code.google.com/p/mansos/source/browse/trunk/tools/IDE/src/helperFunctions.py -- http://mail.python.org/mailman/listinfo/python-list
Seome kind of unblocking input
Hello! I'm building small console like program for embedded system control over serial port. Naturally I need to be able to recieve commands from user and print reply's from embedded device. Since I'm using threads and pipes everything works ok, except that when i call input() there is no way that I could print something, is there any workaround for this?? Note: I don't need to catch any key's before enter or smtng, just be able to print while input() is waiting. I'm thinking that maybe there is a way for two threads to share one stdout, which should resolve this, but I can't make it work, since U can't pickle file like object(stdout) to pass it to other thread. Note 2: I've readed about ways to make nonblocking input by reading single char etc. but that's is all messy and very platform dependent, I would love to have platform independent solution. Thanks in advance! --JJ -- http://mail.python.org/mailman/listinfo/python-list
Re: Seome kind of unblocking input
Thanks for reply's. I'll be looking into threading, it seems like right way to go. btw. Why Python developers don't make a wrapper for input() with callback function using threads, so people can easily use nonblocking input? -- http://mail.python.org/mailman/listinfo/python-list
Re: Quickie - Regexp for a string not at the beginning of the line
On 25.10.2012 22:53, Rivka Miller wrote: > Hello Programmers, > > I am looking for a regexp for a string not at the beginning of the > line. > > For example, I want to find $hello$ that does not occur at the > beginning of the string, ie all $hello$ that exclude ^$hello$. .hello The dot represents any character. But for specific strings that needs adjustments (e.g. looking for hh not at the beginning of a line would require something like ^[^h]+hh - ah, well, you wrote something similar below). Janis > > In addition, if you have a more difficult problem along the same > lines, I would appreciate it. For a single character, eg < not at the > beginning of the line, it is easier, ie > > ^[^<]+< > > but I cant use the same method for more than one character string as > permutation is present and probably for more than one occurrence, > greedy or non-greedy version of [^<]+ would pick first or last but not > the middle ones, unless I break the line as I go and use the non- > greedy version of +. I do have the non-greedy version available, but > what if I didnt? > > If you cannot solve the problem completely, just give me a quick > solution with the first non beginning of the line and I will go from > there as I need it in a hurry. > > Thanks > > -- http://mail.python.org/mailman/listinfo/python-list
Re: Quickie - Regexp for a string not at the beginning of the line
Am 26.10.2012 06:45, schrieb Rivka Miller: Thanks everyone, esp this gentleman. Who is "this"? The solution that worked best for me is just to use a DOT before the string as the one at the beginning of the line did not have any char before it. Which was what I suggested, and where you rudely answered... no one has really helped yet. And obviously... I am a satisfied custormer. ...your perception about yourself and about the role of us Usenet posters seems also not be very sane. Good luck. -- http://mail.python.org/mailman/listinfo/python-list
Simple webserver
I am pondering about writing a client/server software with websockets as communication protocol. The clients will run in browser as Javascript programs and the server may be in any (any sensible) programming language running standalone to be connected remotely by the browser-based JS clients. I found a Python sample[*] but I am neither familiar with Python nor with the 'simple_websocket_server' package that is used in that sample code. But the code looks so simple that I'm considering to learn and use Python for the task. The requirements I have are quite simple; I want to get the client "address"/identifier from an incoming message, store it in a list, and send responses to all active clients for which addresses have been stored. Can anyone tell me whether a simple extension of that "echo incoming message" sample[*] would be easily possible with Python and with that 'simple_websocket_server' package used? Thanks for any hints (or search keywords, or code samples)! Janis [*] https://pypi.org/project/simple-websocket-server/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Simple webserver
On 19.10.2023 01:23, Chris Angelico wrote: > [snip] > > Hope that's enough to get you started! I'd be delighted to help > further if you run into difficulties. Thanks for your quick reply, Chris! This is already great information! I'll dive into your resources soon, and I also appreciate your offer and will probably come back soon with a question... - Thanks again! Janis -- https://mail.python.org/mailman/listinfo/python-list
Re: Simple webserver
On 19.10.2023 01:23, Chris Angelico wrote: > > Broadly speaking, your ideas are great. Any programming language CAN > be used for the server (and I've used several, not just Python). Out of curiosity; what where these languages? - If there's one I already know I might save some time implementing the server. :-) Janis -- https://mail.python.org/mailman/listinfo/python-list
Re: Simple webserver
On 20.10.2023 23:05, Paul Rubin wrote: > Janis Papanagnou writes: >> I found a Python sample[*] but I am neither familiar with >> Python nor with the 'simple_websocket_server' package that >> is used in that sample code. But the code looks so simple >> that I'm considering to learn and use Python for the task. > > I've generally used ThreadingServer(SocketServer) for this purpose > and I think threads are less confusing than async, and performance is > fine if the concurrency level is not too high. But, trying to write a > web server in Python if you don't know Python doesn't seem like a great > idea, except as a learning project. I have a couple decades experience with about a dozen programming languages (not counting assemblers). Asynchronous processing, IPC, multi-processing, client/server architectures, multi-threading, semaphores, etc. etc. are concepts that are not new to me. I'm not, literally, intending to write a web-server. It's a JS application that is running in (browser based) clients, and the server is just centrally coordinating the client applications. My expectation would be that any sophistically designed socket/ web-socket library would not impose any risk. And the intended server by itself has only very limited requirements; listening to incoming request, storing some client information, broadcasting to the attached clients. Basically just (informally written): init server forever: wait for request(s) -> queue handle requests from queue (sequentially): store specific information from new registered clients broadcast some information to all registered clients It seems to me that multi-threading or async I/O aren't necessary. I'd like to ask; where do you see the specific risks with Python (as language per se) and it's (web-socket-)libraries here? If the web-socket IPC is well supported the algorithmic parts in Python seem trivial to learn and implement. - Or am I missing something? (A brief search gave me the impression that for JS communication web-sockets would be the method to use. Otherwise I'd just use basic Unix domain sockets for the purpose and write it, say, in C or C++ that I already know. But I don't know whether (or how) plain sockets are used from JS running in a browser. Here I'm lacking experience. And that lead me to have a look at Python, since the web-sockets/server examples that I found looked simple.) Janis -- https://mail.python.org/mailman/listinfo/python-list
