Re: fork/exec & close file descriptors

The subprocess module uses upper bound MAXFD which is defined as try: MAXFD = os.sysconf("SC_OPEN_MAX") except: MAXFD = 256 /Jean -- https://mail.python.org/mailman/listinfo/python-list

Re: fork/exec & close file descriptors

On Thu, Jun 4, 2015 at 6:07 AM, wrote: > On Wed, Jun 3, 2015, at 09:32, Chris Angelico wrote: >> Write an editor that opens a file and holds it open until the user's >> done with it. Have something that lets you shell out for whatever >> reason. Then trigger the shell-out, and instantly SIGSTOP t

Re: fork/exec & close file descriptors

random...@fastmail.us: > On Wed, Jun 3, 2015, at 10:43, Marko Rauhamaa wrote: >> However, the child process needs to be prepared for os.close() to >> block indefinitely because of an NFS problem or because SO_LINGER has >> been specified by the parent, for example. Setting the close-on-exec >> fla

Re: fork/exec & close file descriptors

On Wed, Jun 3, 2015, at 10:43, Marko Rauhamaa wrote: > However, the child process needs to be prepared for os.close() to block > indefinitely because of an NFS problem or because SO_LINGER has been > specified by the parent, for example. Setting the close-on-exec flag > doesn't help there. Out of

Re: fork/exec & close file descriptors

On Wed, Jun 3, 2015, at 09:32, Chris Angelico wrote: > Write an editor that opens a file and holds it open until the user's > done with it. Have something that lets you shell out for whatever > reason. Then trigger the shell-out, and instantly SIGSTOP the child > process, before it does its work -

Re: fork/exec & close file descriptors

alister : > I meant the program that is supplying your app with file handles > willy- nilly without caring what happens to them You seem to be advocating a strategy whereby the application keeps close track of all file descriptors and closes them individually as needed. Thing is, processes can b

Re: fork/exec & close file descriptors

Marko Rauhamaa : > So the strategy you proposed is the right one: have the child process > ignore any possible errors from os.close(). The parent will have an > opportunity to deal with them. > > And now Linux is back in the good graces, only the man page is > misleading. However, the child proce

Re: fork/exec & close file descriptors

Marko Rauhamaa : > random...@fastmail.us: > >> Why does the child process need to report the error at all? The parent >> process will find out naturally when *it* tries to close the same file >> descriptor. > > That's not how it goes. > > File descriptors are reference counted in the Linux kernel.

Re: fork/exec & close file descriptors

On Wed, Jun 3, 2015 at 11:21 PM, wrote: > On Wed, Jun 3, 2015, at 09:08, Marko Rauhamaa wrote: >> random...@fastmail.us: >> >> > Why does the child process need to report the error at all? The parent >> > process will find out naturally when *it* tries to close the same file >> > descriptor. >> >

Re: fork/exec & close file descriptors

On Wed, Jun 3, 2015, at 09:08, Marko Rauhamaa wrote: > random...@fastmail.us: > > > Why does the child process need to report the error at all? The parent > > process will find out naturally when *it* tries to close the same file > > descriptor. > > That's not how it goes. > > File descriptors a

Re: fork/exec & close file descriptors

random...@fastmail.us writes: > On Wed, Jun 3, 2015, at 03:11, Alain Ketterlin wrote: >> Thank you, I know this. What I mean is: what are the reasons that you >> cannot access your file descriptors one by one? To me closing a range of >> descriptors has absolutely no meaning, simply because ranges

Re: fork/exec & close file descriptors

random...@fastmail.us: > Why does the child process need to report the error at all? The parent > process will find out naturally when *it* tries to close the same file > descriptor. That's not how it goes. File descriptors are reference counted in the Linux kernel. Closes are no-ops except for

Re: fork/exec & close file descriptors

On Wed, 03 Jun 2015 15:27:19 +0300, Marko Rauhamaa wrote: > alister : > >> from the scenario Marco is reporting I get the impression that he is >> having to interact with a system that is fundamentally flawed from the >> ground up. > > Well, yes. It's called linux, but it's not all bad. I just t

Re: fork/exec & close file descriptors

On Wed, Jun 3, 2015, at 08:25, Marko Rauhamaa wrote: > Steven D'Aprano : > > > How does the child process know what action didn't complete? What > > error message are you going to display to the user? > > You definitely must not use sys.stderr from the child process nor are > you allowed to exit

Re: fork/exec & close file descriptors

alister : > from the scenario Marco is reporting I get the impression that he is > having to interact with a system that is fundamentally flawed from the > ground up. Well, yes. It's called linux, but it's not all bad. I just think that man page was being sanctimonious. Marko -- https://mail.p

Re: fork/exec & close file descriptors

Steven D'Aprano : > How does the child process know what action didn't complete? What > error message are you going to display to the user? > > "Error when closing file descriptor 123456" > > What action do you think the user can take on seeing this error > message? Besides, even if you wanted to

Re: fork/exec & close file descriptors

On Wed, Jun 3, 2015, at 03:11, Alain Ketterlin wrote: > Thank you, I know this. What I mean is: what are the reasons that you > cannot access your file descriptors one by one? To me closing a range of > descriptors has absolutely no meaning, simply because ranges have no > meaning for file descript

Re: fork/exec & close file descriptors

On Wed, 03 Jun 2015 22:07:47 +1000, Steven D'Aprano wrote: > On Wed, 3 Jun 2015 07:38 pm, alister wrote: > >> On Wed, 03 Jun 2015 10:41:44 +0300, Marko Rauhamaa wrote: > [...] >>> Here's the deal: the child process is saddled with file descriptors it >>> never wanted in the first place. It can't

Re: fork/exec & close file descriptors

On Wed, 3 Jun 2015 07:38 pm, alister wrote: > On Wed, 03 Jun 2015 10:41:44 +0300, Marko Rauhamaa wrote: [...] >> Here's the deal: the child process is saddled with file descriptors it >> never wanted in the first place. It can't decline them. Now you're >> saying it can't even dispose of them. >>

Re: fork/exec & close file descriptors

On Wed, 03 Jun 2015 10:41:44 +0300, Marko Rauhamaa wrote: > Alain Ketterlin : > >> Marko Rauhamaa writes: >>> Maybe close() will fail for ever. >> >> Your program has to deal with this, something is going wrong, it can't >> just close and go on. > > Here's the deal: the child process is saddled

Re: fork/exec & close file descriptors

Alain Ketterlin : > Marko Rauhamaa writes: >> Maybe close() will fail for ever. > > Your program has to deal with this, something is going wrong, it can't > just close and go on. Here's the deal: the child process is saddled with file descriptors it never wanted in the first place. It can't decl

Re: fork/exec & close file descriptors

Marko Rauhamaa writes: > Alain Ketterlin : > >> Marko Rauhamaa writes: >>> First, if close() fails, what's a poor program to do? >> >> Warn the user? Not assume everything went well? It all depends on the >> application, and what the file descriptor represents. > > The problem here is in the sys

Re: fork/exec & close file descriptors

Chris Angelico writes: > On Wed, Jun 3, 2015 at 7:06 AM, Alain Ketterlin > wrote: >> I've no idea what the OP's program was doing, so I'm not going to split >> hairs. I can't imagine why one would like to mass-close an arbitrary set >> of file descriptors, and I think APIs like os.closerange() a

Re: fork/exec & close file descriptors

Folks, I'm not interested in rehashing this. I don't know what all the open file descriptors are. Some/many/most will have been opened by underlying C++ libraries and will not have been opened by code at the Python level. I do think the addition of an os.fsync() attempt on each file descriptor befo

Re: fork/exec & close file descriptors

On Wed, Jun 3, 2015 at 7:06 AM, Alain Ketterlin wrote: > I've no idea what the OP's program was doing, so I'm not going to split > hairs. I can't imagine why one would like to mass-close an arbitrary set > of file descriptors, and I think APIs like os.closerange() are toxic and > an appeal to slop

Re: fork/exec & close file descriptors

Alain Ketterlin : > Marko Rauhamaa writes: >> First, if close() fails, what's a poor program to do? > > Warn the user? Not assume everything went well? It all depends on the > application, and what the file descriptor represents. The problem here is in the system call contract, which is broken.

Re: fork/exec & close file descriptors

Marko Rauhamaa writes: > Alain Ketterlin : > >> The close(2) manpage has the following warning on my Linux system: >> >> | Not checking the return value of close() is a common but >> | nevertheless serious programming error. It is quite possible that >> | errors on a previous write(2) operation a

Re: fork/exec & close file descriptors

Alain Ketterlin : > The close(2) manpage has the following warning on my Linux system: > > | Not checking the return value of close() is a common but > | nevertheless serious programming error. It is quite possible that > | errors on a previous write(2) operation are first reported at the > | fina

Re: fork/exec & close file descriptors

On 2015-06-02, Marko Rauhamaa wrote: > Skip Montanaro : > >> On Tue, Jun 2, 2015 at 10:28 AM, Marko Rauhamaa wrote: >>> >>> The only problem is that you don't know how high you need to go in >>> general. >> >> Sure, but I didn't know anyway, so no matter what upper bound I choose >> (or what func

Re: fork/exec & close file descriptors

Skip Montanaro : > On Tue, Jun 2, 2015 at 10:28 AM, Marko Rauhamaa wrote: >> >> The only problem is that you don't know how high you need to go in >> general. > > Sure, but I didn't know anyway, so no matter what upper bound I choose > (or what function I choose/implement), it's just going to be

Re: fork/exec & close file descriptors

Skip Montanaro writes: > Reviving (and concluding) a thread I started a couple weeks ago, I asked: > >> The basic fork/exec dance is not a problem, but how do I discover >> all the open file descriptors in the new child process to make sure >> they get closed? Do I simply start at fd 3 and call o

Re: fork/exec & close file descriptors

On Tue, Jun 2, 2015 at 10:28 AM, Marko Rauhamaa wrote: > > The only problem is that you don't know how high you need to go in > general. Sure, but I didn't know anyway, so no matter what upper bound I choose (or what function I choose/implement), it's just going to be a guess. os.closerange just

Re: fork/exec & close file descriptors

Skip Montanaro : > os.closerange(fd_low, fd_high) > Close all file descriptors from fd_low (inclusive) to fd_high > (exclusive), ignoring errors. > > Guido's time machine strikes again... The only problem is that you don't know how high you need to go in general. Marko -- https://m

Re: fork/exec & close file descriptors

Reviving (and concluding) a thread I started a couple weeks ago, I asked: > The basic fork/exec dance is not a problem, but how do I discover > all the open file descriptors in the new child process to make sure > they get closed? Do I simply start at fd 3 and call os.close() on > everything up to

Re: fork/exec & close file descriptors

On Tue, May 19, 2015 at 7:10 PM, Gregory Ewing wrote: >> On Tue, May 19, 2015 at 8:54 AM, Chris Angelico > > wrote: >> >> On Linux (and possibly some other Unixes), /proc/self/fd may be of >> use. > > > On MacOSX, /dev/fd seems to be the equivalent of this. Not a

Re: fork/exec & close file descriptors

On Tue, May 19, 2015 at 8:54 AM, Chris Angelico > wrote: On Linux (and possibly some other Unixes), /proc/self/fd may be of use. On MacOSX, /dev/fd seems to be the equivalent of this. -- Greg -- https://mail.python.org/mailman/listinfo/python-list

Re: fork/exec & close file descriptors

On 05/19/2015 05:59 AM, Skip Montanaro wrote: Due to presumed bugs in an underlying library over which I have no control, I'm considering a restart in the wee hours of the morning. The basic fork/exec dance is not a problem, but how do I discover all the open file descriptors in the new child

Re: fork/exec & close file descriptors

On Wed, May 20, 2015 at 12:31 AM, Skip Montanaro wrote: > On Tue, May 19, 2015 at 8:54 AM, Chris Angelico wrote: >> >> On Linux (and possibly some other Unixes), /proc/self/fd may be of >> use. > > > Good point. Yes, /proc/PID/fd appears to contain all the entries for open > file descriptors (I a

Re: fork/exec & close file descriptors

On 2015-05-19, Skip Montanaro wrote: > Yeah, I'm still on 2.7, and am aware of PEP 446. Note that many of the file > descriptors will not have been created by my Python code. They will have > been created by underlying C/C++ libraries, so I can't guarantee which > flags were set on file open. The

Re: fork/exec & close file descriptors

On Tue, May 19, 2015 at 8:54 AM, Chris Angelico wrote: > On Linux (and possibly some other Unixes), /proc/self/fd may be of > use. > Good point. Yes, /proc/PID/fd appears to contain all the entries for open file descriptors (I am on Linux). Skip -- https://mail.python.org/mailman/listinfo/pyth

Re: fork/exec & close file descriptors

On Tue, May 19, 2015 at 11:44 PM, Skip Montanaro wrote: > > On Tue, May 19, 2015 at 8:33 AM, Chris Angelico wrote: >> >> What Python version are you targeting? Are you aware of PEP 446? > > > Yeah, I'm still on 2.7, and am aware of PEP 446. Note that many of the file > descriptors will not have b

Re: fork/exec & close file descriptors

On Tue, May 19, 2015 at 8:33 AM, Chris Angelico wrote: > What Python version are you targeting? Are you aware of PEP 446? Yeah, I'm still on 2.7, and am aware of PEP 446. Note that many of the file descriptors will not have been created by my Python code. They will have been created by underlyi

Re: fork/exec & close file descriptors

On Tue, May 19, 2015 at 10:59 PM, Skip Montanaro wrote: > Due to presumed bugs in an underlying library over which I have no control, > I'm considering a restart in the wee hours of the morning. The basic > fork/exec dance is not a problem, but how do I discover all the open file > descriptors in

fork/exec & close file descriptors

Due to presumed bugs in an underlying library over which I have no control, I'm considering a restart in the wee hours of the morning. The basic fork/exec dance is not a problem, but how do I discover all the open file descriptors in the new child process to make sure they get closed? Do I simply s