> On 5 Dec 2021, at 17:54, Jen Kris <jenk...@tutanota.com> wrote: > > > Thanks for your comments. > > I put the Python program on its own pthread, and call a small C program to > fork-execv to call the Python program as a child process.
What do you mean by putting python in it’s own pthread? Are you embedding python in an other program? Barry > I revised the Python program to be a multiprocessing loop using the Python > multiprocessing module. That bypasses the GIL and allows Python to run > concurrently with C. So far so good. > > Next I will use Linux pipes, not Python multiprocessing pipes, for IPC > between Python and C. Multiprocessing pipes are (as far as I can tell) only > for commo between two Python processes. I will have the parent thread send a > signal through the pipe to the child process to exit when the parent thread > is ready to exit, then call wait() to finalize the child process. > > I will reply back when it's finished and post the code so you can see what I > have done. > > Thanks again. > > Jen > > > Dec 4, 2021, 09:22 by ba...@barrys-emacs.org: > >>> On 1 Dec 2021, at 16:01, Jen Kris <jenk...@tutanota.com> wrote: >>> >>> Thanks for your comment re blocking. >>> >>> I removed pipes from the Python and C programs to see if it blocks without >>> them, and it does. >>> It looks now like the problem is not pipes. >> >> Ok. >> >> I use fork() and execv() in C to run Python in a child process, but the >> Python process blocks > > Use strace on the parent process to see what is happening. > You will need to use the option to follow subprocesses so that you can see > what goes on in the python process. > > See man strace and the --follow-forks and --output-separately options. > That will allow you to find the blocking system call that your code is making. > >> because fork() does not create a new thread, so the Python global >> interpreter lock (GIL) prevents the C program from running once Python >> starts. > > Not sure why you think this. > >> So the solution appears to be run Python in a separate thread, which I can >> do with pthread create. >> See "Thread State and the Global Interpreter Lock" >> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock >> and the sections below that "Non-Python created threads" and "Cautions >> about fork()." > > I take it you mean that in the parent you think that using pthreads will > affect python after the exec() call? > I does not. After exec() the process has one main thread create by the kernel > and a new address space as defined by the /usr/bin/python. > The only state that in inherited from the parent are open file descriptors, > the current working directory and security state like UID, GID. > >> I'm working on that today and I hope all goes well :) > > You seem to be missing background information on how processes work. > Maybe "Advanced Programming in the UNIX Environment" would be helpful? > > https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk > > > It's a great book and covers a wide range of Unix systems programming topics. > > Have you created a small C program that just does the fork and exec of a > python program to test out your assumptions? > If not I recommend that you do. > > Barry > > >> >> >> >> Nov 30, 2021, 11:42 by ba...@barrys-emacs.org: >> >> >>> On 29 Nov 2021, at 22:31, Jen Kris <jenk...@tutanota.com> wrote: >>> >>> Thanks to you and Cameron for your replies. The C side has an epoll_ctl >>> set, but no event loop to handle it yet. I'm putting that in now with a >>> pipe write in Python-- as Cameron pointed out that is the likely source of >>> blocking on C. The pipes are opened as rdwr in Python because that's >>> nonblocking by default. The child will become more complex, but not in a >>> way that affects polling. And thanks for the tip about the c-string >>> termination. >>> >> >> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK. >> You should not use O_RDWR when you only need O_RDONLY access or only >> O_WRONLY access. >> >> You may find >> >> man 2 open >> >> useful to understand in detail what is behind os.open(). >> >> Barry >> >> >> >>> >>> >>> Nov 29, 2021, 14:12 by ba...@barrys-emacs.org: >>> >>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <python-list@python.org> >>> wrote: >>> >>> I have a C program that forks to create a child process and uses execv to >>> call a Python program. The Python program communicates with the parent >>> process (in C) through a FIFO pipe monitored with epoll(). >>> >>> The Python child process is in a while True loop, which is intended to keep >>> it running while the parent process proceeds, and perform functions for the >>> C program only at intervals when the parent sends data to the child -- >>> similar to a daemon process. >>> >>> The C process writes to its end of the pipe and the child process reads it, >>> but then the child process continues to loop, thereby blocking the parent. >>> >>> This is the Python code: >>> >>> #!/usr/bin/python3 >>> import os >>> import select >>> >>> #Open the named pipes >>> pr = os.open('/tmp/Pipe_01', os.O_RDWR) >>> Why open rdwr if you are only going to read the pipe? >>> pw = os.open('/tmp/Pipe_02', os.O_RDWR) >>> Only need to open for write. >>> >>> ep = select.epoll(-1) >>> ep.register(pr, select.EPOLLIN) >>> >>> Is the only thing that the child does this: >>> 1. Read message from pr >>> 2. Process message >>> 3. Write result to pw. >>> 4. Loop from 1 >>> >>> If so as Cameron said you do not need to worry about the poll. >>> Do you plan for the child to become more complex? >>> >>> while True: >>> >>> events = ep.poll(timeout=2.5, maxevents=-1) >>> #events = ep.poll(timeout=None, maxevents=-1) >>> >>> print("child is looping") >>> >>> for fileno, event in events: >>> print("Python fileno") >>> print(fileno) >>> print("Python event") >>> print(event) >>> v = os.read(pr,64) >>> print("Pipe value") >>> print(v) >>> >>> The child process correctly receives the signal from ep.poll and correctly >>> reads the data in the pipe, but then it continues looping. For example, >>> when I put in a timeout: >>> >>> child is looping >>> Python fileno >>> 4 >>> Python event >>> 1 >>> Pipe value >>> b'10\x00' >>> The C code does not need to write a 0 bytes at the end. >>> I assume the 0 is from the end of a C string. >>> UDS messages have a length. >>> In the C just write 2 byes in the case. >>> >>> Barry >>> child is looping >>> child is looping >>> >>> That suggests that a while True loop is not the right thing to do in this >>> case. My question is, what type of process loop is best for this situation? >>> The multiprocessing, asyncio and subprocess libraries are very extensive, >>> and it would help if someone could suggest the best alternative for what I >>> am doing here. >>> >>> Thanks very much for any ideas. >>> >>> >>> -- >>> https://mail.python.org/mailman/listinfo/python-list >>> >> >> > > -- https://mail.python.org/mailman/listinfo/python-list
