Thanks for your comments. I put the Python program on its own pthread, and call a small C program to fork-execv to call the Python program as a child process. I revised the Python program to be a multiprocessing loop using the Python multiprocessing module. That bypasses the GIL and allows Python to run concurrently with C. So far so good.
Next I will use Linux pipes, not Python multiprocessing pipes, for IPC between Python and C. Multiprocessing pipes are (as far as I can tell) only for commo between two Python processes. I will have the parent thread send a signal through the pipe to the child process to exit when the parent thread is ready to exit, then call wait() to finalize the child process. I will reply back when it's finished and post the code so you can see what I have done. Thanks again. Jen Dec 4, 2021, 09:22 by ba...@barrys-emacs.org: > > >> On 1 Dec 2021, at 16:01, Jen Kris <>> jenk...@tutanota.com>> > wrote: >> >> Thanks for your comment re blocking. >> >> I removed pipes from the Python and C programs to see if it blocks without >> them, and it does. >> >> It looks now like the problem is not pipes. >> > > Ok. > > >> I use fork() and execv() in C to run Python in a child process, but the >> Python process blocks >> > > Use strace on the parent process to see what is happening. > You will need to use the option to follow subprocesses so that you can see > what goes on in the python process. > > See man strace and the --follow-forks and --output-separately options. > That will allow you to find the blocking system call that your code is making. > > >> because fork() does not create a new thread, so the Python global >> interpreter lock (GIL) prevents the C program from running once Python >> starts. >> > > Not sure why you think this. > > >> So the solution appears to be run Python in a separate thread, which I can >> do with pthread create. >> >> See "Thread State and the Global Interpreter Lock" >> >> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock>> >> and the sections below that "Non-Python created threads" and "Cautions >> about fork()." >> > > I take it you mean that in the parent you think that using pthreads will > affect python after the exec() call? > I does not. After exec() the process has one main thread create by the kernel > and a new address space as defined by the /usr/bin/python. > The only state that in inherited from the parent are open file descriptors, > the current working directory and security state like UID, GID. > > >> I'm working on that today and I hope all goes well :) >> > > You seem to be missing background information on how processes work. > Maybe "Advanced Programming in the UNIX Environment" > would be helpful? > > https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk> > > > It's a great book and covers a wide range of Unix systems programming topics. > > Have you created a small C program that just does the fork and exec of a > python program to test out your assumptions? > If not I recommend that you do. > > Barry > > > >> >> >> >> Nov 30, 2021, 11:42 by >> ba...@barrys-emacs.org>> : >> >>> >>> >>> >>>> On 29 Nov 2021, at 22:31, Jen Kris <>>>> jenk...@tutanota.com>>>> > wrote: >>>> >>>> Thanks to you and Cameron for your replies. The C side has an epoll_ctl >>>> set, but no event loop to handle it yet. I'm putting that in now with a >>>> pipe write in Python-- as Cameron pointed out that is the likely source of >>>> blocking on C. The pipes are opened as rdwr in Python because that's >>>> nonblocking by default. The child will become more complex, but not in a >>>> way that affects polling. And thanks for the tip about the c-string >>>> termination. >>>> >>>> >>> >>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK. >>> You should not use O_RDWR when you only need O_RDONLY access or only >>> O_WRONLY access. >>> >>> You may find >>> >>> man 2 open >>> >>> useful to understand in detail what is behind os.open(). >>> >>> Barry >>> >>> >>> >>> >>>> >>>> >>>> Nov 29, 2021, 14:12 by >>>> ba...@barrys-emacs.org>>>> : >>>> >>>>> >>>>> >>>>>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <>>>>>> >>>>>> python-list@python.org>>>>>> > wrote: >>>>>> >>>>>> I have a C program that forks to create a child process and uses execv >>>>>> to call a Python program. The Python program communicates with the >>>>>> parent process (in C) through a FIFO pipe monitored with epoll(). >>>>>> >>>>>> The Python child process is in a while True loop, which is intended to >>>>>> keep it running while the parent process proceeds, and perform functions >>>>>> for the C program only at intervals when the parent sends data to the >>>>>> child -- similar to a daemon process. >>>>>> >>>>>> The C process writes to its end of the pipe and the child process reads >>>>>> it, but then the child process continues to loop, thereby blocking the >>>>>> parent. >>>>>> >>>>>> This is the Python code: >>>>>> >>>>>> #!/usr/bin/python3 >>>>>> import os >>>>>> import select >>>>>> >>>>>> #Open the named pipes >>>>>> pr = os.open('/tmp/Pipe_01', os.O_RDWR) >>>>>> >>>>> Why open rdwr if you are only going to read the pipe? >>>>> >>>>>> pw = os.open('/tmp/Pipe_02', os.O_RDWR) >>>>>> >>>>> Only need to open for write. >>>>> >>>>>> >>>>>> ep = select.epoll(-1) >>>>>> ep.register(pr, select.EPOLLIN) >>>>>> >>>>> >>>>> Is the only thing that the child does this: >>>>> 1. Read message from pr >>>>> 2. Process message >>>>> 3. Write result to pw. >>>>> 4. Loop from 1 >>>>> >>>>> If so as Cameron said you do not need to worry about the poll. >>>>> Do you plan for the child to become more complex? >>>>> >>>>>> >>>>>> while True: >>>>>> >>>>>> events = ep.poll(timeout=2.5, maxevents=-1) >>>>>> #events = ep.poll(timeout=None, maxevents=-1) >>>>>> >>>>>> print("child is looping") >>>>>> >>>>>> for fileno, event in events: >>>>>> print("Python fileno") >>>>>> print(fileno) >>>>>> print("Python event") >>>>>> print(event) >>>>>> v = os.read(pr,64) >>>>>> print("Pipe value") >>>>>> print(v) >>>>>> >>>>>> The child process correctly receives the signal from ep.poll and >>>>>> correctly reads the data in the pipe, but then it continues looping. >>>>>> For example, when I put in a timeout: >>>>>> >>>>>> child is looping >>>>>> Python fileno >>>>>> 4 >>>>>> Python event >>>>>> 1 >>>>>> Pipe value >>>>>> b'10\x00' >>>>>> >>>>> The C code does not need to write a 0 bytes at the end. >>>>> I assume the 0 is from the end of a C string. >>>>> UDS messages have a length. >>>>> In the C just write 2 byes in the case. >>>>> >>>>> Barry >>>>> >>>>>> child is looping >>>>>> child is looping >>>>>> >>>>>> That suggests that a while True loop is not the right thing to do in >>>>>> this case. My question is, what type of process loop is best for this >>>>>> situation? The multiprocessing, asyncio and subprocess libraries are >>>>>> very extensive, and it would help if someone could suggest the best >>>>>> alternative for what I am doing here. >>>>>> >>>>>> Thanks very much for any ideas. >>>>>> >>>>>> >>>>>> -- >>>>>> https://mail.python.org/mailman/listinfo/python-list >>>>>> >>>> >>>> >> >> -- https://mail.python.org/mailman/listinfo/python-list
