By embedding, I think you may be referring to embedding Python in a C program with the Python C API. That's not what I'm doing here -- I'm not using the Python C API. The C program creates two threads (using pthreads), one for itself and one for the child process. On creation, the second pthread is pointed to a C program that calls fork-execv to run the Python program. That way Python runs on a separate thread. The multiprocessing library "effectively side-step[s] the Global Interpreter Lock by using subprocesses instead of threads." https://docs.python.org/3/library/multiprocessing.html. This way I can get the Python functionality I want on call from the C program through pipes and shared memory.
I don't want to use the C API because I will be making certain library calls from the C program, and the syntax is much easier with native Python code than with C API code. I hope that clarifies what I'm doing. Jen Dec 5, 2021, 15:19 by ba...@barrys-emacs.org: > > > > >> On 5 Dec 2021, at 17:54, Jen Kris <jenk...@tutanota.com> wrote: >> >> >> Thanks for your comments. >> >> I put the Python program on its own pthread, and call a small C program to >> fork-execv to call the Python program as a child process. >> > > What do you mean by putting python in it’s own pthread? > Are you embedding python in an other program? > > Barry > > > >> I revised the Python program to be a multiprocessing loop using the Python >> multiprocessing module. That bypasses the GIL and allows Python to run >> concurrently with C. So far so good. >> >> Next I will use Linux pipes, not Python multiprocessing pipes, for IPC >> between Python and C. Multiprocessing pipes are (as far as I can tell) only >> for commo between two Python processes. I will have the parent thread send >> a signal through the pipe to the child process to exit when the parent >> thread is ready to exit, then call wait() to finalize the child process. >> >> I will reply back when it's finished and post the code so you can see what I >> have done. >> >> Thanks again. >> >> Jen >> >> >> Dec 4, 2021, 09:22 by ba...@barrys-emacs.org: >> >>> >>> >>>> On 1 Dec 2021, at 16:01, Jen Kris <>>>> jenk...@tutanota.com>>>> > wrote: >>>> >>>> Thanks for your comment re blocking. >>>> >>>> I removed pipes from the Python and C programs to see if it blocks without >>>> them, and it does. >>>> >>>> It looks now like the problem is not pipes. >>>> >>> >>> Ok. >>> >>> >>>> I use fork() and execv() in C to run Python in a child process, but the >>>> Python process blocks >>>> >>> >>> Use strace on the parent process to see what is happening. >>> You will need to use the option to follow subprocesses so that you can see >>> what goes on in the python process. >>> >>> See man strace and the --follow-forks and --output-separately options. >>> That will allow you to find the blocking system call that your code is >>> making. >>> >>> >>>> because fork() does not create a new thread, so the Python global >>>> interpreter lock (GIL) prevents the C program from running once Python >>>> starts. >>>> >>> >>> Not sure why you think this. >>> >>> >>>> So the solution appears to be run Python in a separate thread, which I >>>> can do with pthread create. >>>> >>>> See "Thread State and the Global Interpreter Lock" >>>> >>>> https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock>>>> >>>> and the sections below that "Non-Python created threads" and "Cautions >>>> about fork()." >>>> >>> >>> I take it you mean that in the parent you think that using pthreads will >>> affect python after the exec() call? >>> I does not. After exec() the process has one main thread create by the >>> kernel and a new address space as defined by the /usr/bin/python. >>> The only state that in inherited from the parent are open file descriptors, >>> the current working directory and security state like UID, GID. >>> >>> >>>> I'm working on that today and I hope all goes well :) >>>> >>> >>> You seem to be missing background information on how processes work. >>> Maybe "Advanced Programming in the UNIX Environment" >>> would be helpful? >>> >>> https://www.amazon.co.uk/Programming-Environment-Addison-Wesley-Professional-Computing-dp-0321637739/dp/0321637739/ref=dp_ob_image_bk>>> >>> >>> >>> It's a great book and covers a wide range of Unix systems programming >>> topics. >>> >>> Have you created a small C program that just does the fork and exec of a >>> python program to test out your assumptions? >>> If not I recommend that you do. >>> >>> Barry >>> >>> >>> >>>> >>>> >>>> >>>> Nov 30, 2021, 11:42 by >>>> ba...@barrys-emacs.org>>>> : >>>> >>>>> >>>>> >>>>> >>>>>> On 29 Nov 2021, at 22:31, Jen Kris <>>>>>> jenk...@tutanota.com>>>>>> > >>>>>> wrote: >>>>>> >>>>>> Thanks to you and Cameron for your replies. The C side has an epoll_ctl >>>>>> set, but no event loop to handle it yet. I'm putting that in now with a >>>>>> pipe write in Python-- as Cameron pointed out that is the likely source >>>>>> of blocking on C. The pipes are opened as rdwr in Python because that's >>>>>> nonblocking by default. The child will become more complex, but not in >>>>>> a way that affects polling. And thanks for the tip about the c-string >>>>>> termination. >>>>>> >>>>>> >>>>> >>>>> flags is a bit mask. You say its BLOCKing by not setting os.O_NONBLOCK. >>>>> You should not use O_RDWR when you only need O_RDONLY access or only >>>>> O_WRONLY access. >>>>> >>>>> You may find >>>>> >>>>> man 2 open >>>>> >>>>> useful to understand in detail what is behind os.open(). >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> Nov 29, 2021, 14:12 by >>>>>> ba...@barrys-emacs.org>>>>>> : >>>>>> >>>>>>> >>>>>>> >>>>>>>> On 29 Nov 2021, at 20:36, Jen Kris via Python-list <>>>>>>>> >>>>>>>> python-list@python.org>>>>>>>> > wrote: >>>>>>>> >>>>>>>> I have a C program that forks to create a child process and uses >>>>>>>> execv to call a Python program. The Python program communicates with >>>>>>>> the parent process (in C) through a FIFO pipe monitored with epoll(). >>>>>>>> >>>>>>>> The Python child process is in a while True loop, which is intended to >>>>>>>> keep it running while the parent process proceeds, and perform >>>>>>>> functions for the C program only at intervals when the parent sends >>>>>>>> data to the child -- similar to a daemon process. >>>>>>>> >>>>>>>> The C process writes to its end of the pipe and the child process >>>>>>>> reads it, but then the child process continues to loop, thereby >>>>>>>> blocking the parent. >>>>>>>> >>>>>>>> This is the Python code: >>>>>>>> >>>>>>>> #!/usr/bin/python3 >>>>>>>> import os >>>>>>>> import select >>>>>>>> >>>>>>>> #Open the named pipes >>>>>>>> pr = os.open('/tmp/Pipe_01', os.O_RDWR) >>>>>>>> >>>>>>> Why open rdwr if you are only going to read the pipe? >>>>>>> >>>>>>>> pw = os.open('/tmp/Pipe_02', os.O_RDWR) >>>>>>>> >>>>>>> Only need to open for write. >>>>>>> >>>>>>>> >>>>>>>> ep = select.epoll(-1) >>>>>>>> ep.register(pr, select.EPOLLIN) >>>>>>>> >>>>>>> >>>>>>> Is the only thing that the child does this: >>>>>>> 1. Read message from pr >>>>>>> 2. Process message >>>>>>> 3. Write result to pw. >>>>>>> 4. Loop from 1 >>>>>>> >>>>>>> If so as Cameron said you do not need to worry about the poll. >>>>>>> Do you plan for the child to become more complex? >>>>>>> >>>>>>>> >>>>>>>> while True: >>>>>>>> >>>>>>>> events = ep.poll(timeout=2.5, maxevents=-1) >>>>>>>> #events = ep.poll(timeout=None, maxevents=-1) >>>>>>>> >>>>>>>> print("child is looping") >>>>>>>> >>>>>>>> for fileno, event in events: >>>>>>>> print("Python fileno") >>>>>>>> print(fileno) >>>>>>>> print("Python event") >>>>>>>> print(event) >>>>>>>> v = os.read(pr,64) >>>>>>>> print("Pipe value") >>>>>>>> print(v) >>>>>>>> >>>>>>>> The child process correctly receives the signal from ep.poll and >>>>>>>> correctly reads the data in the pipe, but then it continues looping. >>>>>>>> For example, when I put in a timeout: >>>>>>>> >>>>>>>> child is looping >>>>>>>> Python fileno >>>>>>>> 4 >>>>>>>> Python event >>>>>>>> 1 >>>>>>>> Pipe value >>>>>>>> b'10\x00' >>>>>>>> >>>>>>> The C code does not need to write a 0 bytes at the end. >>>>>>> I assume the 0 is from the end of a C string. >>>>>>> UDS messages have a length. >>>>>>> In the C just write 2 byes in the case. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>>> child is looping >>>>>>>> child is looping >>>>>>>> >>>>>>>> That suggests that a while True loop is not the right thing to do in >>>>>>>> this case. My question is, what type of process loop is best for this >>>>>>>> situation? The multiprocessing, asyncio and subprocess libraries are >>>>>>>> very extensive, and it would help if someone could suggest the best >>>>>>>> alternative for what I am doing here. >>>>>>>> >>>>>>>> Thanks very much for any ideas. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> https://mail.python.org/mailman/listinfo/python-list >>>>>>>> >>>>>> >>>>>> >>>> >>>> >> >> -- https://mail.python.org/mailman/listinfo/python-list
