Feature Requests item #1432694, was opened at 2006-02-16 18:11 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1432694&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Threads Group: None Status: Open Resolution: Invalid Priority: 5 Submitted By: Andrey Petrov (darkprokoba) Assigned to: Michael Hudson (mwh) Summary: Implement preemptive threads in Python Initial Comment: Cooperative threads have their uses but I find the lack of real native pre-emptive threads a major issue in a 21-st century language. We should consider the inertia of programmers from C++/Java/.NET background. They are used to and experienced in solving certain kind of problems with pre-emptive threads, and it's just too plain inconvenient to step back and try to cope with cooperative threads. The idea behind Python is that programming should not be a pain, right? This applies especially to GUI programs, where threads are used to perform time-consuming work in order to avoid the GUI from freezing (was I surprised the first time I tried this in Python!) Just look at some of the modern Python GUI applications, found in some recent Linux distributions. 99% of Fedora's configuration tools suffer extreme usability issues caused by their single-threadedness. I hate using yumex because the GUI is basically frozen for the half minute it takes to reload all repos I have configured on my machine!!! I know there might be a way to alleviate this particual class of problems by more careful cooperative thread programing... but I believe we need a more straightforward solution (i.e. preemptive threads) and not workarounds. I apologize if this issue has already been raised before. Cheers, darkprokoba ---------------------------------------------------------------------- >Comment By: Nick Coghlan (ncoghlan) Date: 2006-04-07 21:28 Message: Logged In: YES user_id=1038590 One addition to Josiah's comments is that the previous attempts at free-threading failed to gain an appreciable performance benefit, as the gains from removing the global lock were lost in the subsequent need to add explicit synchronisation everywhere. Free threading is not a panacea. If Queue.Queue is used to pass data to worker threads, idle threads will not be consuming any GIL time. If there is OS level context-switch thrashing going on, then its time to look at using thread pools instead of a thread per task. Now, if you were to rephrase this request as "I want to be able to make the GUI thread higher priority than the worker threads so I can have a gazillion threads without hurting responsiveness of the main event loop so badly", that would be an entirely different story, and not a question I have seen asked recently (or at all, in fact). (Josiah's comments about any such thing being a couple of years away still hold, however - the multi-process support libraries are here and now, and scale a lot better than threads ever will, no matter what language you use) ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2006-02-22 18:35 Message: Logged In: YES user_id=341410 How many worker threads do you expect? If you work the sys.setcheckinterval() just right, you may be able to prioritize your threads... Any C library which does not acquire and release the GIL when it is supposed to is broken. Claiming that Python is broken in relation to C libraries being broken is misplaced blame. Python uses a GIL because it makes interaction with Python objects easier across threads. There have been previous failed efforts to make Python scalable across multiple processors; search for "python free threading" on google to see all of the relevant information about rewriting the Python interpreter with multiple processors in mind. Alternatively, you can always use one of the half-dozen parallel processing libraries for Python to distribute your work onto different processors, who all synchronize up with the main GUI process. Some are quite intuitive (check out Kamaelia), and will work today (any effort to get a 'free threading' idea into Python core will have to wait until at least Python 2.6, which is at least 2 years away). ---------------------------------------------------------------------- Comment By: Andrey Petrov (darkprokoba) Date: 2006-02-22 17:55 Message: Logged In: YES user_id=950037 > thread context switches happen at a regular > rate thanks for your response, but with enough worker threads it's still easy to starve the GUI thread. and the other problems of this model remain: 1) a stupid native module could forget to release the global interpreter lock and block on I/O, freezing the entire interpreter (i.e. GIL is a pain for native module writers) 2) poor CPU utilization on SMP machines (which are quite mainstream now) Python's model just doesn't cut it for me (I'm well aware now of how it works) so my question boils down to: Do the python authors plan to make the interpreter reentrant and if so when can we expect to see this in a production release? ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2006-02-22 15:44 Message: Logged In: YES user_id=341410 If GUI applications are written properly, they don't need to hang when they are doing "real work". The applications you are using which seem to hang are doing so not because Python "doesn't have preemptive threads", they are doing so because the writers of those GUI applications have no idea how to either 1. write their programs asynchronously via generators or callbacks or 2. use threads to do the actual work which post GUI update events. Talk to the writers of those applications to get them fixed. Note that regardless of Python's lack of preemptive multithreading, thread context switches happen at a regular rate, discoverable via sys.getcheckinterval(), and setable via sys.setcheckinterval(). ---------------------------------------------------------------------- Comment By: Andrey Petrov (darkprokoba) Date: 2006-02-17 19:52 Message: Logged In: YES user_id=950037 OK, let me try again :-) The problem is in the global interpreter lock: http://docs.python.org/api/threads.html this basically says that you can have as many native threads as you like but only one of them could have the global interpreter lock and could therefore be executing python code at a time. The only use of python's multiple threads then is so they could release the global lock and block on i/o operations while one lucky thread has the lock and executes python code and accesses python objects happily. So we have multiple native threads (e.g. pthreads) but they pass arround the global lock in a cooperative manner. This is not preemptive threading imho. It's a severely limited model and has the following potential problem: a thread may enter a time-consuming i/o operation but forget to release the global interpreter lock - leading to a freeze in the entire interpreter (all threads are waiting for the global lock, while the thread that has it waits on the i/o operation) Are there any plans for alleviating this problem? A thread should not voluntarily release the lock so that the rest of the threads get a chance to execute python code. It should be possible to preempt any thread at any time without its consent in order to have a true preemptive threading model. Or please correct me If I'm wrong. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2006-02-17 00:14 Message: Logged In: YES user_id=6656 We _have_ preemptive threads in Python ("import threading"). Suggest you report a real problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1432694&group_id=5470 _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
