Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
18.12.13 04:40, Benjamin Peterson написав(ла): Mostly yes, but at least you could tell people to upgrade straight to 2.7.7 and skip 2.7.6. It'll make the people to postpone the upgrade to 2.7.6 (which fixes many security bugs) until 2.7.7 release, instead of correcting their morally-broken programs. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
On Wed, Dec 18, 2013 at 8:43 PM, Serhiy Storchaka wrote: > 18.12.13 04:40, Benjamin Peterson написав(ла): > >> Mostly yes, but at least you could tell people to upgrade straight to >> 2.7.7 and skip 2.7.6. > > > It'll make the people to postpone the upgrade to 2.7.6 (which fixes many > security bugs) until 2.7.7 release, instead of correcting their > morally-broken programs. If this is considered enough breakage to be a problem, would it be possible to issue a 2.7.6.1 or 2.7.6+fixed release that's identical to 2.7.6 but with this change reverted? It'd be a minor mess, but then people would still get those security fixes, and it means not breaking stuff in a point release. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
On 18 December 2013 20:17, Chris Angelico wrote: > On Wed, Dec 18, 2013 at 8:43 PM, Serhiy Storchaka wrote: >> 18.12.13 04:40, Benjamin Peterson написав(ла): >> >>> Mostly yes, but at least you could tell people to upgrade straight to >>> 2.7.7 and skip 2.7.6. >> >> >> It'll make the people to postpone the upgrade to 2.7.6 (which fixes many >> security bugs) until 2.7.7 release, instead of correcting their >> morally-broken programs. > > If this is considered enough breakage to be a problem, would it be > possible to issue a 2.7.6.1 or 2.7.6+fixed release that's identical to > 2.7.6 but with this change reverted? It'd be a minor mess, but then > people would still get those security fixes, and it means not breaking > stuff in a point release. If we revert it, it means we do 2.7.7 ASAP so that *everyone* can just skip straight to 2.7.7. That kind of user visible change shouldn't have been made in a point release, regardless of what the docs said. It just isn't worth the risk of breaking the code of people that are relying on what's possible rather than what the docs say. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread issues when embedding Python
Another link that fills in some gaps and finally helped me make this work: http://www.codevate.com/blog/7-concurrency-with-embedded-python-in-a-multi-threaded-c-application In particular, I found that PyGILState_Ensure/PyGILState_Release as described in the Python docs is not sufficient - as described in that blog link, I had to a) obtain PyInterpreterState from the first thread where Py_Initialize() was called b) when each worker thread starts, call PyThreadState_New(mInterpreterState) and save the result in a thread local mPyThreadState c) use the mPyThreadState with PyEval_RestoreThread and PyEval_SaveThread before and after calling Python methods Is this a bug in PyGILState_Ensure or is it a deficiency in the documentation? I also found one bug in my own code, although that was not related to the problem just described with PyGILState_Ensure and I had to fix both problems to make it work. Specifically, the PyWorkerThread constructor was taking an object argument when it should have taken a reference argument and this was creating an invalid Py::Callable member in my worker. On 18/12/13 00:19, Daniel Pocock wrote: > > I've successfully embedded Python for a single thread > > I tried to extend the implementation for multiple threads (a worker > thread scenario) and I'm encountering either deadlocks or seg faults > depending upon how I got about it. > > There seems to be some inconsistency between what is covered in the docs > here: > > http://docs.python.org/2/c-api/init.html#non-python-created-threads > > and the experiences of other users trying the same thing, e.g. > > http://bugs.python.org/issue19576 > http://wiki.blender.org/index.php/Dev:2.4/Source/Python/API/Threads > > Can anybody comment on the situation, in particular, > > Is the non-python-created-threads documentation accurate for v2.7? > > If a main thread does things like importing a module and obtaining a > reference to a Python method, can those things be used by other C++ > threads or do they have to repeat those lookups? > > Is there any logic that needs to be executed once only as each thread is > started? (the doc suggests just PyGILState_Ensure/PyGILState_Release > each time a thread accesses Python methods - is there anything else?) > > Given the bug 19576, what is the most portable way to code this to work > reliably on unfixed Python versions? (e.g. should users always > explicitly call PyEval_InitThreads() in their main thread or worker > threads or both?) > > > > > Here is my actual source code: > > https://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/ > > (see example.py for a trivial example of what it does) > > The problem that I encounter: > > - the init stuff runs fine in PyRoutePlugin.cxx, > it calls Py_Initialize, PyEval_InitThreads, PyImport_ImportModule > and looks up the "provide_route" method in the module > it creates a PyRouteWorker object, > giving it a reference to "provide_route" > it creates a thread pool to run the worker > > - the PyRouteWorker::process() method is invoked in one of those threads > > - it crashes when trying to call the "provide_route" method > PyRouteWorker.cxx: > routes = mAction.apply(args); > > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x730b8700 (LWP 23965)] > 0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0 > (gdb) bt > #0 0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0 > #1 0x73d6b647 in PyEval_CallObjectWithKeywords () >from /usr/lib/libpython2.7.so.1.0 > #2 0x7414885a in apply (args=..., this=) > at /usr/include/python2.7/CXX/Python2/Objects.hxx:3215 > #3 repro::PyRouteWorker::process (this=0x6f00a0, msg=) > at PyRouteWorker.cxx:98 > #4 0x77b879e1 in repro::WorkerThread::thread (this=0x68e110) > at WorkerThread.cxx:36 > #5 0x770b7a2f in threadIfThreadWrapper (threadParm=) > at ThreadIf.cxx:51 > #6 0x765ffb50 in start_thread (arg=) > at pthread_create.c:304 > #7 0x75999a7d in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 > #8 0x in ?? () > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/daniel%40pocock.com.au ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread issues when embedding Python
On Wed, Dec 18, 2013 at 9:26 PM, Daniel Pocock wrote: > b) when each worker thread starts, call > PyThreadState_New(mInterpreterState) and save the result in a thread > local mPyThreadState > > c) use the mPyThreadState with PyEval_RestoreThread and > PyEval_SaveThread before and after calling Python methods > > Is this a bug in PyGILState_Ensure or is it a deficiency in the > documentation? It doesn't surprise me that you would need to do step b - I do seem to recall the need to call that for each new thread. Not so sure about c. Once you fixed the unrelated bug, do you still need that step? (Been a while since I last embedded Python though, and I might well be wrong.) ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread issues when embedding Python
On 18/12/13 16:02, Chris Angelico wrote:
> On Wed, Dec 18, 2013 at 9:26 PM, Daniel Pocock wrote:
>> b) when each worker thread starts, call
>> PyThreadState_New(mInterpreterState) and save the result in a thread
>> local mPyThreadState
>>
>> c) use the mPyThreadState with PyEval_RestoreThread and
>> PyEval_SaveThread before and after calling Python methods
>>
>> Is this a bug in PyGILState_Ensure or is it a deficiency in the
>> documentation?
> It doesn't surprise me that you would need to do step b - I do seem to
> recall the need to call that for each new thread. Not so sure about c.
> Once you fixed the unrelated bug, do you still need that step? (Been a
> while since I last embedded Python though, and I might well be wrong.)
Yes, I definitely needed to use this PyThreadState_New call even after
my unrelated bug fix
Should it be added to the documentation?
I created a C++ wrapper around this logic, it is here:
https://github.com/resiprocate/resiprocate/blob/master/repro/plugins/pyroute/PyThreadSupport.hxx
and the use case is something like:
// in constructor:
PyExternalUser* mPyUser = new PyExternalUser(mInterpreterState);
and each time Python calls are made, just do:
{
PyExternalUser::Use use(*mPyUser);
// now call Python code
}
When the PyExternalUser::Use instance is created it does PyEval_RestoreThread()
When the PyExternalUser::Use instance goes out of scope it is destroyed and
PyEval_SaveThread() is called
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread issues when embedding Python
On Wed, 18 Dec 2013 00:19:23 +0100 Daniel Pocock wrote: > > If a main thread does things like importing a module and obtaining a > reference to a Python method, can those things be used by other C++ > threads or do they have to repeat those lookups? The C++ threads must use the PyGILState API to initialize corresponding Python thread states, and to hold the GIL. However, you don't have to *repeat* the lookups - pointers valid in one thread are valid in other threads, as long as you own a strong reference to the PyObject (beware functions which return a borrowed reference). > Is there any logic that needs to be executed once only as each thread is > started? (the doc suggests just PyGILState_Ensure/PyGILState_Release > each time a thread accesses Python methods - is there anything else?) If you use the PyGILState API, there shouldn't be anything else. > (e.g. should users always > explicitly call PyEval_InitThreads() in their main thread or worker > threads or both?) You only need to call PyEval_InitThreads() once in the main Python thread. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread issues when embedding Python
2013/12/18 Antoine Pitrou : > You only need to call PyEval_InitThreads() once in the main Python > thread. This is not well documented. For your information, PyGILState_Ensure() now calls PyEval_InitThreads() in Python 3.4, see: http://bugs.python.org/issue19576 Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
BTW, I bet a lavish dinner at PyCon that it is *only* Zope/ZODB that does this. In the time we added this bogus dependency on undocumented parameters, the PythonLabs team was at Zope and we didn't always get our boundaries straight. On Wed, Dec 18, 2013 at 2:51 AM, Nick Coghlan wrote: > On 18 December 2013 20:17, Chris Angelico wrote: >> On Wed, Dec 18, 2013 at 8:43 PM, Serhiy Storchaka >> wrote: >>> 18.12.13 04:40, Benjamin Peterson написав(ла): >>> Mostly yes, but at least you could tell people to upgrade straight to 2.7.7 and skip 2.7.6. >>> >>> >>> It'll make the people to postpone the upgrade to 2.7.6 (which fixes many >>> security bugs) until 2.7.7 release, instead of correcting their >>> morally-broken programs. >> >> If this is considered enough breakage to be a problem, would it be >> possible to issue a 2.7.6.1 or 2.7.6+fixed release that's identical to >> 2.7.6 but with this change reverted? It'd be a minor mess, but then >> people would still get those security fixes, and it means not breaking >> stuff in a point release. > > If we revert it, it means we do 2.7.7 ASAP so that *everyone* can just > skip straight to 2.7.7. > > That kind of user visible change shouldn't have been made in a point > release, regardless of what the docs said. It just isn't worth the > risk of breaking the code of people that are relying on what's > possible rather than what the docs say. > > Cheers, > Nick. > > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
But who could forget njzrs' wasp UAV software line 107, using int=float? https://github.com/nzjrs/wasp/blob/master/sw/groundstation/wasp/__init__.py#L107 On Wed, Dec 18, 2013 at 10:49 AM, Guido van Rossum wrote: > BTW, I bet a lavish dinner at PyCon that it is *only* Zope/ZODB that > does this. In the time we added this bogus dependency on undocumented > parameters, the PythonLabs team was at Zope and we didn't always get > our boundaries straight. > > On Wed, Dec 18, 2013 at 2:51 AM, Nick Coghlan wrote: >> On 18 December 2013 20:17, Chris Angelico wrote: >>> On Wed, Dec 18, 2013 at 8:43 PM, Serhiy Storchaka >>> wrote: 18.12.13 04:40, Benjamin Peterson написав(ла): > Mostly yes, but at least you could tell people to upgrade straight to > 2.7.7 and skip 2.7.6. It'll make the people to postpone the upgrade to 2.7.6 (which fixes many security bugs) until 2.7.7 release, instead of correcting their morally-broken programs. >>> >>> If this is considered enough breakage to be a problem, would it be >>> possible to issue a 2.7.6.1 or 2.7.6+fixed release that's identical to >>> 2.7.6 but with this change reverted? It'd be a minor mess, but then >>> people would still get those security fixes, and it means not breaking >>> stuff in a point release. >> >> If we revert it, it means we do 2.7.7 ASAP so that *everyone* can just >> skip straight to 2.7.7. >> >> That kind of user visible change shouldn't have been made in a point >> release, regardless of what the docs said. It just isn't worth the >> risk of breaking the code of people that are relying on what's >> possible rather than what the docs say. >> >> Cheers, >> Nick. >> >> >> -- >> Nick Coghlan | [email protected] | Brisbane, Australia >> ___ >> Python-Dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
On Wed, 18 Dec 2013 11:31:47 -0500 Daniel Holth wrote: > But who could forget njzrs' wasp UAV software line 107, using > int=float? > https://github.com/nzjrs/wasp/blob/master/sw/groundstation/wasp/__init__.py#L107 > And the purpose is quite Pythonesque: """ Generates a noisy random walk """ Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
[Daniel Holth] > But who could forget njzrs' wasp UAV software line 107, using > int=float? > https://github.com/nzjrs/wasp/blob/master/sw/groundstation/wasp/__init__.py#L107 I could forget it ;-) The remarkable thing about the two instances of: random.randrange(0.0,1.0, int=float) in that file is that they're obscure and inefficient ways to spell: random.random() Which reminds me. I used to think there was no such thing as a stupid question. Then I discovered Stack Overflow ;-) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
On Dec 18, 2013 11:54 AM, "Tim Peters" wrote: > [Daniel Holth] > > But who could forget njzrs' wasp UAV software line 107, using > > int=float? > https://github.com/nzjrs/wasp/blob/master/sw/groundstation/wasp/__init__.py#L107 > > I could forget it ;-) The remarkable thing about the two instances of: > > random.randrange(0.0,1.0, int=float) > > in that file is that they're obscure and inefficient ways to spell: > > random.random() > You can keep your "premature optimizations" thank you :-) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
On 12/18/2013 08:54 AM, Tim Peters wrote: Which reminds me. I used to think there was no such thing as a stupid question. Then I discovered Stack Overflow ;-) +1 QOTW ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thread issues when embedding Python
On 18/12/13 16:29, Victor Stinner wrote: > 2013/12/18 Antoine Pitrou : >> You only need to call PyEval_InitThreads() once in the main Python >> thread. > > This is not well documented. For your information, PyGILState_Ensure() > now calls PyEval_InitThreads() in Python 3.4, see: > http://bugs.python.org/issue19576 I did see that - but from my own experience, I do not believe it is calling PyThreadState_New(..) and it is not even checking if PyThreadState_New(..) has ever been called for the active thread Consequently, the thread is blocked or there is a seg fault I've now written up a much more thorough overview of my experience on my blog: http://danielpocock.com/embedding-python-multi-threaded-cpp ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cffi in stdlib
On Tue, Dec 17, 2013 at 8:43 AM, Stefan Krah wrote: > Maciej Fijalkowski wrote: > > I would like to discuss on the language summit a potential inclusion > > of cffi[1] into stdlib. This is a project Armin Rigo has been working > > for a while, with some input from other developers. > > I've tried cffi (admittedly only in a toy script) and find it very nice > to use. > > Here's a comparison (pi benchmark) between wrapping libmpdec using a > C-extension (_decimal), cffi and ctypes: > > > +---+--+--+-+ > | | _decimal | ctypes | cffi | > +===+==+==+=+ > | cpython-tip (with-system-ffi) | 0.19s | 5.40s | 5.14s | > +---+--+--+-+ > | cpython-2.7 (with-system-ffi) |n/a | 4.46s | 5.18s | > +---+--+--+-+ > | Ubuntu-cpython-2.7 |n/a | 3.63s |-| > +---+--+--+-+ > | pypy-2.2.1-linux64 |n/a | 125.9s | 0.94s | > +---+--+--+-+ > | pypy3-2.1-beta1-linux64 |n/a | 264.9s | 2.93s | > +---+--+--+-+ > > > I guess the key points are that C-extensions are hard to beat and that > cffi performance on pypy-2 is outstanding. Additionally it's worth noting > that Ubuntu does something in their Python build that we should do, too. > Ubuntu compiles their Python with FDO (feedback directed optimization / profile guided optimization) enabled. All distros should do this if they don't already. It's generally 20% interpreter speedup. Our makefile already supports it but it isn't the default build as it takes a long time given that it needs to compile everything twice and do a profiled benchmark run between compilations. -gps > > +1 for cffi in the stdlib. > > > > Stefan Krah > > > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
