Re: [Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC...
I ran into this the other day. I had put in hooks in the PyMem_MALLOC to track memory per tasklet, and it crashed in those cases because it was being called without the GIL. My local patch was simply to _not_ release the GIL. Clearly, calling PyMem_MALLOC without the GIL is an API violation. K > -Original Message- > From: Python-Dev [mailto:python-dev- > [email protected]] On Behalf Of Trent Nelson > Sent: 21. desember 2012 03:13 > To: Gregory P. Smith > Cc: Python-Dev > Subject: Re: [Python-Dev] Possible GIL/threading issue involving subprocess > and PyMem_MALLOC... > > On Thu, Dec 20, 2012 at 05:47:40PM -0800, Gregory P. Smith wrote: > >On Thu, Dec 20, 2012 at 10:43 AM, Trent Nelson > >wrote: > > > > This seems odd to me so I wanted to see what others think. The > > unit > > test Lib/unittest/test/test_runner.py:Test_TextRunner.test_warnings > > will eventually hit subprocess.Popen._communicate. > > > > The `mswindows` implementation of this method relies on threads to > > buffer stdin/stdout. That'll eventually result in > > PyOs_StdioReadline > > being called without the GIL being held. PyOs_StdioReadline calls > > PyMem_MALLOC, PyMem_FREE and possibly PyMem_REALLOC. > > > >Those threads are implemented in Python so how would the GIL ever not > be > >held? > >-gps > > PyOS_Readline drops the GIL prior to calling PyOS_StdioReadline: > > Py_BEGIN_ALLOW_THREADS > ^^ > #ifdef WITH_THREAD > PyThread_acquire_lock(_PyOS_ReadlineLock, 1); > #endif > > /* This is needed to handle the unlikely case that the > * interpreter is in interactive mode *and* stdin/out are not > * a tty. This can happen, for example if python is run like > * this: python -i < test1.py > */ > if (!isatty (fileno (sys_stdin)) || !isatty (fileno (sys_stdout))) > rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt); > > -^^^ > else > rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout, > prompt); > Py_END_ALLOW_THREADS > > > Trent. > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python- > dev/kristjan%40ccpgames.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC...
Le Fri, 21 Dec 2012 09:31:44 +, Kristján Valur Jónsson a écrit : > I ran into this the other day. I had put in hooks in the > PyMem_MALLOC to track memory per tasklet, and it crashed in those > cases because it was being called without the GIL. My local patch > was simply to _not_ release the GIL. Clearly, calling PyMem_MALLOC > without the GIL is an API violation. Indeed, this deserves fixing. (it would be better to still release the GIL around the low-level I/O call, of course) Thanks Trent for finding this! Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC...
On Fri, Dec 21, 2012 at 01:43:11AM -0800, Antoine Pitrou wrote: > Le Fri, 21 Dec 2012 09:31:44 +, > Kristján Valur Jónsson a écrit : > > I ran into this the other day. I had put in hooks in the > > PyMem_MALLOC to track memory per tasklet, and it crashed in those > > cases because it was being called without the GIL. My local patch > > was simply to _not_ release the GIL. Clearly, calling PyMem_MALLOC > > without the GIL is an API violation. > > Indeed, this deserves fixing. > (it would be better to still release the GIL around the low-level I/O > call, of course) Created http://bugs.python.org/issue16742 to capture the issue for now. I want to make some more progress on the parallel stuff first so if somebody wants to tackle it in the meantime, be my guest. > Thanks Trent for finding this! Unexpected (but handy) side-effect of the parallel context work :-) (I wonder if that's the only thread-safe issue in our code base...) > Antoine. Trent. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3145 (With Contents)
On Thu, Dec 20, 2012 at 7:35 PM, Chris Jerdonek wrote: > On Thu, Dec 20, 2012 at 1:12 PM, Brett Cannon wrote: > > > > On Thu, Dec 20, 2012 at 3:55 PM, Chris Jerdonek < > [email protected]> > > wrote: > >> > >> On Thu, Dec 20, 2012 at 12:18 PM, Brett Cannon > wrote: > >> > > >> > And please do not CC the peps mailing list on discussions. It should > >> > only be > >> > used to mail in new PEPs or acceptable patches to PEPs. > >> > >> PEP 1 should perhaps be clarified if the above is the case. > >> Currently, PEP 1 says all PEP-related e-mail should be sent there: > >> > >> "The PEP editors assign PEP numbers and change their status. Please > >> send all PEP-related email to (no cross-posting > >> please). Also see PEP Editor Responsibilities & Workflow below." > >> > >> as well as: > >> > >> "A PEP editor must subscribe to the list. All > >> PEP-related correspondence should be sent (or CC'd) to > >> (but please do not cross-post!)." > >> > >> (Incidentally, the statement not to cross-post seems contradictory if > >> a PEP-related e-mail is also sent to python-dev, for example.) > > > > > > But it very clearly states to NOT cross-post which is exactly what > Anatoly > > did and that is what I take issue with the most. I personally don't see > any > > confusion with the wording. It clearly states that if you are a PEP > author > > you should mail the peps editors and NOT cross-post. If you are an > editor, > > make sure any emailing you do with an individual CCs the list but do NOT > > cross-post. > > I don't disagree that he shouldn't have cross-posted. I was just > pointing out that the language should be clarified. What's confusing > is that the current language implies that one shouldn't send any > PEP-related e-mails to any mailing list other than peps@. In > particular, how can one discuss PEPs on python-dev or python-ideas > without violating that language (e.g. this e-mail which is related to > PEP 1)? It is probably just a matter of clarifying what "PEP-related" > means. > I'm just not seeing the confusion, sorry. And we have never really had any confusion over this wording before. If you want to send a patch to tweak the wording to me more clear then please go ahead and I will consider it, but I'm not worried enough about it to try to come up with some rewording myself. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2012-12-14 - 2012-12-21) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open3844 (+18) closed 24677 (+46) total 28521 (+64) Open issues with patches: 1686 Issues opened (38) == #16682: Document that audioop works with bytes, not strings http://bugs.python.org/issue16682 opened by serhiy.storchaka #16684: Unicode property value abbreviated names and long names http://bugs.python.org/issue16684 opened by PanderMusubi #16685: Deprecate accepting strings as arguments in audioop functions http://bugs.python.org/issue16685 opened by serhiy.storchaka #16686: audioop overflow issues http://bugs.python.org/issue16686 opened by serhiy.storchaka #16688: Backreferences make case-insensitive regex fail on non-ASCII s http://bugs.python.org/issue16688 opened by pyos #16689: stdout stderr redirection mess http://bugs.python.org/issue16689 opened by techtonik #16690: Reference leak with custom tp_dealloc in PyType_FromSpec http://bugs.python.org/issue16690 opened by bfroehle #16692: Support TLS 1.1 and TLS 1.2 http://bugs.python.org/issue16692 opened by pitrou #16694: Add pure Python operator module http://bugs.python.org/issue16694 opened by zach.ware #16695: Clarify fnmatch & glob docs about the handling of leading "."s http://bugs.python.org/issue16695 opened by hynek #16698: test_posix.test_getgroups fails on some systems http://bugs.python.org/issue16698 opened by rosslagerwall #16699: Mountain Lion buildbot lacks disk space http://bugs.python.org/issue16699 opened by pitrou #16700: Document that bytes OS API can returns unusable results on Win http://bugs.python.org/issue16700 opened by serhiy.storchaka #16701: Docs missing the behavior of += (in-place add) for lists. http://bugs.python.org/issue16701 opened by montysinngh #16702: Force urllib2_localnet test not to use http proxies http://bugs.python.org/issue16702 opened by jeffknupp #16705: Use concrete classes inherited from OSError instead of errno c http://bugs.python.org/issue16705 opened by asvetlov #16709: unittest discover order is filesystem specific - hard to repro http://bugs.python.org/issue16709 opened by rbcollins #16712: collections.abc.Sequence should not provide __reversed__ http://bugs.python.org/issue16712 opened by naoki #16713: "tel" URIs should support params http://bugs.python.org/issue16713 opened by pitrou #16715: Get rid of IOError. Use OSError instead http://bugs.python.org/issue16715 opened by asvetlov #16716: Deprecate OSError aliases in the doc http://bugs.python.org/issue16716 opened by asvetlov #16718: Mysterious atexit fail http://bugs.python.org/issue16718 opened by techtonik #16720: Get rid of os.error. Use OSError instead http://bugs.python.org/issue16720 opened by serhiy.storchaka #16721: configure incorrectly adds -OPT:Olimit=0 for clang http://bugs.python.org/issue16721 opened by Vladimir.Timofeev #16723: io.TextIOWrapper on urllib.request.urlopen terminates prematur http://bugs.python.org/issue16723 opened by mdehoon #16726: expat ParseFile expects bytes, not string http://bugs.python.org/issue16726 opened by mdehoon #16728: Missing cross-reference in sequence glossary entry http://bugs.python.org/issue16728 opened by naoki #16729: Document how to provide defaults for setup.py commands options http://bugs.python.org/issue16729 opened by techtonik #16730: _fill_cache in _bootstrap.py crashes without directory execute http://bugs.python.org/issue16730 opened by David.Pritchard #16731: xxlimited/xxmodule docstrings ambiguous http://bugs.python.org/issue16731 opened by danielsh #16732: setup.py support for xxmodule without tkinker http://bugs.python.org/issue16732 opened by danielsh #16733: Solaris ctypes_test failures http://bugs.python.org/issue16733 opened by yippi #16737: Different behaviours in script run directly and via runpy.run_ http://bugs.python.org/issue16737 opened by vinay.sajip #16739: texttestresult should decorate the stream with _WritelnDecorat http://bugs.python.org/issue16739 opened by elopio #16741: `int()`, `float()`, etc think python strings are null-terminat http://bugs.python.org/issue16741 opened by gangesmaster #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which is http://bugs.python.org/issue16742 opened by trent #16743: mmap accepts files > 1 GB, but processes only 1 GB http://bugs.python.org/issue16743 opened by schlamar #16744: sys.path.append causes wrong behaviour http://bugs.python.org/issue16744 opened by rappy Most recent 15 issues with no replies (15) == #16744: sys.path.append causes wrong behaviour http://bugs.python.org/issue16744 #16743: mmap accepts files > 1 GB, but processes only 1 GB http://bugs.python.org/issue16743 #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadli
Re: [Python-Dev] compile python 3.3 with bz2 support
Hello. We are sorry but we cannot help you. This mailing list is to work on developing Python (adding new features to Python itself and fixing bugs); if you're having problems learning, understanding or using Python, please find another forum. Probably python-list/comp.lang.python mailing list/news group is the best place; there are Python developers who participate in it; you may get a faster, and probably more complete, answer there. See http://www.python.org/community/ for other lists/news groups/fora. Thank you for understanding. On Fri, Dec 21, 2012 at 02:03:40PM +0800, Isml <[email protected]> wrote: > hi, everyone: > I want to compile python 3.3 with bz2 support on RedHat 5.5 but fail to > do that. Here is how I do it: > 1??download bzip2 and compile it(make??make -f Makefile_libbz2_so??make > install) > 2??chang to python 3.3 source directory : ./configure > --with-bz2=/usr/local/include > 3??make > 4??make install > > after installation complete, I test it?? > [root@localhost Python-3.3.0]# python3 -c "import bz2" > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python3.3/bz2.py", line 21, in > from _bz2 import BZ2Compressor, BZ2Decompressor > ImportError: No module named '_bz2' You have to install bz2 development files (headers and libraries) before recompiling python. > By the way, RedHat 5.5 has a built-in python 2.4.3. Would it be a problem? Depends on what are you going to do. Oleg. -- Oleg Broytmanhttp://phdru.name/[email protected] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testing the tests by modifying the ordering of dict items.
Hello Mark, Did you raise bug for this? -- View this message in context: http://python.6.n6.nabble.com/Testing-the-tests-by-modifying-the-ordering-of-dict-items-tp3221386p5000138.html Sent from the Python - python-dev mailing list archive at Nabble.com. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
On Friday, December 21, 2012 at 1:57 PM, Guido van Rossum wrote: > Dear python-dev *and* python-ideas, > > I am posting PEP 3156 here for early review and discussion. As you can > see from the liberally sprinkled TBD entries it is not done, but I am > about to disappear on vacation for a few weeks and I am reasonably > happy with the state of things so far. (Of course feedback may change > this. :-) Also, there has already been some discussion on python-ideas > (and even on Twitter) so I don't want python-dev to feel out of the > loop -- this *is* a proposal for a new standard library module. (But > no, I haven't picked the module name yet. :-) > > There's an -- also incomplete -- reference implementation at > http://code.google.com/p/tulip/ -- unlike the first version of tulip, > this version actually has (some) unittests. > > Let the bikeshedding begin! > > (Oh, happy holidays too. :-) > > -- > --Guido van Rossum (python.org/~guido (http://python.org/~guido)) > I really do like tulip as the name. It's quite pretty. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
On Fri, Dec 21, 2012 at 11:06 AM, Jesse Noller wrote: > I really do like tulip as the name. It's quite pretty. I chose it because Twisted and Tornado both start with T. But those have kind of dark associations; I wanted to offset that with something lighter. (OTOH we could use a black tulip as a logo. :-) Regardless, it's not the kind of name we tend to use for the stdlib. It'll probably end up being asynclib or something... -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3145 (With Contents)
Please stop copying me on this thread. Thanks, Jean-Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
Hello, > To get the current event loop, use get_event_loop(). This returns an > instance of the EventLoop class defined below or an equivalent > object. It is possible that get_event_loop() returns a different > object depending on the current thread, or depending on some other > notion of context. > > To set the current event loop, use set_event_loop(event_loop), where > event_loop is an instance of the EventLoop class or equivalent. This > uses the same notion of context as get_event_loop(). So can we instantiate an EventLoop directly and then call set_event_loop() with it? Or is the use case different? > - ``create_transport(protocol_factory, host, port, **kwargs)``. > Creates a transport and a protocol and ties them together. Returns > a Future whose result on success is a (transport, protocol) pair. > Note that when the Future completes, the protocol's > ``connection_made()`` method has not yet been called; that will > happen when the connection handshake is complete. When it is > impossible to connect to the given host and port, the Future will > raise an exception instead. > > Optional keyword arguments: > > - ``family``, ``type``, ``proto``, ``flags``: Address familty, > socket type, protcol, and miscellaneous flags to be passed through > to ``getaddrinfo()``. These all default to ``0`` except ``type`` > which defaults to ``socket.SOCK_STREAM``. > > - ``ssl``: Pass ``True`` to create an SSL transport (by default a > plain TCP is created). Or pass an ``ssl.SSLContext`` object to > override the default SSL context object to be used. > > TBD: Should this be called create_connection()? Either create_connection() or create_client(). create_transport() is wrong, since server transports wouldn't use that function. I would favour create_client() if this function is also meant to support UDP (I know you haven't thought about UDP yet, but it is an important and common use case). I have another question about that API: if I want to cancel the connection attempt after a given delay, how do I do that? If I call cancel() on the future, does it cancel the connect() call? As for SSL, there are security issues with having a "default SSL context" (notably, any decent client use of SSL *must* check the server certificate against an appropriate set of CAs). It's much better to force users to pass a context explicitly. Choosing default settings should only be for higher-level APIs like urllib.request. (btw, don't you mean that family defaults to AF_INET?) > If executor is None, a default ThreadPoolExecutor with 5 threads is used Is it because Twisted's thread pool has minThreads=5? :) > The transport is free to buffer the bytes, but it must eventually > cause the bytes to be transferred to the entity at the other end, and > it must maintain stream behavior. That is, t.write(b'abc'); > t.write(b'def') is equivalent to t.write(b'abcdef') I think this is a bad idea. The kernel's network stack should do the buffering (and choose appropriate algorithms for that), not the user-level framework. The transport should write the bytes as soon as the fd is ready for writing, and it should write the same chunks as given by the user, not a concatenation of them. Besides, it would be better if transports weren't automatically *streaming* transports. There are connected datagram protocols, such as named pipes under Windows (multiprocessing already uses non-blocking Windows named pipes). > Proposal: let the transport call protocol.pause() and > protocol.resume() if they exist; if they don't exist, the protocol > doesn't support flow control. +1. The Protocol base class can provide default no-op implementations. > TBD: Discuss whether user code needs to do anything to make sure that > protocol and transport aren't garbage-collected prematurely. The transport should be tied to the event loop as long as the connection holds, and the protocol will hold to the transport. > TBD: Need an interface to wait for the first of a collection of Futures. Have you looked at Twisted's DeferredList? http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.DeferredList.html I think par() could take a keyword-only argument to specify you want the callback to be triggered on the first result (and perhaps being able to choose between "the first success result" and "the first success or error result"). > A trick used by Richard Oudkerk in the tulip project's proactor > branch makes calls like recv() either return a regular result or > raise a Future. The caller (likely a transport) must then write code > like this: Isn't it a case of premature optimization? If we want to keep this, there should be a nicer API, perhaps like Twisted's maybeDeferred: http://twistedmatrix.com/documents/current/api/twisted.internet.defer.html#maybeDeferred > We might also introduce explicit locks (though these will be a bit of > a pain to use, as we can't use the with lock: block syntax). I
Re: [Python-Dev] peps: Specify start_serving(). Add Post-History.
On Fri, 21 Dec 2012 20:34:18 +0100 (CET) guido.van.rossum wrote: > > - In either case, once it has a socket, it will wrap it in a > - transport, and then enter a loop accepting connections (the best way > - to implement such a loop depends on the platform). Each time a > - connection is accepted, a transport and protocol are created for it. > + TBD: Support SSL? I don't even know how to do that synchronously, > + and I suppose it needs a certificate. You need a SSLContext, and that SSLContext must have a cert / key pair defined using the `load_cert_chain()` method. I supposed you meant "asynchronously", not "synchronously". The listening socket doesn't have to be a SSL socket, only the connected sockets need to be wrapped. The non-blocking handshake shouldn't be different (AFAICT) for a server SSL socket than for a client SSL socket. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] peps: Specify start_serving(). Add Post-History.
I really meant *synchronously*... I usually start with working sync code and then figure out what to do to make it async. I'll give what you suggest a try. --Guido van Rossum (sent from Android phone) On Dec 21, 2012 11:54 AM, "Antoine Pitrou" wrote: > On Fri, 21 Dec 2012 20:34:18 +0100 (CET) > guido.van.rossum wrote: > > > > - In either case, once it has a socket, it will wrap it in a > > - transport, and then enter a loop accepting connections (the best way > > - to implement such a loop depends on the platform). Each time a > > - connection is accepted, a transport and protocol are created for it. > > + TBD: Support SSL? I don't even know how to do that synchronously, > > + and I suppose it needs a certificate. > > You need a SSLContext, and that SSLContext must have a cert / key pair > defined using the `load_cert_chain()` method. > > I supposed you meant "asynchronously", not "synchronously". The > listening socket doesn't have to be a SSL socket, only the connected > sockets need to be wrapped. The non-blocking handshake shouldn't be > different (AFAICT) for a server SSL socket than for a client SSL socket. > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] peps: Specify start_serving(). Add Post-History.
On Fri, 21 Dec 2012 12:37:25 -0800 Guido van Rossum wrote: > I really meant *synchronously*... I usually start with working sync code > and then figure out what to do to make it async. I'll give what you suggest > a try. Ah. Then I hope the doc example can help you: http://docs.python.org/dev/library/ssl.html#server-side-operation Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
Looks reasonable to me :) Comments: create_transport "combines" a transport and a protocol. Is that process reversible? that might seem like an exotic thing (and I guess it kind of is), but I've wanted this e.g for websockets, and I guess there's a few other cases where it could be useful :) eof_received on protocols seems unusual. What's the rationale? I know we disagree that callbacks (of the line_received variety) are a good idea for blocking IO (I think we should have universal protocol implementations), but can we agree that they're what we want for tulip? If so, I can try to figure out a way to get them to fit together :) I'm assuming that this means you'd like protocols and transports in this PEP? A generic comment on yield from APIs that I'm sure has been discussed in some e-mail I missed: is there an obvious way to know up front whether something needs to be yielded or yield frommed? In twisted, which is what I'm used to it's all deferreds; but here a future's yield from but sleep's yield? Will comment more as I keep reading I'm sure :) On Fri, Dec 21, 2012 at 8:09 PM, Guido van Rossum wrote: > On Fri, Dec 21, 2012 at 11:06 AM, Jesse Noller wrote: > > I really do like tulip as the name. It's quite pretty. > > I chose it because Twisted and Tornado both start with T. But those > have kind of dark associations; I wanted to offset that with something > lighter. (OTOH we could use a black tulip as a logo. :-) > > Regardless, it's not the kind of name we tend to use for the stdlib. > It'll probably end up being asynclib or something... > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-ideas mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-ideas > -- cheers lvh ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
Inline. --Guido van Rossum (sent from Android phone) On Dec 21, 2012 11:47 AM, "Antoine Pitrou" wrote: > > > Hello, > > > To get the current event loop, use get_event_loop(). This returns an > > instance of the EventLoop class defined below or an equivalent > > object. It is possible that get_event_loop() returns a different > > object depending on the current thread, or depending on some other > > notion of context. > > > > To set the current event loop, use set_event_loop(event_loop), where > > event_loop is an instance of the EventLoop class or equivalent. This > > uses the same notion of context as get_event_loop(). > > So can we instantiate an EventLoop directly and then call > set_event_loop() with it? Or is the use case different? That's an abstract class, but if you know a concert implementation you can use this. E.g. latest Tulip unit tests. > > - ``create_transport(protocol_factory, host, port, **kwargs)``. > > Creates a transport and a protocol and ties them together. Returns > > a Future whose result on success is a (transport, protocol) pair. > > Note that when the Future completes, the protocol's > > ``connection_made()`` method has not yet been called; that will > > happen when the connection handshake is complete. When it is > > impossible to connect to the given host and port, the Future will > > raise an exception instead. > > > > Optional keyword arguments: > > > > - ``family``, ``type``, ``proto``, ``flags``: Address familty, > > socket type, protcol, and miscellaneous flags to be passed through > > to ``getaddrinfo()``. These all default to ``0`` except ``type`` > > which defaults to ``socket.SOCK_STREAM``. > > > > - ``ssl``: Pass ``True`` to create an SSL transport (by default a > > plain TCP is created). Or pass an ``ssl.SSLContext`` object to > > override the default SSL context object to be used. > > > > TBD: Should this be called create_connection()? > > Either create_connection() or create_client(). create_transport() is > wrong, since server transports wouldn't use that function. > > I would favour create_client() if this function is also meant to > support UDP (I know you haven't thought about UDP yet, but it is an > important and common use case). OK. > I have another question about that API: if I want to cancel the > connection attempt after a given delay, how do I do that? If I call > cancel() on the future, does it cancel the connect() call? It does in Tulip, because it's really a task. Maybe this should be in the spec? > As for SSL, there are security issues with having a "default SSL > context" (notably, any decent client use of SSL *must* check the server > certificate against an appropriate set of CAs). It's much better to > force users to pass a context explicitly. Choosing default settings > should only be for higher-level APIs like urllib.request. Hm. That makes simple tests harder. But I understand the concern. > (btw, don't you mean that family defaults to AF_INET?) > > > If executor is None, a default ThreadPoolExecutor with 5 threads is used > > Is it because Twisted's thread pool has minThreads=5? :) Yes, and to encourage the use of set_default_executor() ... :-) > > The transport is free to buffer the bytes, but it must eventually > > cause the bytes to be transferred to the entity at the other end, and > > it must maintain stream behavior. That is, t.write(b'abc'); > > t.write(b'def') is equivalent to t.write(b'abcdef') > > I think this is a bad idea. The kernel's network stack should do the > buffering (and choose appropriate algorithms for that), not the > user-level framework. The transport should write the bytes as soon as > the fd is ready for writing, and it should write the same chunks as > given by the user, not a concatenation of them. I asked Glyph about this. It depends on the OS... Mac syscalls are so slow that it is better to join in user space. This should really be up to the transport, although for stream transports the given equivalency should definitely hold. > Besides, it would be better if transports weren't automatically > *streaming* transports. There are connected datagram protocols, such as > named pipes under Windows (multiprocessing already uses non-blocking > Windows named pipes). I think we need to support datagrams, but the default ought to be stream. > > Proposal: let the transport call protocol.pause() and > > protocol.resume() if they exist; if they don't exist, the protocol > > doesn't support flow control. > > +1. The Protocol base class can provide default no-op implementations. OK. > > TBD: Discuss whether user code needs to do anything to make sure that > > protocol and transport aren't garbage-collected prematurely. > > The transport should be tied to the event loop as long as the > connection holds, and the protocol will hold to the transport. OK. > > TBD: Need an interface to wait for the first of a collection of Futures. > > Have you looked at Twisted's D
Re: [Python-Dev] PEP 3145 (With Contents)
On Fri, Dec 21, 2012 at 6:46 AM, Brett Cannon wrote: > > On Thu, Dec 20, 2012 at 7:35 PM, Chris Jerdonek > wrote: >> >> I don't disagree that he shouldn't have cross-posted. I was just >> pointing out that the language should be clarified. What's confusing >> is that the current language implies that one shouldn't send any >> PEP-related e-mails to any mailing list other than peps@. In >> particular, how can one discuss PEPs on python-dev or python-ideas >> without violating that language (e.g. this e-mail which is related to >> PEP 1)? It is probably just a matter of clarifying what "PEP-related" >> means. > > > I'm just not seeing the confusion, sorry. And we have never really had any > confusion over this wording before. If you want to send a patch to tweak the > wording to me more clear then please go ahead and I will consider it, but > I'm not worried enough about it to try to come up with some rewording > myself. I uploaded a proposed patch to this issue: http://bugs.python.org/issue16746 --Chris ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
As far as I understand, "yield from" will always work, because a Future object can act like an iterator, and you can delegate your own generator to this iterator at the place of "yield from". "yield" only works if the parameter behind yield is already a Future object. Right Guido? In case of sleep, sleep could be implemented to return a Future object. 2012/12/21 Laurens Van Houtven <[email protected]> > A generic comment on yield from APIs that I'm sure has been discussed in > some e-mail I missed: is there an obvious way to know up front whether > something needs to be yielded or yield frommed? In twisted, which is what > I'm used to it's all deferreds; but here a future's yield from but sleep's > yield? > > > > On Fri, Dec 21, 2012 at 8:09 PM, Guido van Rossum wrote: > >> On Fri, Dec 21, 2012 at 11:06 AM, Jesse Noller wrote: >> > I really do like tulip as the name. It's quite pretty. >> >> I chose it because Twisted and Tornado both start with T. But those >> have kind of dark associations; I wanted to offset that with something >> lighter. (OTOH we could use a black tulip as a logo. :-) >> >> Regardless, it's not the kind of name we tend to use for the stdlib. >> It'll probably end up being asynclib or something... >> >> -- >> --Guido van Rossum (python.org/~guido) >> ___ >> Python-ideas mailing list >> [email protected] >> http://mail.python.org/mailman/listinfo/python-ideas >> > > > > -- > cheers > lvh > > ___ > Python-ideas mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-ideas > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
We were tentatively calling it "concurrent.eventloop" at the 2011 language summit. -- Sent from my phone, thus the relative brevity :) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Fri, Dec 21, 2012 at 1:04 PM, Laurens Van Houtven <[email protected]> wrote: > Looks reasonable to me :) Comments: > > create_transport "combines" a transport and a protocol. Is that process > reversible? that might seem like an exotic thing (and I guess it kind of > is), but I've wanted this e.g for websockets, and I guess there's a few > other cases where it could be useful :) If you really need this, it's probably best to start out doing this as a nonstandard extension of an implementation. The current *implementation* makes it simple enough, but I don't think it's worth complicating the PEP. Working code might convince me otherwise. > eof_received on protocols seems unusual. What's the rationale? Well how else would you indicate that the other end did a half-close (in Twisted terminology)? You can't call connection_lost() because you might still want to write more. E.g. this is how HTTP servers work if there's no Content-length or chunked encoding on a request body: they read until EOF, then do their thing and write the response. > I know we disagree that callbacks (of the line_received variety) are a good > idea for blocking IO (I think we should have universal protocol > implementations), but can we agree that they're what we want for tulip? If > so, I can try to figure out a way to get them to fit together :) I'm > assuming that this means you'd like protocols and transports in this PEP? Sorry, I have no idea what you're talking about. Can you clarify? I do know that the PEP is weakest in specifying how a coroutine can implement a transport. However my plans are clear: ild the old tulip code there's a BufferedReader; somehow the coroutine will receive a "stdin" and a "stdout" where the "stdin" is a BufferedReader, which has methods like read(), readline() etc. which return Futures and must be invoked using yield from; and "stdout" is a transport, which has write() and friends that don't return anything but just buffer stuff and start the I/O asynchronous (and may try to slow down the protocol by calling its pause() method). > A generic comment on yield from APIs that I'm sure has been discussed in > some e-mail I missed: is there an obvious way to know up front whether > something needs to be yielded or yield frommed? In twisted, which is what > I'm used to it's all deferreds; but here a future's yield from but sleep's > yield? In PEP 3156 conformant code you're supposed always to use 'yield from'. The only time you see a bare yield is when it's part of the implementation's internals. (However I think tulip actually will handle a yield the same way as a yield from, except that it's slower because it makes a roundtrip to the scheduler, a.k.a. trampoline.) > Will comment more as I keep reading I'm sure :) Please do! -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Fri, Dec 21, 2012 at 2:26 PM, Jonathan Slenders wrote: > As far as I understand, "yield from" will always work, because a Future > object can act like an iterator, and you can delegate your own generator to > this iterator at the place of "yield from". > "yield" only works if the parameter behind yield is already a Future object. > Right Guido? Correct! Sounds like you got it now. That's the magic of yield from.. > In case of sleep, sleep could be implemented to return a Future object. It does; in tulip/futures.py: def sleep(when, result=None): future = Future() future._event_loop.call_later(when, future.set_result, result) return future -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] peps: Specify start_serving(). Add Post-History.
On Fri, Dec 21, 2012 at 1:00 PM, Antoine Pitrou wrote: > On Fri, 21 Dec 2012 12:37:25 -0800 > Guido van Rossum wrote: >> I really meant *synchronously*... I usually start with working sync code >> and then figure out what to do to make it async. I'll give what you suggest >> a try. > > Ah. Then I hope the doc example can help you: > http://docs.python.org/dev/library/ssl.html#server-side-operation Heh. Thanks. However I wouldn't know where to get a certificate. And for unittests, distributing a certificate sounds like an obvious bad idea. :-) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] peps: Specify start_serving(). Add Post-History.
2012/12/21 Guido van Rossum : > On Fri, Dec 21, 2012 at 1:00 PM, Antoine Pitrou wrote: >> On Fri, 21 Dec 2012 12:37:25 -0800 >> Guido van Rossum wrote: >>> I really meant *synchronously*... I usually start with working sync code >>> and then figure out what to do to make it async. I'll give what you suggest >>> a try. >> >> Ah. Then I hope the doc example can help you: >> http://docs.python.org/dev/library/ssl.html#server-side-operation > > Heh. Thanks. However I wouldn't know where to get a certificate. And > for unittests, distributing a certificate sounds like an obvious bad > idea. :-) It's fairly easy to generate a "fake" self-signed one for testing purposes. We already have some in the test suite. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Fri, Dec 21, 2012 at 5:17 PM, Jasper St. Pierre wrote: > On Fri, Dec 21, 2012 at 8:02 PM, Guido van Rossum wrote: > > ... snip ... > >> In PEP 3156 conformant code you're supposed always to use 'yield >> from'. The only time you see a bare yield is when it's part of the >> implementation's internals. (However I think tulip actually will >> handle a yield the same way as a yield from, except that it's slower >> because it makes a roundtrip to the scheduler, a.k.a. trampoline.) > > > Would it be possible to fail on "yield"? Silently being slower when you > forget to type a keyword is something I can imagine will creep up a lot by > mistake, and I don't think it's a good idea to silently be slower when the > only different is five more characters. That's also a possibility. If someone can figure out a patch that would be great. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted
On Fri, Dec 21, 2012 at 8:02 PM, Guido van Rossum wrote: ... snip ... In PEP 3156 conformant code you're supposed always to use 'yield > from'. The only time you see a bare yield is when it's part of the > implementation's internals. (However I think tulip actually will > handle a yield the same way as a yield from, except that it's slower > because it makes a roundtrip to the scheduler, a.k.a. trampoline.) > Would it be possible to fail on "yield"? Silently being slower when you forget to type a keyword is something I can imagine will creep up a lot by mistake, and I don't think it's a good idea to silently be slower when the only different is five more characters. > Will comment more as I keep reading I'm sure :) > > Please do! > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-ideas mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-ideas > -- Jasper ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
