[Python-Dev] Completing the email6 API changes.
R. David Murray writes:
> But I would certainly appreciate review from anyone so moved, since I
> haven't gotten any yet.
I'll try to make time for a serious (but obviously partial) review by
Monday.
I don't know if this is "serious" bikeshedding, but I have a comment
or two on the example:
> from email.message import MIMEMessage
> from email.headerregistry import Address
> fullmsg = MIMEMessage()
> fullmsg['To'] = Address('Foö Bar', '[email protected]')
> fullmsg['From'] = "mè "
> fullmsg['Subject'] = "j'ai un problème de python."
This is very nice! *I* *love* it.
But (sorry!) I worry that it's not obvious to "naive" users. Maybe it
would be useful to have a Message() factory which has one semantic
difference from MIMEMessage: it "requires" RFC 822-required headers
(or optionally RFC 1036 for news). Eg:
# This message will be posted and mailed
# These would conform to the latest Draft Standards
# and be DKIM-signed
fullmsg = Message('rfc822', 'rfc1036', 'dmarc')
I'm not sure how "required" would be implemented (perhaps through a
.validate() method). So the signature of the API suggested above is
Message(*validators, **kw).
For MIMEMessage, I think I prefer the name "MIMEPart". To naive
users, the idea of MIMEMessages containing MIMEMessages is a bit
disconcerting, except in the case of multipart/digest, I think.
> fullmsg.set_content("et la il est monté sur moi et il commence"
>" a m'étouffer.")
> htmlmsg = MIMEMessage()
> htmlmsg.set_content("et la il est monté sur moi et il commence"
> " a m'étouffer.",
> subtype='html')
I think I'd like to express the suite above as
fullmsg.payload.add_alternative(...)
fullmsg.payload.add_alternative(..., subtype='html')
This would automatically convert the MIME type of fullmsg to
'multipart/alternative', and .payload to a list where necessary.
.set_content() would be available but it's "dangerous" (it could
replace an arbitrary multipart -- this would be useful operation to
replace it with a textual URL or external-body part).
Aside: it occurs to me that the .payload attribute (and other such
attributes) could be avoided by the device of using names prefixed by
":" such as ":payload" as keys: "fullmsg[':payload']" since colon is
illegal in field names (cf RFC 5322). Probably I've just been writing
too much Common Lisp, though.
I'm not sure whether "payload" is a better name than "content" for
that attribute.
Now the suite
> with open('python.jpg', 'rb') as python:
> htmlmsg.add_related(python.read(), 'image', 'jpg', cid='image1'
> disposition='inline')
> fullmsg.make_alternative()
> fullmsg.attach(htmlmsg)
becomes just
with open('python.jpg', 'rb') as python:
fullmsg.payload['text/html'].add_related(...)
At this point, "fullmsg.add_related()" without the .payload attribute
would be an error, unless a "insertPart=True" keyword argument were
present. With "insertPart=True", a new top-level multipart/related
would be interposed with the existing multipart/alternative as its
first child, and the argument of add_related as the second. Maybe
that's too complicated, but I suspect it's harder for people who think
of MIME messages as trees, than for people who think of messages as
documents and don't want to hear about mimes other than Marcel
Marceau.
The indexing of the .payload attribute by part type is perhaps too
smart for my own good, haven't thought carefully about it. It's
plausible, though, since a message with multiple parts of the same
type can only have one displayed -- normally that shouldn't happen.
OTOH, this wouldn't work without modification for multipart/mixed or
multipart/related. Could use Yet Another Keyword Argument, maybe.
(BTW, it's really annoying when the text/plain part refers to images
etc that are attached only to the text/html part. AFAICT from RFC
2387 it ought to be possible to put the multipart/related part at the
top so both text/html and text/plain can refer to it.)
> with open('police-report.txt') as report:
> fullmsg.add_attachment(report.read(), filename='pölice-report.txt',
>params=dict(wrap='flow'), headers=(
> 'X-Secret-Level: top',
> 'X-Authorization: Monty'))
I can't find an RFC that specifies a "wrap" parameter to text/plain.
Do you mean RFC 2646 'format="flowed"' here?
(A "validate" method could raise a warning on unregistered parameters.)
> (Hmm. Looking at that I see I didn't fully fix a bug I had meant to fix:
> some of the parts have a MIME-Version header that don't need it.)
Another reason why the top-level part should be treated differently in
the API.
Steve
___
Python-Dev mailing list
[email protected]
ht
Re: [Python-Dev] Completing the email6 API changes.
On 31/08/13 15:21, R. David Murray wrote: If you've read my blog (eg: on planet python), you will be aware that I dedicated August to full time email package development. [...] The API looks really nice! Thank you for putting this together. A question comes to mind though: All input strings are unicode, and the library takes care of doing whatever encoding is required. When you pull data out of a parsed message, you get unicode, without having to worry about how to decode it yourself. How well does your library cope with emails where the encoding is declared wrongly? Or no encoding declared at all? Conveniently, your email is an example of this. Although it contains non-ASCII characters, it is declared as us-ascii: --===1633676851== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline which may explain why Stephen Turnbull's reply contains mojibake. -- Steven ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Completing the email6 API changes.
Steven D'Aprano writes: > which may explain why Stephen Turnbull's reply contains mojibake. Nah. It was already there, I just copied it. Could be my MUA's fault, though; I've tweaked it for Japanese, and it doesn't handle odd combinations well. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pre-PEP: Redesigning extension modules
*bump* Does this sound like a viable solution? Stefan Stefan Behnel, 25.08.2013 14:36: > Hi, > > thanks for bringing this up. It clearly shows that there is more to this > problem than I initially thought. > > Let me just add one idea that your post gave me. > > PJ Eby, 25.08.2013 06:12: >> My "Importing" package offers lazy imports by creating module objects >> in sys.modules that are a subtype of ModuleType, and use a >> __getattribute__ hook so that trying to use them fires off a reload() >> of the module. > > I wonder if this wouldn't be an approach to fix the reloading problem in > general. What if extension module loading, at least with the new scheme, > didn't return the module object itself and put it into sys.modules but > created a wrapper that redirects its __getattr__ and __setattr__ to the > actual module object? That would have a tiny performance impact on > attribute access, but I'd expect that to be negligible given that the usual > reason for the extension module to exist is that it does non-trivial stuff > in whatever its API provides. Reloading could then really create a > completely new module object and replace the reference inside of the wrapper. > > That way, code that currently uses "from extmodule import xyz" would > continue to see the original version of the module as of the time of its > import, and code that just did "import extmodule" and then used attribute > access at need would always see the current content of the module as it was > last loaded. I think that, together with keeping module global state in the > module object itself, would nicely fix both cases. > > Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Completing the email6 API changes.
On Sat, 31 Aug 2013 20:37:30 +1000, Steven D'Aprano wrote: > On 31/08/13 15:21, R. David Murray wrote: > > If you've read my blog (eg: on planet python), you will be aware that > > I dedicated August to full time email package development. > [...] > > > The API looks really nice! Thank you for putting this together. Thanks. > A question comes to mind though: > > > All input strings are unicode, and the library takes care of doing > > whatever encoding is required. When you pull data out of a parsed > > message, you get unicode, without having to worry about how to decode > > it yourself. > > How well does your library cope with emails where the encoding is declared > wrongly? Or no encoding declared at all? It copes as best it can :) The bad bytes are preserved (unless you modify a part) but are returned as the "unknown character" in a string context. You can get the original bytes out by using the bytes access interface. (There are probably some places where how to do that isn't clear in the current API, but bascially either you use BytesGenerator or you drop down to a lower level API.) An attempt is made to interpret "bad bytes" as utf-8, before giving up and replacing them with the 'unknown character' character. I'm not 100% sure that is a good idea. > Conveniently, your email is an example of this. Although it contains > non-ASCII characters, it is declared as us-ascii: Oh, yeah, my MUA is a little quirky and I forgot the step that would have made that correct. Wanting to rewrite it is one of the reasons I embarked on this whole email thing a few years ago :) --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Completing the email6 API changes.
On Sat, 31 Aug 2013 18:57:56 +0900, "Stephen J. Turnbull" wrote: > R. David Murray writes: > > > But I would certainly appreciate review from anyone so moved, since I > > haven't gotten any yet. > > I'll try to make time for a serious (but obviously partial) review by > Monday. Thanks. > I don't know if this is "serious" bikeshedding, but I have a comment > or two on the example: Yeah, you engaged in some serious bikeshedding there ;) I like the idea of a top level part that requires the required headers, and I agree that MIMEPart is better than MIMEMessage for that class. Full validation is something that is currently a "future objective". There's infrastructure to do it, but not all of the necessary knowledge has been coded in yet. I take your point about the relationship between related and alternative not being set in stone. I'll have to think through the consequences of that, but I think it is just a matter of removing a couple error checks and updating the documentation. I'll also have to sit and think through your other ideas (the more extensive bikeshedding :) before I can comment, and I'm heading out to take my step-daughter to her freshman year of college, so I won't be able to do thorough responses until tomorrow. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Completing the email6 API changes.
R. David Murray writes: > Full validation is something that is currently a "future > objective". I didn't mean it to be anything else. :-) > There's infrastructure to do it, but not all of the necessary knowledge > has been coded in yet. Well, I assume you already know that there's no way that can ever happen (at least until we abandon messaging entirely): new RFCs will continue to be published. So it needs to be an extensible mechanism, a "pipeline" of checks (Barry would say a "chain of rules", I think). Enjoy your trip! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pre-PEP: Redesigning extension modules
On 1 Sep 2013 00:10, "Stefan Behnel" wrote: > > *bump* > > Does this sound like a viable solution? This isn't likely to progress until we have Eric's PEP 451 to a point where it's ready for python-dev discussion and pronouncement. However, the revised loader API is being designed to allow for the loader returning arbitrary objects, so something along these lines should work. There will likely be some adjustments to the API signature to allow extension modules to optionally support reloading if they so desire. Cheers, Nick. > > Stefan > > > Stefan Behnel, 25.08.2013 14:36: > > Hi, > > > > thanks for bringing this up. It clearly shows that there is more to this > > problem than I initially thought. > > > > Let me just add one idea that your post gave me. > > > > PJ Eby, 25.08.2013 06:12: > >> My "Importing" package offers lazy imports by creating module objects > >> in sys.modules that are a subtype of ModuleType, and use a > >> __getattribute__ hook so that trying to use them fires off a reload() > >> of the module. > > > > I wonder if this wouldn't be an approach to fix the reloading problem in > > general. What if extension module loading, at least with the new scheme, > > didn't return the module object itself and put it into sys.modules but > > created a wrapper that redirects its __getattr__ and __setattr__ to the > > actual module object? That would have a tiny performance impact on > > attribute access, but I'd expect that to be negligible given that the usual > > reason for the extension module to exist is that it does non-trivial stuff > > in whatever its API provides. Reloading could then really create a > > completely new module object and replace the reference inside of the wrapper. > > > > That way, code that currently uses "from extmodule import xyz" would > > continue to see the original version of the module as of the time of its > > import, and code that just did "import extmodule" and then used attribute > > access at need would always see the current content of the module as it was > > last loaded. I think that, together with keeping module global state in the > > module object itself, would nicely fix both cases. > > > > Stefan > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] EINTR handling...
On Fri, Aug 30, 2013 at 10:57 AM, Guido van Rossum wrote: > I don't have a strong opinion on this either. The distinction between > send() and send_all() makes sense to me though (send_all() works hard to > get all your data out, send() only does what it can quickly). > > Personally for calls like select() I think returning early on EINTR makes > sense, it's usually part of a select loop anyway. > > The only thing I care deeply about is that you shouldn't restart anything > before letting a Python-level signal handler run. A program might well have > a Python signal handler that must run before the I/O is restarted, and the > signal handler might raise an exception (like the default SIGINT handler, > which raises KeyboardInterrupt). > I see http://bugs.python.org/issue18885 has been filed to track this discussion so we should probably move it there (I've added comments). TL;DR you can't simply retry a system call with the exact same arguments when you receive an EINTR. There are some system calls for which that will not do what the programmer intended. > > > On Fri, Aug 30, 2013 at 10:02 AM, Antoine Pitrou wrote: > >> On Fri, 30 Aug 2013 12:29:12 +0200 >> Charles-François Natali wrote: >> > >> > Furthermore, the stdlib code base is not consistent: some code paths >> > handle EINTR, e.g. subprocess, multiprocessing, sock_sendall() does >> > but not sock_send()... >> > Just grep for EINTR and InterruptedError and you'll be amazed. >> > >> > GHC, the JVM and probably other platforms handle EINTR, maybe it's >> > time for us too? >> >> I don't have any precise opinion on this. It's true that we should have >> a systematic approach, I just don't know if all interfaces should >> handler EINTR automatically, or only the high-level ones. >> (for the sake of clarity, I'm fine with either :-)) >> >> Regards >> >> Antoine. >> >> >> ___ >> Python-Dev mailing list >> [email protected] >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pre-PEP: Redesigning extension modules
Oops, had a draft from a few days ago that I was interrupted before sending. Finished editing the parts I believe are still relevant. On 25 Aug 2013 21:56, "Stefan Behnel" wrote: > > Nick Coghlan, 24.08.2013 23:43: > > On 25 Aug 2013 01:44, "Stefan Behnel" wrote: > >> Nick Coghlan, 24.08.2013 16:22: > >>> The new _PyImport_CreateAndExecExtensionModule function does the heavy > >>> lifting: > >>> > >>> https://bitbucket.org/ncoghlan/cpython_sandbox/src/081f8f7e3ee27dc309463b48e6c67cf4880fca12/Python/importdl.c?at=new_extension_imports#cl-65 > >>> > >>> One key point to note is that it *doesn't* call > >>> _PyImport_FixupExtensionObject, which is the API that handles all the > >>> PEP 3121 per-module state stuff. Instead, the idea will be for modules > >>> that don't need additional C level state to just implement > >>> PyImportExec_NAME, while those that *do* need C level state implement > >>> PyImportCreate_NAME and return a custom object (which may or may not > >>> be a module subtype). > >> > >> Is it really a common case for an extension module not to need any C level > >> state at all? I mean, this might work for very simple accelerator modules > >> with only a few stand-alone functions. But anything non-trivial will > >> almost > >> certainly have some kind of global state, cache, external library, etc., > >> and that state is best stored at the C level for safety reasons. In my experience, most extension authors aren't writing high performance C accelerators, they're exposing an existing C API to Python. It's the cffi use case rather than the Cython use case. My primary experience of C extensions is with such wrapper modules, and for those, the exec portion of the new API is exactly what you want. The components of the wrapper module don't share global state, they just translate between Python and a pre-existing externally stateless C API. For that use case, a precreated module to populate with types and functions is exactly what you want to keep things simple and stateless at the C level. > > I'd prefer to encourage people to put that state on an exported *type* > > rather than directly in the module global state. So while I agree we need > > to *support* C level module globals, I'd prefer to provide a simpler > > alternative that avoids them. > > But that has an impact on the API then. Why do you want the users of an > extension module to go through a separate object (even if it's just a > singleton, for example) instead of going through functions at the module > level? We don't currently encourage or propose this design for Python > modules either. Quite the contrary, it's extremely common for Python > modules to provide most of their functionality at the function level. And > IMHO that's a good thing. Mutable module global state is always a recipe for obscure bugs, and not something I will ever let through code review without a really good rationale. Hidden process global state is never good, just sometimes a necessary evil. However, keep in mind my patch is currently just the part I can implement without PEP 451 module spec objects. Once those are available, then I can implement the initial hook that supports returning a completely custom object. > Note that even global functions usually hold state, be it in the form of > globally imported modules, global caches, constants, ... If they can be shared safely across multiple instances of the module (e.g. immutable constants), then these can be shared at the C level. Otherwise, a custom Python type will be needed to make them instance specific. > > We also need the create/exec split to properly support reloading. Reload > > *must* reinitialize the object already in sys.modules instead of inserting > > a different object or it completely misses the point of reloading modules > > over deleting and reimporting them (i.e. implicitly affecting the > > references from other modules that imported the original object). > > Interesting. I never thought of it that way. > > I'm not sure this can be done in general. What if the module has threads > running that access the global state? In that case, reinitialising the > module object itself would almost certainly lead to a crash. My current proposal on import-sig is to make the first hook "prepare_module", and pass in the existing object in the reload case. For the extension loader, this would be reflected in the signature of the C level hook as well, so the module could decide for itself if it supported reloading. > And what if you do "from extmodule import some_function" in a Python > module? Then reloading couldn't replace that reference, just as for normal > Python modules. Meaning that you'd still have to keep both modules properly > alive in order to prevent crashes due to lost global state of the imported > function. > > The difference to Python modules here is that in Python code, you'll get > some kind of exception if state is lost during a reload. In C code, you'll > most likely get a cra
Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations
First, I really like this. +1 On Wed, Aug 28, 2013 at 6:07 PM, Victor Stinner wrote: > 2013/8/29 Victor Stinner : > > My proposed implementation for Python 3.4 is different: > > > > * no enable() / disable() function: tracemalloc can only be enabled > > before startup by setting PYTHONTRACEMALLOC=1 environment variable > > > > * traces (size of the memory block, Python filename, Python line > > number) are stored directly in the memory block, not in a separated > > hash table > > > > I chose PYTHONTRACEMALLOC env var instead of enable()/disable() > > functions to be able to really trace *all* memory allocated by Python, > > especially memory allocated at startup, during Python initialization. > > I'm not sure that having to set an environment variable is the most > convinient option, especially on Windows. > > Storing traces directly into memory blocks should use less memory, but > it requires to start tracemalloc before the first memory allocation. > It is possible to add again enable() and disable() methods to > dynamically install/uninstall the hook on memory allocators. I solved > this issue in the current implementation by using a second hash table > (pointer => trace). > > We can keep the environment variable as PYTHONFAULTHANDLER which > enables faulthandler at startup. faulthandler has also a command line > option: -X faulthandler. We may add -X tracemalloc. > We should be consistent with faulthandler's options. Why do you not want to support both the env var and enable()/disable() functions? Users are likely to want snapshots captured by enable()/disable() around particular pieces of code just as much as whole program information. Think of the possibilities, you could even setup a test runner to enable/disable before and after each test, test suite or test module to gather narrow statistics as to what code actually _caused_ the allocations rather than the ultimate individual file/line doing it. Taking that further: file and line information is great, but what if you extend the concept: could you allow for C API or even Python hooks to gather additional information at the time of each allocation or free? for example: Gathering the actual C and Python stack traces for correlation to figure out what call patterns lead allocations is powerful. (Yes, this gets messy fast as hooks should not trigger calls back into themselves when they allocate or free, similar to the "fun" involved in writing coverage tools) let me know if you think i'm crazy. :) -gps ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations
2013/8/29 Victor Stinner : > Charles-François Natali and Serhiy Storchaka asked me to add this > module somewhere in Python 3.4: "how about adding pyfailmalloc to the > main repo (maybe under Tools), with a script making it easy to run the > tests suite with it enabled?" There are two reasons I think it would be a great addition: - since OOM conditions are - almost - never tested, the OOM handling code is - almost - always incorrect: indeed, Victor has found and fixed several dozens crashes thanks to this module - this module is actually really simple (~150 LOC) I have two comments on the API: 1) failmalloc.enable(range: int=1000): schedule a memory allocation failure in random.randint(1, range) allocations. That's one shot, i.e. only one failure will be triggered. So if this failure occurs in a place where the code is prepared to handle MemoryError (e.g. bigmem tests), no failure will occur in the remaining test. It would be better IMO to repeat this (i.e. reset the next failure counter), to increase the coverage. 2) It's a consequence of 1): since only one malloc() failure is triggered, it doesn't really reflect how a OOM condition would appear in real life: usually, it's either because you've exhausted your address space or the machine is under memory pressure, which means that once you've hit OOM, you're likely to encounter it again on subsequent allocations, for example if your OOM handling code allocates new memory (that's why it's so complicated to properly handle OOM, and one might want to use "memory parachutes"). It might be interesting to be able to pass an absolute maximum memory usage, or an option where once you've triggered an malloc() failure, you record the current memory usage, and use it as ceiling for subsequent allocations. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] error in test suite
Am I the only one experiencing this? 262 tests OK. 93 tests failed: test___all__ test_abc test_array test_ast test_asynchat test_asyncore test_bisect test_buffer test_bufio test_bytes test_codeccallbacks test_codecs test_colorsys test_compileall test_configparser test_contextlib test_crypt test_ctypes test_dbm test_dbm_dumb test_dbm_ndbm test_dictcomps test_enum test_exceptions test_faulthandler test_file test_fileinput test_frozen test_future test_future3 test_future5 test_genericpath test_getargs2 test_getpass test_hash test_hashlib test_heapq test_idle test_imaplib test_imp test_import test_index test_io test_ioctl test_ipaddress test_iterlen test_json test_keyword test_largefile test_locale test_macpath test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn test_namespace_pkgs test_ntpath test_operator test_osx_env test_pdb test_pep352 test_posixpath test_print test_py_compile test_random test_regrtest test_robotparser test_runpy test_sched test_set test_shutil test_site test_smtpd test_sndhdr test_source_encoding test_sqlite test_stat test_strftime test_sundry test_tarfile test_textwrap test_threading test_time test_unicode test_univnewlines test_urllib test_urllib2net test_userstring test_uuid test_warnings test_wave test_webbrowser test_xml_dom_minicompat test_zipfile 24 tests skipped: test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses test_dbm_gnu test_devpoll test_gdb test_kqueue test_lzma test_msilib test_ossaudiodev test_smtpnet test_socketserver test_startfile test_timeout test_tk test_ttk_guionly test_urllibnet test_winreg test_winsound test_xmlrpc_net test_zipfile64 and the failure appears to always be: test [...] crashed -- Traceback (most recent call last): File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line 1265, in runtest_inner huntrleaks) File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line 1381, in dash_R indirect_test() File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line 1261, in test_runner = lambda: support.run_unittest(tests) File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", line 1683, in run_unittest _run_suite(suite) File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", line 1649, in _run_suite result = runner.run(suite) File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", line 1548, in run test(result) File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line 76, in __call__ return self.run(*args, **kwds) File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line 114, in run test(result) File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line 76, in __call__ return self.run(*args, **kwds) File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line 114, in run test(result) TypeError: 'NoneType' object is not callable ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] error in test suite
On Sat, 31 Aug 2013 10:23:27 -0700 Ethan Furman wrote: > Am I the only one experiencing this? http://bugs.python.org/issue11798 perhaps? Regards Antoine. > 262 tests OK. > 93 tests failed: > test___all__ test_abc test_array test_ast test_asynchat > test_asyncore test_bisect test_buffer test_bufio test_bytes > test_codeccallbacks test_codecs test_colorsys test_compileall > test_configparser test_contextlib test_crypt test_ctypes test_dbm > test_dbm_dumb test_dbm_ndbm test_dictcomps test_enum > test_exceptions test_faulthandler test_file test_fileinput > test_frozen test_future test_future3 test_future5 test_genericpath > test_getargs2 test_getpass test_hash test_hashlib test_heapq > test_idle test_imaplib test_imp test_import test_index test_io > test_ioctl test_ipaddress test_iterlen test_json test_keyword > test_largefile test_locale test_macpath test_multiprocessing_fork > test_multiprocessing_forkserver test_multiprocessing_spawn > test_namespace_pkgs test_ntpath test_operator test_osx_env > test_pdb test_pep352 test_posixpath test_print test_py_compile > test_random test_regrtest test_robotparser test_runpy test_sched > test_set test_shutil test_site test_smtpd test_sndhdr > test_source_encoding test_sqlite test_stat test_strftime > test_sundry test_tarfile test_textwrap test_threading test_time > test_unicode test_univnewlines test_urllib test_urllib2net > test_userstring test_uuid test_warnings test_wave test_webbrowser > test_xml_dom_minicompat test_zipfile > 24 tests skipped: > test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp > test_codecmaps_kr test_codecmaps_tw test_curses test_dbm_gnu > test_devpoll test_gdb test_kqueue test_lzma test_msilib > test_ossaudiodev test_smtpnet test_socketserver test_startfile > test_timeout test_tk test_ttk_guionly test_urllibnet test_winreg > test_winsound test_xmlrpc_net test_zipfile64 > > and the failure appears to always be: > > test [...] crashed -- Traceback (most recent call last): >File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line > 1265, in runtest_inner > huntrleaks) >File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line > 1381, in dash_R > indirect_test() >File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line > 1261, in > test_runner = lambda: support.run_unittest(tests) >File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", > line 1683, in run_unittest > _run_suite(suite) >File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", > line 1649, in _run_suite > result = runner.run(suite) >File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", > line 1548, in run > test(result) >File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line > 76, in __call__ > return self.run(*args, **kwds) >File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line > 114, in run > test(result) >File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line > 76, in __call__ > return self.run(*args, **kwds) >File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line > 114, in run > test(result) > TypeError: 'NoneType' object is not callable ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] error in test suite
Sorry, this is mine. This is related to http://bugs.python.org/issue11798 Error happens when tests regrtest executed with -R option. I've temporary disabled this feature until finally fix it. On Sat, Aug 31, 2013 at 8:23 PM, Ethan Furman wrote: > Am I the only one experiencing this? > > 262 tests OK. > 93 tests failed: > test___all__ test_abc test_array test_ast test_asynchat > test_asyncore test_bisect test_buffer test_bufio test_bytes > test_codeccallbacks test_codecs test_colorsys test_compileall > test_configparser test_contextlib test_crypt test_ctypes test_dbm > test_dbm_dumb test_dbm_ndbm test_dictcomps test_enum > test_exceptions test_faulthandler test_file test_fileinput > test_frozen test_future test_future3 test_future5 test_genericpath > test_getargs2 test_getpass test_hash test_hashlib test_heapq > test_idle test_imaplib test_imp test_import test_index test_io > test_ioctl test_ipaddress test_iterlen test_json test_keyword > test_largefile test_locale test_macpath test_multiprocessing_fork > test_multiprocessing_forkserver test_multiprocessing_spawn > test_namespace_pkgs test_ntpath test_operator test_osx_env > test_pdb test_pep352 test_posixpath test_print test_py_compile > test_random test_regrtest test_robotparser test_runpy test_sched > test_set test_shutil test_site test_smtpd test_sndhdr > test_source_encoding test_sqlite test_stat test_strftime > test_sundry test_tarfile test_textwrap test_threading test_time > test_unicode test_univnewlines test_urllib test_urllib2net > test_userstring test_uuid test_warnings test_wave test_webbrowser > test_xml_dom_minicompat test_zipfile > 24 tests skipped: > test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp > test_codecmaps_kr test_codecmaps_tw test_curses test_dbm_gnu > test_devpoll test_gdb test_kqueue test_lzma test_msilib > test_ossaudiodev test_smtpnet test_socketserver test_startfile > test_timeout test_tk test_ttk_guionly test_urllibnet test_winreg > test_winsound test_xmlrpc_net test_zipfile64 > > and the failure appears to always be: > > test [...] crashed -- Traceback (most recent call last): > File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line > 1265, in runtest_inner > huntrleaks) > File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line > 1381, in dash_R > indirect_test() > File "/home/ethan/source/python/issue18780/Lib/test/regrtest.py", line > 1261, in > test_runner = lambda: support.run_unittest(tests) > File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", > line 1683, in run_unittest > _run_suite(suite) > File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", > line 1649, in _run_suite > result = runner.run(suite) > File "/home/ethan/source/python/issue18780/Lib/test/support/__init__.py", > line 1548, in run > test(result) > File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line > 76, in __call__ > return self.run(*args, **kwds) > File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line > 114, in run > test(result) > File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line > 76, in __call__ > return self.run(*args, **kwds) > File "/home/ethan/source/python/issue18780/Lib/unittest/suite.py", line > 114, in run > test(result) > TypeError: 'NoneType' object is not callable > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pre-PEP: Redesigning extension modules
Nick Coghlan, 31.08.2013 18:49: > On 25 Aug 2013 21:56, "Stefan Behnel" wrote: > One key point to note is that it *doesn't* call > _PyImport_FixupExtensionObject, which is the API that handles all the > PEP 3121 per-module state stuff. Instead, the idea will be for modules > that don't need additional C level state to just implement > PyImportExec_NAME, while those that *do* need C level state implement > PyImportCreate_NAME and return a custom object (which may or may not > be a module subtype). Is it really a common case for an extension module not to need any C level state at all? I mean, this might work for very simple accelerator modules with only a few stand-alone functions. But anything non-trivial will almost certainly have some kind of global state, cache, external library, etc., and that state is best stored at the C level for safety reasons. > > In my experience, most extension authors aren't writing high performance C > accelerators, they're exposing an existing C API to Python. It's the cffi > use case rather than the Cython use case. Interesting. I can't really remember a case where I could afford the runtime overhead of implementing a wrapper in Python and going through something like ctypes or cffi. I mean, testing C libraries with Python tools would be one, but then, you wouldn't want to write an extension module for that and instead want to call it directly from the test code as directly as possible. I'm certainly aware that that use case exists, though, and also the case of just wanting to get things done as quickly and easily as possible. > Mutable module global state is always a recipe for obscure bugs, and not > something I will ever let through code review without a really good > rationale. Hidden process global state is never good, just sometimes a > necessary evil. I'm not necessarily talking about mutable state. Rather about things like pre-initialised data or imported functionality. For example, I often have a bound method of a compiled regex lying around somewhere in my Python modules as a utility function. And the same kind of stuff exists in C code, some may be local to a class, but other things can well be module global. And given that we are talking about module internals here I'd always keep them at the C level rather than exposing them through the module dict. The module dict involves a much higher access overhead, in addition to the reduced safety due to user accessibility. Exported C-APIs are also a use case. You'd import the C-API of another module at init time and from that point on only go through function pointers etc. Those are (sub-)interpreter specific, i.e. they are module global state that is specific to the currently loaded module instances. > However, keep in mind my patch is currently just the part I can implement > without PEP 451 module spec objects. Understood. >> Note that even global functions usually hold state, be it in the form of >> globally imported modules, global caches, constants, ... > > If they can be shared safely across multiple instances of the module (e.g. > immutable constants), then these can be shared at the C level. Otherwise, a > custom Python type will be needed to make them instance specific. I assume you meant a custom module (extension) type here. Just to be clear, the "module state at the C-level" is meant to be stored in the object struct fields of the extension type that implements the module, at least for modules that want to support reloading and sub-interpreters. Obviously, nothing should be stored in static (global) variables etc. >>> We also need the create/exec split to properly support reloading. Reload >>> *must* reinitialize the object already in sys.modules instead of >>> inserting >>> a different object or it completely misses the point of reloading >>> modules >>> over deleting and reimporting them (i.e. implicitly affecting the >>> references from other modules that imported the original object). >> >> Interesting. I never thought of it that way. >> >> I'm not sure this can be done in general. What if the module has threads >> running that access the global state? In that case, reinitialising the >> module object itself would almost certainly lead to a crash. > > My current proposal on import-sig is to make the first hook > "prepare_module", and pass in the existing object in the reload case. For > the extension loader, this would be reflected in the signature of the C > level hook as well, so the module could decide for itself if it supported > reloading. I really don't like the idea of reloading by replacing module state. It would be much simpler if the module itself would be replaced, then the original module could stay alive and could still be used by those who hold a reference to it or parts of its contents. Especially the from-import case would benefit from this. Obviously, you could still run into obscure bugs whe
Re: [Python-Dev] Pre-PEP: Redesigning extension modules
On Sat, 31 Aug 2013 21:16:10 +0200 Stefan Behnel wrote: > > Our experience is very different - my perspective is that the normal case > > either eschews C level global state in the extension module, because it > > causes so many problems, or else just completely ignores subinterpreter > > support and proper module cleanup. > > As soon as you have more than one extension type in your module, and they > interact with each other, they will almost certainly have to do type checks > against each other to make sure users haven't passed them rubbish before > they access any C struct fields of the object. Doing a type check means > that at least one type has a pointer to the other, meaning that it holds > global module state. > > I really think that having some kind of global module state is the > exceedingly common case for an extension module. Since we are eating our own dogfood here (and the work which prompted this discussion was indeed about trying to make our extension modules more cleanup-friendly), it would be nice to take a look at the Modules directory and count which proportion of CPython extension modules have state. Caution: "state" is a bit vague here. Depending on which API you use, custom extension types can be a part of "module state". Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pre-PEP: Redesigning extension modules
Antoine Pitrou, 31.08.2013 21:27: > On Sat, 31 Aug 2013 21:16:10 +0200 > Stefan Behnel wrote: >>> Our experience is very different - my perspective is that the normal case >>> either eschews C level global state in the extension module, because it >>> causes so many problems, or else just completely ignores subinterpreter >>> support and proper module cleanup. >> >> As soon as you have more than one extension type in your module, and they >> interact with each other, they will almost certainly have to do type checks >> against each other to make sure users haven't passed them rubbish before >> they access any C struct fields of the object. Doing a type check means >> that at least one type has a pointer to the other, meaning that it holds >> global module state. >> >> I really think that having some kind of global module state is the >> exceedingly common case for an extension module. > > Since we are eating our own dogfood here (and the work which prompted > this discussion was indeed about trying to make our extension modules > more cleanup-friendly), it would be nice to take a look at the Modules > directory and count which proportion of CPython extension modules have > state. There seem to be 81 modules in there currently (grepped for PyMODINIT_FUNC). 16 of them come up when you grep for '(TypeCheck|IsInstance)', all using global extension type pointers. 32 use some kind of global "static PyObject* something;". Both add up to 41. That's half of the modules already. I'm sure there's more if you dig deeper. Some modules only define functions or only one type (e.g. md5). They would get away with no global state, I guess - if they all used heap types. > Caution: "state" is a bit vague here. Depending on which API you use, > custom extension types can be a part of "module state". Yep, as I said. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations
Le 31 août 2013 19:09, "Gregory P. Smith" a écrit : > First, I really like this. +1 Current votes: +3 (i also want tracemalloc!). No opposition against such addition. > We should be consistent with faulthandler's options. Why do you not want to support both the env var and enable()/disable() functions? The reason was technical but I have issues with enabling tracemalloc before Python is fully initialized. Enabling tracemalloc when Python is almost initialized and disabling it before Python exit is more reliable. In the last implementation, enable() and disable() are back. PYTHONTRACEMALLOC is still available. I will also add -X tracemalloc command line. > Users are likely to want snapshots captured by enable()/disable() around particular pieces of code just as much as whole program information. enable()/disable() are useful for tests of the tracemalloc module! > Taking that further: file and line information is great, but what if you extend the concept: could you allow for C API or even Python hooks to gather additional information at the time of each allocation or free? for example: Gathering the actual C and Python stack traces for correlation to figure out what call patterns lead allocations is powerful. There is no portable function to retrieve the C traceback. For the Python traceback: it may be possible to get it but you have to remember that the hook is on PyMem_Malloc(), a low level memory allocator. The GIL is hold. It is possible to call Python code in some cases, but they are corner cases where it would lead to a loop, deadlock or worse. I prefer to only read available Python data without calling any Python code and try to write a reliable debug tool. I still have some issues like issues with reentrant calls to the hook, but I'm working on them. A nice improvement compared to the implementation on PyPI would be to hook PyMem_RawMalloc(), to trace also the memory used by gzip, bz2, lzma and other C modules. Locking the GIL in PyMem_RawMalloc() to get the filename and locking internals tracemalloc structures causes also new issues (like a funny deadlock related to subinterpreter). > (Yes, this gets messy fast as hooks should not trigger calls back into themselves when they allocate or free, similar to the "fun" involved in writing coverage tools) tracemalloc uses a "reentrant" variable to do nothing on a reentrant call to the hook. > let me know if you think i'm crazy. :) You are not crazy. It is just hard to implement it. I'm not sure that it is possible. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pre-PEP: Redesigning extension modules
On 1 Sep 2013 05:18, "Stefan Behnel" wrote: > > Nick Coghlan, 31.08.2013 18:49: > > On 25 Aug 2013 21:56, "Stefan Behnel" wrote: > > One key point to note is that it *doesn't* call > > _PyImport_FixupExtensionObject, which is the API that handles all the > > PEP 3121 per-module state stuff. Instead, the idea will be for modules > > that don't need additional C level state to just implement > > PyImportExec_NAME, while those that *do* need C level state implement > > PyImportCreate_NAME and return a custom object (which may or may not > > be a module subtype). > > Is it really a common case for an extension module not to need any C > level > state at all? I mean, this might work for very simple accelerator > modules > with only a few stand-alone functions. But anything non-trivial will > almost > certainly have some kind of global state, cache, external library, > etc., > and that state is best stored at the C level for safety reasons. > > > > In my experience, most extension authors aren't writing high performance C > > accelerators, they're exposing an existing C API to Python. It's the cffi > > use case rather than the Cython use case. > > Interesting. I can't really remember a case where I could afford the > runtime overhead of implementing a wrapper in Python and going through > something like ctypes or cffi. I mean, testing C libraries with Python > tools would be one, but then, you wouldn't want to write an extension > module for that and instead want to call it directly from the test code as > directly as possible. > > I'm certainly aware that that use case exists, though, and also the case of > just wanting to get things done as quickly and easily as possible. Keep in mind I first came to Python as a tool for test automation of custom C++ hardware APIs that could be written to be SWIG friendly. I now work for an OS vendor where the 3 common languages for system utilities are C, C++ and Python. For those use cases, dropping a bunch of standard Python objects in a module dict is often going to be a quick and easy solution that avoids a lot of nasty pointer lifecycle issues at the C level. This style of extension code would suffer similar runtime checking overhead as Python, including for function calls, but, like CPython itself, would still often be "fast enough". However, as soon as you want to manually optimise for *speed* at all, you're going to want to remove those module internal indirections through the Python API. There are at least three ways to do this (internally, CPython uses all of them in various places): * type checks followed by direct function calls on the optimised path and falling back to the abstract object APIs on the compatibility path * type checks followed by an exception for unknown types * hidden state that isn't exposed directly at the Python level and hence can be trusted to only be changed through the module APIs. The third approach can be implemented in three ways, with various consequences: * C static variables. For mutable state, including pointers to Python types, this breaks subinterpreters, reloading in place and loading a fresh copy of the module * PEP 3121 per-interpreter shared state. Handles subinterpreters, *may* handle reloading (but may segfault if references are held to old types and functions from before the reload), doesn't handle loading a fresh copy at all. * PEP 3121 with a size of "0". As above, but avoids the module state APIs in order to support reloading. All module state (including type cross-references) is stored in hidden state (e.g. an instance of a custom type not exposed to Python, with a reference stored on each custom type object defined in the module, and any module level "functions" actually being methods of a hidden object). Still doesn't support loading a *fresh* copy due to the hidden PEP 3121 module cache. The proposed new approach is to bypass the PEP 3121 cache entirely, and instead recommend providing an instance of a custom type to be placed in sys.modules. Extension modules will be given the ability to explicitly disallow in-place reloading *or* to make it work reliably, rather than the status quo where the import system assumes it will work, and instead may fail in unusual ways. > > Mutable module global state is always a recipe for obscure bugs, and not > > something I will ever let through code review without a really good > > rationale. Hidden process global state is never good, just sometimes a > > necessary evil. > > I'm not necessarily talking about mutable state. Rather about things like > pre-initialised data or imported functionality. For example, I often have a > bound method of a compiled regex lying around somewhere in my Python > modules as a utility function. And the same kind of stuff exists in C code, > some may be local to a class, but other things can well be module global. > And given that we are talking about module interna
Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations
+1 from me for both tracemalloc and failmalloc in the same vein as faulthandler (and using similar API concepts to faulthandler). While I like the flat top level naming structure, we should clearly document these as implementation dependent runtime services. Other implementations may not provide them at all, and even if they do, many details will likely be different. The gc module may even fall into the same category. Cheers, Nick. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
