[issue19838] test.test_pathlib.PosixPathTest.test_touch_common fails on FreeBSD with ZFS
Larry Hastings added the comment: I can confirm that the behavior is fixed in ZFS on Linux. My test case C program now prints "Everything is okay." when run on a ZFS partition on Linux, and test_touch_common from the current tree passes every time. ZFS fixing this was the best possible outcome. I'll go ahead and just close it now--why wait! If somebody confirms that the test still fails on FreeBSD, please open a new issue. Thanks for checking in, Irit! -- stage: patch review -> resolved status: pending -> closed ___ Python tracker <https://bugs.python.org/issue19838> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43424] Document the `controller.name` field in `webbrowser` module
Change by Larry Hastings : -- components: -2to3 (2.x to 3.x conversion tool), Build, C API, Cross-Build, Demos and Tools, Distutils, Documentation, Extension Modules, FreeBSD, IDLE, IO, Installation, Interpreter Core, Library (Lib), Parser, Regular Expressions, SSL, Subinterpreters, Tests, Tkinter, Unicode, Windows, XML, asyncio, ctypes, email, macOS ___ Python tracker <https://bugs.python.org/issue43424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43424] Document the `controller.name` field in `webbrowser` module
Change by Larry Hastings : -- components: +Library (Lib) -Argument Clinic ___ Python tracker <https://bugs.python.org/issue43424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43424] Document the `controller.name` field in `webbrowser` module
Change by Larry Hastings : -- components: +Documentation nosy: -larry ___ Python tracker <https://bugs.python.org/issue43424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46347] memory leak in PyEval_EvalCodeEx
Larry Hastings added the comment: The function will still leak "kwnames" and "default" if creating the "func" object fails. Admittedly that would only happen in a low-memory condition which is a) rare and b) probably only happens just before the interpreter completely dies, so it's not worth addressing during today's mild emergency. -- nosy: +larry ___ Python tracker <https://bugs.python.org/issue46347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46347] memory leak in PyEval_EvalCodeEx
Larry Hastings added the comment: (Sorry--it'll leak "kwnames", "newargs", and "defaults".) -- ___ Python tracker <https://bugs.python.org/issue46347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: So, can we shoot for adding this to 3.11? Jack, do you consider the code is in good shape? I'd be up for shepherding it along in the process. In particular, I can contribute the bindings so BLAKE3 is a first-class citizen of hashlib. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46314] Stray RESUME opcode for unused lambda
Change by Larry Hastings : -- components: -2to3 (2.x to 3.x conversion tool), Argument Clinic, Build, C API, Cross-Build, Demos and Tools, Distutils, Documentation, Extension Modules, FreeBSD, IDLE, IO, Installation, Library (Lib), Parser, Regular Expressions, SSL, Subinterpreters, Tests, Tkinter, Unicode, Windows, XML, asyncio, ctypes, email, macOS nosy: -larry ___ Python tracker <https://bugs.python.org/issue46314> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: > In setup.py I assume that the target platform of the build is the same as the > current interpreter's platform. If this is included in CPython, it won't be using setup.py, so this isn't a concern. I don't think there's a way to use setup.py to cross-compile, so I'm not sure this ever was a concern. > - Compiling assembly files. AFAICT Python currently ships exactly one assembly file, "Modules/_decimal/libmpdec/vcdiv64.asm", which is only built on Windows. It would be a brave new world of configure.ac hacking to build assembly language files on POSIX platforms. As a first pass I say we merge the reference C implementation. Maybe someday we could add the SIMD assembly language stuff--or use the one built in to OpenSSL (if they ever add BLAKE3). > I assume we don't want to check in the .obj files? Correct, we don't. > - blake3module.c contains an awful lot of gotos to handle allocation failure > cases. Works for me, please keep it. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: I assume by "intrinsics" you mean using the GCC SIMD stuff, not like inlining memcpy() or something. My assumption is yes, that's fine, we appear to already be using them for BLAKE2. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
New submission from Larry Hastings : It's considered good hygiene to use functools.update_wrapper() to make your wrapped functions look like the original. However, when using functools.partial() to pre-supply arguments to a function, if you then call functools.update_wrapper() to update that partial object, inspect.signature() returns the *original* function's signature, not the *wrapped* function's signature. To be precise: if you wrap a function with functools.partial() to pre-provide arguments, then immediately call inspect.signature() on that partial object, it returns the correct signature with the pre-filled parameters removed. If you then call functools.update_wrapper() to update the partial from the original function, inspect.signature() now returns the *wrong* signature. I looked into it a little. The specific thing changing inspect.signature()'s behavior is the '__wrapped__' attribute added by functools.update_wrapper(). By default inspect.signature() will unwrap partial objects, but only if it has a '__wrapped__' attribute. This all looks pretty deliberate. And it seems like there was some thought given to this wrinkle; inspect.signature() takes a "follow_wrapper_chains" parameter the user can supply to control this behavior. But the default is True, meaning that by default it unwraps partial objects if they have a '__wrapped__'. I admit I don't have any context for this. Why do we want inspect.signature() to return the wrong signature by default? -- components: Library (Lib) files: update_wrapper.breaks.partial.signature.test.py messages: 413299 nosy: larry priority: normal severity: normal stage: test needed status: open title: functools.update_wrapper breaks the signature of functools.partial objects type: behavior versions: Python 3.10, Python 3.11, Python 3.7, Python 3.8, Python 3.9 Added file: https://bugs.python.org/file50625/update_wrapper.breaks.partial.signature.test.py ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: I thought someone volunteered to do it--if that's not happening, I could take a look at it next week. Shouldn't be too hard... unless I have to touch autoconf, which I only barely understand. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46783] Add a new feature to enumerate(iterable, start=0) built-in function
Change by Larry Hastings : -- components: -Argument Clinic nosy: -larry ___ Python tracker <https://bugs.python.org/issue46783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Just checking--I can liberally pull code from https://github.com/BLAKE3-team/BLAKE3 yes? -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: Yury, Ka-Ping, can you guys shed any light on this? Your names are still on inspect.py. -- nosy: +ping, yselivanov ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46807] Wrong class __annotations__ when field name and type are equal
Larry Hastings added the comment: Yeah, it's the same behavior. In that class, you have defined Str to be "", then you reference Str in the annotation, and its value is "". Whatever you set Str to in the example in [21], that's the value of the annotation. GvR closed the previous report as "not a bug", so I'm gonna do that here too. -- ___ Python tracker <https://bugs.python.org/issue46807> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: Ofey, I appreciate your enthusiasm, but you should probably slow down. Fixing the bug is probably going to be the easy part here. But we're not to that stage quite yet. First, we need to determine * why the code behaves like this--is this behavior a genuine bug, or is it actually a bugfix for some worse behavior? * will fixing the bug cause problems for Python users? and if so, can we still fix the bug while mitigating the damage to people who are unfortunately depending on the bug? The next step is not to write a bugfix for this exact behavior, it's to determine why the code is the way it is. If it was genuinely just a mistake, and we can simply fix it and people will thank us, then we may have a use for your patch. But, generally, people who work on Python are smart, and they don't tend to commit dumb mistakes, so we can't simply assume it's a simple bug and fix it. -- ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: Okay, so, I considered the problem for a while, and I have a reasonable theory about what follow_wrapper_chains was for in the first place. If you have a generic decorator, like functools.cache(), it usually looks like this: def my_decorator(fn): def generic_version(*args, **kwargs): args, kwargs = do_something(args, kwargs) return fn(*args, **kwargs) return generic_version @my_decorator def add_five(i): return i+5 If you take the signature of add_five, you'd get (*args, **kwargs), because that's the signature of the wrapper function returned by the decorator. The decorator doesn't change the parameters of the function, but because of how decorators work it can occlude the proper function signature. In that instance, follow_wrapper_chains does the right thing, and as a result you get a precise function signature. (Of course, it would do the wrong thing if your hand-written decorator *also* behaved like a partial application, adding in its own hard-coded arguments, so that the resulting function signature changed.) Still, obviously it's doing the wrong thing when it comes to functools.partial() functions. My suspicion is that I'm the rare individual who actually uses update_wrapper on a functools.partial object. So maybe we have the situation here where, yeah, it's a bug, and we can fix it without causing further breakage. Maybe we can loop in someone who works on a popular runtime function introspection library (FastAPI, Pydantic) to see if they have any take on it. -- ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46846] functools.partial objects should set __signature__ and _annotations__
New submission from Larry Hastings : I ran across an interesting bug in issue #46761. If you call functools.update_wrapper on a functools.partial object, inspect.signature will return the wrong (original) signature for the partial object. We're still figuring that one out. And, of course, it's telling that the bug has been there for a long time. I suspect this isn't something that has inconvenienced a lot of people. But: I suggest that it's time functools.partial participated in signature stuff. Specifically, I think functools.partial should generate a new and correct __signature__ for the partial object. And I propose it should also generate a new and correct __annotations__ for the partial, by removing all entries for parameters that are filled in by the partial object. Right now inspect.signature has special support for functools.partial objects. It finds the underlying function, and . Which means there's code in both modules that has to understand the internals of partial objects. Just from a code hygiene perspective, it'd be better if all that logic lived under functools. I wonder if functools.partial objects should generally do a better job of impersonating the original function. Should they adopt the same __name__? __file__? __qualname__? My intuition is, it'd be nice if it did. But I might be forgetting something important. (I suspect everything I said about functools.partial also applies to functools.partialmethod.) -- components: Library (Lib) messages: 413897 nosy: larry, rhettinger priority: normal severity: normal stage: test needed status: open title: functools.partial objects should set __signature__ and _annotations__ type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46846> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46847] functools.update_wrapper doesn't understand partial objects and annotations
New submission from Larry Hastings : functools.update_wrapper currently copies over every attribute listed in the "assigned" parameter, which defaults to WRAPPER_ASSIGNMENTS, which means it copies the wrapped function's __annotations__ to the wrapper. This is slightly wrong if the wrapper occludes an annotated parameter: def foo(a: int, b: str, c: float): print(a, b, c) import functools foo_a = functools.partial(foo, 3) functools.update_wrapper(foo_a, foo) print(foo_a.__annotations__) In this case, foo_a.__annotations__ contains an annotation for a parameter named "a", even though foo_a doesn't have a parameter named "a". This problem occurred to me just after I filed #46846; the two issues are definitely related. -- components: Library (Lib) messages: 413898 nosy: larry, rhettinger priority: normal severity: normal stage: test needed status: open title: functools.update_wrapper doesn't understand partial objects and annotations type: behavior versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46847> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28824] os.environ should preserve the case of the OS keys ?
Change by Larry Hastings : -- nosy: -larry, loewis ___ Python tracker <https://bugs.python.org/issue28824> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: I emailed the Pydantic and FastAPI guys and didn't hear back. Given what you found on their issue trackers, I think it's unlikely that they care a lot about this issue (but were too busy to reply). It's far more likely that they don't care. Doing a little research (git blame), it looks like the "follow the wrapped chain to find the original signature" work was done by Nick Coghlan about nine years ago; he touched both functools.update_wrapper and the inspect module. The only other people to touch the code recently are Yuri and Batuhan. I've nosied Nick and Batuhan (already looped in Yuri), just to ping them and see if they have any strong opinions. If nobody has anything remarkable to say about it, honestly we probably *can* just merge your patch, Ofey. I see your name is now marked with a star; are you now authorized to contribute to CPython? (Note that I only glanced at your patch so far; if we were going to merge this bugfix I would of course first do a full review.) -- nosy: +BTaskaya, ncoghlan ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Change by Larry Hastings : -- pull_requests: +29805 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/31686 ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Okay, so. Here's a PR that adds BLAKE3 support to hashlib. The code was straightforward; I just took the BLAKE2 module and modified it to only have one algorithm. I also copied over the whole C directory tree from BLAKE3, which is totally okay fine by their license. I didn't have to modify it a bit. The tricky part was building it. Building extension modules is done inside setup.py using distutils (or the modern replacement), and in order to get those sexy SIMD implementations I had to compile assembly-language files, or C files with extra flags. What I did works great on my Linux box, and I have my fingers crossed that it'll work on other POSIX platforms. I wrote a lng comment in setup.py explaining what I did and why. Steve: I didn't do anything for Windows support. Do you have time to take a pass at it? Or if you know another core dev who does Windows build stuff, please nosy them. Whoever does the work. I suggest you read https://github.com/BLAKE3-team/BLAKE3/blob/master/c/README.md for a little guidance on how to build BLAKE3 on Windows with SIMD support. Also, I see you now build Python for Windows on ARM! Does this mean Python can use BLAKE3's NEON support? Maybe it's time to find out! Get hyped! Jack and I corresponded last year (or maybe 2020?) about what the API should look like. The idea is, Jack also maintains a BLAKE3 Python extension on pypi, written in Rust. Our goal was to make the two types behave identically, so that it could be like the old stringio / cstringio situation. You can use the built-in one for convenience, but also you can install the Rust version from PyPI for even more performance. Jack, it wouldn't hurt my feelings overly much if you checked my PR to make sure I got the interface right. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Also, for what it's worth: I just ran my checksum benchmarker using a freshly built python a la my PR. Here are my results when hashing 462183782 bytes (dicey-dungeons-linux64.zip): hash algorithm timebytes/sec size hash - -- -- - blake3 0.18976406 2435570699 64 0c72d0a07ba767b75b0c99fed38fda... sha1 0.21692419 2130623562 40 9fb83614394cd3b39e42b1d9f84a3c... sha256 0.26399648 1750719480 64 2320129f2545ff8606d3db1d706d86... sha224 0.28097475 1644929957 56 133b5da8d8b387f2bcfd69b0c73ed8... md4 0.34185237 1351998195 32 dea7585ea9fa4520687bab1dc671858e blake2b 0.53724666 860282275 128 e3653f33858a83b386c2fe865280a1... md5 0.58128106 795112407 32 299440e1968cf8f8abc288bac8c0a4fa sha512_224 0.64589952 715566066 56 413d48b782f114870ef80815540678... sha384 0.64645893 714946859 96 b1c1cd96cef79c15f2171b8aa81304... sha512 0.65424513 706438241 128 e7d0cec3fe8b73d1534a7bdb484176... sha512_256 0.68371638 675987586 64 3f58faba70cea4d6ea8a8371e71bbb... md5-sha1 0.80361958 575127576 72 299440e1968cf8f8abc288bac8c0a4... shake_128 0.84424524 547452041 64 c62a813897b81f67822fc07115deae... blake2s 0.85661793 539544839 64 cb8bd19c6ca446bbf7a8abbec61dc5... sha3_224 0.95759645 482649850 56 6f96d117c7fcbcd802b222854db644... shake_256 1.0152032 455262322 64 2d9f9dafe0ddf792c6407910946845... sha3_256 1.015744455019929 64 cc5d55fe0ac31f6e335da1bc6abaf3... sha3_384 1.3235858 349190644 96 13206910ff231fe51a38fe637ded30... sm3 1.4478934 319211203 64 021cd913540d95b13a03342b54f80d... ripemd160 1.4737549 313609670 40 1a956000b88267ec8fc23327d22548... sha3_512 1.9131832 241578418 128 e84b9f499b013956f6f36c93234ca3... "time" is wall time in seconds. So, yes, BLAKE3 was the fastest hash algorithm available on my machine--2.4GB/sec! (I'm a little surprised by that result actually. My CPU is pretty modern, so I assume it has the SHA1 extensions. And I further assume we're using OpenSSL, and OpenSSL will use those extensions when available. If BLAKE3 is *still* faster that OpenSSL, well! hats off to the BLAKE3 team. And that just goes to show you how useful SIMD extensions are!) -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Jack O'Connor: > Was any of the experimental C extension code [...] useful to you? > I was wondering if it could be possible to copy blake3module.c from > there verbatim. I concede I didn't even look at it. The glue code to mate the library with the CPython runtime wasn't the hard part. And, Python has its own way of working (Argument Clinic etc). I did what seemed easiest, which was to start with CPython's BLAKE2 support and hack it up. Given that I had it working in maybe half a day, this seems reasonable. I should take a peek at your experimental build though, just to confirm I got the interfaces right. > The setup.py build there also has working Windows and NEON support. The CPython build process doesn't use setup.py to build extensions on Windows. I haven't looked recently, but back in the day they were built like any other DLL, using its own "project file". This will require someone to use the MSVS GUI to create a new project, add files, etc etc. I assume they can just use the .asm files for the SIMD extensions so hopefully this will be straightforward. As for NEON support, the technique I used theoretically should Just Work. I look forward to hearing back from someone building on ARM. (I have a cross-compiler setup for building for MiSTer, which is an ARM platform with NEON support. But the cross-compiler is a royal PITA to use... it's a Docker image, and requires a bunch of manual configuration, and I haven't touched any of that in more than a year.) You seem to have done at least a partial code review of my PR, given your comments to follow. I appreciate it! I tried to add you as a "reviewer" but GitHub wouldn't permit it--I assume this is some sort of permissions problem. Still, it'd be nice if you were able to do future code reviews using the GitHub review tool; are you permitted to use that for my PR? > - My derive_key_context parameter requires a string and refuses to > accept bytes. This is consistent with our Rust and C APIs (though the C > API does include a _raw version specifically for bindings, which we're > using here). I was considering going the other way with it actually, requiring bytes. Note that Python has first-class support for hard-coded bytes strings: b = blake3.blake3(derive_key_context=b"My Funny Valentine (1984)") The C interface takes "char *", not a "wchar_t *", and this seemed like the most direct and relatable way to reflect that. But I'm not militant about this, and I'm willing to change the interface to require an actual string (as in Unicode). I note that your C API already dictates that Unicode be encoded as UTF-8, so we can do that, and if the encoding fails the user can deal with it. > - I've included an `AUTO` constant that provides a special value (-1) > for the `max_threads` argument, and I explicitly don't support > `max_threads=0`. I can do that too; again, I prefer the 0 there, but I'm not militant about it. However, it would make sense to me if you had that constant somewhere in the BLAKE3 C .h files, which AFAICT you currently don't. > - I enforce that the `data` arguments are positional-only and that the > other keyword arguments are keyword-only. I think this is consistent > with the rest of hashlib. I suspect hashlib is mostly like that, due to being chock full of legacy code. But I don't see why that's necessary. I think permitting "data" to be a named argument is fine. So unless you have a strong conviction about it--which I bet you don't--I'll leave it as positional-or-keyword. There are rare circumstances where positional-only arguments are useful; this isn't one of them. > - I include a `.reset()` method. I don't mind adding that. > - Unrelated to tests: I haven't made any attempt to zero memory in my > `dealloc` function. But if that's what other hashlib functions do, > then I'm certainly in favor of doing it here too. I inherited that from the BLAKE2 code I carved up to make the BLAKE3 version. And yeah, it made sense to me, so I kept it. Christian Heimes: > GH-31686 is a massive patch set. I'm feeling uncomfortable adding > such much new code for a new hashing algorithm. Did you ask the > Steering Council for approval? I didn't. Like most hashing algorithms, BLAKE3 doesn't allocate memory and doesn't perform any I/O. All it does is meditate on the data you pass in, and write to various pre-allocated fixed-size buffers. As large codebases go this seems pretty harmless, almost inert. The Modules/_blake3/impl directory is about 32kloc. I note that the Modules/_blake2/impl directory you checked in in 2016 is about 21kloc, and you didn't require Steering Council (or BDFL) approval fo
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Right, and I did say "(or BDFL)". Apparently you didn't bother to consult with the BDFL in advance, or at least not in the usual public venues--I haven't found a record of such a conversation on the bpo issue, nor in python-dev. BTW you simultaneously proposed adding SHA3/SHAKE. The total kloc for all this work was over 26k; you didn't mention any discomfort with the size of these patches at the time in public correspondance. In fact, quite the opposite. On 2016/05/28 you said: > I also don't get your obsession with lines of code. The gzip and expat > are far bigger than the KeccakCodePackage. https://mail.python.org/archives/list/python-...@python.org/message/3YHVN2I74UQC36AVY5BGRJJUE4PMU6GX/ -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46930] Iterating over cls.__dict__ in classmethod causes RuntimeError when printing __annotations__
Larry Hastings added the comment: When accessing __annotations__ *in a class without annotations*, and *for the first time*. And your workaround seems reasonable. -- ___ Python tracker <https://bugs.python.org/issue46930> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1007] Fix dumbdbm, which fixes test_shelve (for me); instrument other tests so we catch this sooner (and more directly)
Larry Hastings added the comment: Whoops, copy & paste error on the title, there. -- title: ix dumbdbm, which fixes test_shelve (for me); instrument other tests so we catch this sooner (and more directly) -> Fix dumbdbm, which fixes test_shelve (for me); instrument other tests so we catch this sooner (and more directly) __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1007> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1007] [py3k] Fix dumbdbm, which fixes test_shelve (for me); instrument other tests so we catch this sooner (and more directly)
Larry Hastings added the comment: Yeah, Neil Norwitz added the one-line dumbdbm fix in r57358. Thanks! __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1007> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] Expose nanosecond precision from system calls
Larry Hastings added the comment: Mark Dickinson wrote: > I think this could work. "could"? Oh ye of little faith! Attached is a patch against a nice fresh trunk (2b47f0146639) that adds Decimal attributes "ctime", "mtime", and "atime" to the object returned by os.stat(). The functions that consume mtime and atime values (os.utime, os.futimes, os.lutimes, os.futimesat) now also accept Decimal objects for those values. My smoke-test using os.stat and os.utime works fine, and CPython passes the unit test suite with only the expected skippage. However, the patch isn't ready to be checked in; I didn't update the documentation, for one. I'd be happy to post the patch on Rietveld--just ask. The implementation was complicated by the fact that Decimal is pure Python ;) I had to do some "build the ship in a bottle" work. Also, it's possible for os.stat to be called before the Python interpreter is really ready. So internally it has to gracefully handle an import error on "decimal". Along the way I noticed some minor resource leaks, both in os.utime: * for Windows, if passed a Unicode filename it would leak "obwpath". * for non-Windows, if the call to the C function utime() failed it would leak "opath". I fixed these, along with a spelling error. I also cleared up some sillyness. When built on non-Windows, extract_time etc. used nine places of precision; on Windows it only used six. Windows only calls extract_time for os.utime--the other functions that use extract_time aren't available on Windows. And in the Windows implementation of os.utime, it multiplied the time it got from extract_time by a thousand! This might even be throwing away some precision--not sure. Definitely it was cruft. However, modifying this meant changing the Windows code, which I can't test! So I'm not 100% certain it's right. Finally the bad news: this patch contributes a major performance regression on os.stat. On my laptop, timeit says os.stat takes 10x longer when building the three Decimal fields. My immediate thought: lazy-create them. This would mean some major brain surgery; I'd have to make a subtype of PyStructSequence and override... something (tp_getattr? tp_getattro?). (Though this might also neatly ameliorate the early-startup import problem above.) I'd also have to hide the exact integers in the object somewhere--but since I'd be subclassing anyway this'd be no big deal. My second thought: maybe one of the other Decimal constructors is faster? I'm currently using the "parse a string" form. My guess is, one of the other forms might be faster but not by an order of magnitude. Martin van Löwis wrote: > For example, gcc doesn't support __float128 in 32-bit > mode on x86. That was only true for GCC 4.3. GCC 4.4 and newer support __float128 in 32- and 64-bit modes on Intel. That release has been out for more than two years. But consider the matter dropped ;-) -- Added file: http://bugs.python.org/file23246/larry.decimal.utime.patch.1.txt ___ Python tracker <http://bugs.python.org/issue11457> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13053] Add Capsule migration documentation to "cporting"
New submission from Larry Hastings : After the great Capsule flame wars of 2011, it became clear that we need documentation on migrating from CObject to Capsules, as CObject is gone as of 3.2. Nick made me promise to write the documentation, and Raymond steered me in the direction of "cporting.rst" (Porting Extension Modules To Python 3.0). I already have a patch in reasonable shape. However, I understand we're doing "forward-porting" (what Monotone calls "daggy fixes"). I think this would be valuable information for 2.7 users. It includes no code changes (though it does include some sample code). Would checking it in to the 2.7 head be appropriate? Once we decide where the checkin should go, I'll produce a patch against that head that will hopefully garner your approval. -- assignee: larry components: Documentation messages: 144567 nosy: larry, ncoghlan, rhettinger priority: normal severity: normal stage: needs patch status: open title: Add Capsule migration documentation to "cporting" type: feature request versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue13053> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13086] Update howto/cporting.rst so it talks about 3.x instead of 3.0
New submission from Larry Hastings : The title of howto/cporting.rst is "Porting Extension Modules To 3.0". It then talks about 3.0 in a whole bunch of places. Considering that we're working on 3.3, and considering that 3.0 is end-of-lifed (not even meriting a branch in hg), wouldn't it be better for the document to talk about "3.x"? It already talks about "2.x" in several places, so it's not like this would confuse the reader. Alternatively, we could remove the ".0" (and maybe the ".x"s) so the document talks about porting from "Python 2" to "Python 3". I'd be happy to make the patch / check in the change. -- assignee: larry components: Documentation messages: 144721 nosy: larry priority: low severity: normal status: open title: Update howto/cporting.rst so it talks about 3.x instead of 3.0 type: feature request versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4 ___ Python tracker <http://bugs.python.org/issue13086> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13053] Add Capsule migration documentation to "cporting"
Larry Hastings added the comment: Attached is a patch against trunk branch "2.7" (rev dec00ae64ca8) adding documentation on how to migrate CObjects to Capsules. Delta the inevitable formatting bikeshedding, this should be ready to go. I've smoke-tested the "capsulethunk.h" locally and it works fine. When accepted, I'll check this in to the 2.7 branch, then merge into the 3.1, 3.2, and trunk branches. -- ___ Python tracker <http://bugs.python.org/issue13053> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13053] Add Capsule migration documentation to "cporting"
Larry Hastings added the comment: Whoops, forgot to attach. *Here's* the patch. -- keywords: +patch Added file: http://bugs.python.org/file23282/larry.cporting.capsules.r1.diff ___ Python tracker <http://bugs.python.org/issue13053> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13086] Update howto/cporting.rst so it talks about 3.x instead of 3.0
Larry Hastings added the comment: Why shouldn't I check this in to the 2.7 / 3.1 branches? -- ___ Python tracker <http://bugs.python.org/issue13086> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13053] Add Capsule migration documentation to "cporting"
Larry Hastings added the comment: Attached is r2 of the patch, incorporating Nick's suggestions. Base revision hasn't changed. -- Added file: http://bugs.python.org/file23301/larry.cporting.capsules.r2.diff ___ Python tracker <http://bugs.python.org/issue13053> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13053] Add Capsule migration documentation to "cporting"
Larry Hastings added the comment: In case you're curious, here's how I tested "capsulethunk.h". I added the file to Python 2.7 (hg head), 3.0.0 (tarball), and 3.1.0 (tarball). For 2.7 ad 3.0.0 I quickly hacked four files to use the Capsule API instead of CObjects: * Python/compile.c * Python/getargs.c * Modules/_ctypes/callproc.c * Modules/_ctypes/cfield.c (For 3.1 I simply included the file in those four files, as they already use the Capsule API.) I then built and ran regrtest.py. While developing capsulethunk.h, I had a more thorough test suite; sadly that's on a laptop that is shut off, and I'm on vacation across the Atlantic and can't get at it. But everything was working fine last I checked ;-) -- ___ Python tracker <http://bugs.python.org/issue13053> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13053] Add Capsule migration documentation to "cporting"
Larry Hastings added the comment: New patch based on comments from Ezio Melotti--thanks, Ezio! * capsulethunk.h is now its own file in Doc/includes. * Various minor formatting touchups. * I added some rationale behind the thunked PyCapsule_SetName behavior. -- Added file: http://bugs.python.org/file23315/larry.cporting.capsules.r3.diff ___ Python tracker <http://bugs.python.org/issue13053> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13105] Please elaborate on how 2.x and 3.x are different heads
New submission from Larry Hastings : It wasn't clear to me after reading the "Forward Porting" section exactly what was going on. Nick Coghlan spelled it out for me in a private email, and suggested that maybe this stuff should be in the devguide proper. Here's some specific stuff that I didn't understand until Nick explained it to me with simple words: * 2.x and 3.x have separate heads in the same repository * Since they're totally divorced, the order you check in to 2.x and 3.x does not matter * DO NOT MERGE between 2.x and 3.x * Branches that are in security-fix-only mode (e.g. 3.1) don't get bugfixes or documentation fixes (surely mentioned elsewhere, but I personally would have been helped with a reminder) I suggest it'd be clearer to start with discussing "2.x and 3.x are separate heads", and *then* move on to "But when merging changes solely inside a major version" and talk about forward-porting. Would you be interested in a patch? -- components: Devguide messages: 144930 nosy: larry priority: normal severity: normal status: open title: Please elaborate on how 2.x and 3.x are different heads type: feature request ___ Python tracker <http://bugs.python.org/issue13105> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13105] Please elaborate on how 2.x and 3.x are different heads
Larry Hastings added the comment: What follows is the original email from Nick. -- We maintain two independent heads in hg: 2.7 and default 3.2 is open for general bugfixes 2.5 (IIRC), 2.6 and 3.1 are open for security fixes Security fixes (if applicable to both heads) go: 2.5 -> 2.6 -> 2.7 3.1 -> 3.2 -> default General bug fixes (if applicable to both heads) go: 2.7 3.2 -> default New features are added to default only The relative ordering of 2.x and 3.x changes doesn't really matter - the important thing is not to merge them in *either* direction. I think you can theoretically do cherry-picking with Hg, but most people seem to just do independent commits to the two streams. If the devguide doesn't align with the above, then a tracker issue pointing that out would be handy :) -- ___ Python tracker <http://bugs.python.org/issue13105> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] os.stat(): add new fields to get timestamps as Decimal objects with nanosecond resolution
Larry Hastings added the comment: Can I get some thoughts / votes on whether to a) check in with the current performance regression, or b) do the work to make it lazy-created? -- ___ Python tracker <http://bugs.python.org/issue11457> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13086] Update howto/cporting.rst so it talks about Python 3 instead of 3.0
Larry Hastings added the comment: Attached is my first revision patch. I did some other editing for clarity / 80 columns, though I can back those out from this patch (and put in another) if that's best. Patch is against the 2.7 branch; once this goes in I'll port all my recent cporting.rst changes to 3.2 and trunk. -- keywords: +patch Added file: http://bugs.python.org/file23357/larry.cporting.to.python.3.r1.diff ___ Python tracker <http://bugs.python.org/issue13086> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12326] Linux 3: code should avoid using sys.platform == 'linux2'
Larry Hastings added the comment: So what about doing the same for FreeBSD, SunOS, and Windows? The conversation about this point sort of trailed off. But, dammit, a foolish consistency is the hobgoblin of my small mind. If we're changing "linux2" / "linux3" to just "linux", we should be consistent and do it for everybody. I propose sys.platform under 3.3 should contain things like "linux", "freebsd", "openbsd", "darwin", and "windows". I further propose we add a "sys.platform_major_version" which would contain the OS major version number (as an integer!). That'd be 3 for Linux 3.0, 8 for FreeBSD 8, 32 for Win32, and so on. That'd let users easily reconstitute the old value of "sys.platform". (Except on Windows. But I am strangely averse to setting sys.platform to "win" on Windows.) Of course, we should determine this value at runtime rather than build time on platforms where the number could change at runtime. (A counter-example: it need not be late-binding for "windows" 32 vs 64.) With that addition, we can now address plat-freebsd.: * Move the all existing plat-freebsd/IN.py to plat-freebsd/IN.py * Write a new plat-freebsd/IN.py that uses sys.platform_version to read in the correct IN.py. * Alter plat-freebsd/regen to generate IN.py I suspect plat-freebsd should have used the run-time OS major version all along, so this would be an improvement! And since FreeBSD is the only OS with more than one plat-* entry, the plat-* problem is solved. I'm happy to open a new issue discussing this if that's the right thing to do. -- nosy: +larry ___ Python tracker <http://bugs.python.org/issue12326> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12898] add opendir() for POSIX platforms
New submission from Larry Hastings : With the recent spate of POSIX *at() functions added to os, we now have a bunch of places in the API that take directory fds. But afaict there's no way to get a directory fd in Python! The only calls to opendir() in the tree are internal, in os.listdir() and in the import machinery. (Though in practice most people will use AT_FDCWD anyway.) I propose adding a new function, os.opendir(), the implementation to be much the same as (aka a hacked-up copy and paste of) os.unlink() in Modules/posixmodule.c. I'd be happy to contribute the patch. -- components: Extension Modules messages: 143522 nosy: larry priority: normal severity: normal status: open title: add opendir() for POSIX platforms type: feature request versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue12898> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12899] Change os.utimensat() and os.futimens() to use float for atime & mtime
New submission from Larry Hastings : The new functions os.futimens() and os.utimensat() update the timestamps of a file with nanosecond precision. However, both functions take atime and mtime as a tuple: (seconds since epoch, nanoseconds). Contrast this with os.utime(), which produces atime and mtime as a floating point number of seconds since epoch. Why the mismatch between the APIs? It simply forces the user to do the conversion themselves. You can see this in the regression tests for these two functions--there's a lot of multiplying by 1e9 going on. The only justification I can contrive is that the conversion of (secs+(1e-9*nsecs)) can be inaccurate; a double would need roughly one more byte of mantissa to be able to accurately preserve all possible nanosecond values. But that ship has sailed--os.utime() produces inaccurate results, and afaik there's no other way to do it in the Python library. os.futimens() and os.utimensat() should take atime and mtime in the floating-point format as produced by os.utime(). I'm happy to contribute the patch. -- components: Extension Modules messages: 143529 nosy: larry priority: normal severity: normal status: open title: Change os.utimensat() and os.futimens() to use float for atime & mtime type: behavior versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue12899> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12898] add opendir() for POSIX platforms
Larry Hastings added the comment: Well, there's no os.fdopendir(); I think you're referring to fdlistdir(), which uses the C function fdopendir() internally. The DIR structure is not exposed to the Python caller at any point. I did miss the whole opendir-returns-a-DIR-not-a-fd thing, hopefully that's my dumb thing of the day. This suggests opendir() would have to call dirfd() on your behalf. Would it be safe to call close() on the fd returned from dirfd(opendir())? If not, I guess we'd need a special closedir() function, which would use the C functions fdopendir() then closedir() internally. ... or we could forget the whole thing, I guess. -- ___ Python tracker <http://bugs.python.org/issue12898> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12898] add opendir() for POSIX platforms
Larry Hastings added the comment: fdlistdir() is largely irrelevant to the discussion. I was proposing adding a function to open directory fds, because there isn't one; fdlistdir(), like many other POSIX functions available in Python, consumes directory fds. -- ___ Python tracker <http://bugs.python.org/issue12898> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12898] add opendir() for POSIX platforms
Larry Hastings added the comment: > Is there anything you want to do on a "directory fd" > except listing its contents? In the first message in this bug, I wrote: "With the recent spate of POSIX *at() functions added to os, we now have a bunch of places in the API that take directory fds." To be specific: faccessat, fchmodat, fchownat, fstatat, futimesat, linkat, mkdirat, mknodat, openat, readlinkat, renameat, symlinkat, unlinkat, utimensat, mkfifoat. At the time I created this ticket I didn't realize you could just call open() on a directory. It seem that works fine and is supported everywhere that these *at functions exist, so perhaps it's best if we just close this ticket. -- ___ Python tracker <http://bugs.python.org/issue12898> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12899] Change os.utimensat() and os.futimens() to use float for atime & mtime
Larry Hastings added the comment: I'm withdrawing this, as I've found a better way to approach the problem. -- resolution: -> rejected status: open -> closed ___ Python tracker <http://bugs.python.org/issue12899> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12904] Change os.utime &c functions to use nanosecond precision where possible
New submission from Larry Hastings : Since Linux 2.5 stat() has supported reading nanosecond resolution (1b/sec) for atime and mtime. However, the utime() family of functions could only write atime and mtime with microsecond resolution (1m/sec) until relatively recently. On POSIX platforms Python reads atime and mtime at nanosecond resolution but only writes them at microsecond resolution. This impedance mismatch can cause undesirable behavior. Consider the following code: import os import shutil import sys def mtime(path): return os.stat(path).st_mtime src = sys.argv[0] dest = src + ".new" shutil.copy2(src, dest) assert mtime(src) == mtime(dest) When run under Python on any modern Linux system, the assert fails. (I think any Python since 2.5 will do; I tested with 2.7.1 and a fresh 3.3 trunk.) The accompanying patch modifies Modules/posixmodule.c so all functions that write atime and mtime use nanosecond resolution when possible. With this patch applied, all the regression tests pass (except the ones for extension modules I didn't compile), and the above code runs to completion. Happy to hoist the patch up on Rietveld if there's interest. p.s. I have the commit bit set, so I'd like to be the one to check this in if we get that far. -- assignee: larry components: Extension Modules files: larry.utimensat.patch.r1 messages: 143563 nosy: larry priority: normal severity: normal status: open title: Change os.utime &c functions to use nanosecond precision where possible type: feature request versions: Python 3.3 Added file: http://bugs.python.org/file23104/larry.utimensat.patch.r1 ___ Python tracker <http://bugs.python.org/issue12904> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12904] Change os.utime &c functions to use nanosecond precision where possible
Larry Hastings added the comment: A small aside: An IEEE 754 double is insufficient to store a modern timestamp with nanosecond accuracy. We're currently at just over 1.3 billion seconds since the epoch. This eats 28 bits of the mantissa. (The value is actually 31 bits to the left of the decimal point, but you get the leading 1 and the subsequent two 0s for free.) Precisely storing a billion values to the right of the decimal point would require another 30 bits. The mantissa of a double is sadly only 52 bits. So we're shy by 6 bits, give or take. So this patch doesn't make Python 100% accurate with respect to atime/mtime. What it *does* do is makes Python's loss of accuracy consistent. I claim that's a definite improvement. -- ___ Python tracker <http://bugs.python.org/issue12904> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12904] Change os.utime &c functions to use nanosecond precision where possible
Larry Hastings added the comment: > It's a duplicate of issue #11457. No. #11457 is a how-many-angels-on-the-head-of-a-pin discussion proposing new representations of ctime/mtime/atime to preserve nanosecond accuracy. This patch changes the behavior of os.utime, os.futimes, os.lutimes, and os.futimesat so they conserve all of the existing representation's accuracy, making them consistent with os.stat. > (Python >=3.3 has os.utimensat() and os.futimens().) How is that relevant? I mean, sure, I leveraged some of the work from that support in my patch--it's heavily reliant on the preprocessor macros HAVE_UTIMENSAT and HAVE_FUTIMENS now generated by configure. (Thanks, Ross!) But the presence of those functions does not mitigate the loss of accuracy inside os.utime &c. -- ___ Python tracker <http://bugs.python.org/issue12904> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] Expose nanosecond precision from system calls
Larry Hastings added the comment: This is probably a terrible idea, but: what about using a subclass of float that internally preserves the original sec / usec values? Call it a utime_float for now. os.stat would produce them, and os.utime would look for them--and if it found one it'd pull out the precise numbers. Type promotion as a result of binary operators: utime_float OP int = utime_float utime_float OP float = degrades to float I suspect code rarely does math on atime/utime/ctime and then writes out the result. Most of the time they simply propogate the utime values around, comparing them to each other, or setting them unchanged. For those rare occasions when someone wants to change the fractional part of a utime_float, we could provide a function utime_fractional(int) -> utime_float. Advantages: * Nobody has to change any Python code. In almost all circumstances they get complete accuracy for free. Disadvantages: * Complicated. * Is a yucky hack. * Is a terrible idea. (Now I'm sure of it!) -- nosy: +larry ___ Python tracker <http://bugs.python.org/issue11457> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] Expose nanosecond precision from system calls
Larry Hastings added the comment: Here's a better idea: we add a new IEEE 754-2008 quad-precision float type. The IEEE 754-2008 quad precision float has 1 sign bit, 15 bits of exponent, and 112 bits of mantissa, so it should have enough precision to last utime until humanity transforms itself into a single angelic being of pure light and energy. GCC has had __float128 since 4.3, Clang has __float128 now too, Intel's compiler has _Quad. It looks like Visual C++ doesn't support it yet--it does support a funny 80-bit float but I don't think Python wants to go there. I realize a new float type would be a major undertaking, but it seems to me that that's really the right way to do it. Nobody would have to change their code, and it'd behave like the existing float. It'd be just like 2.x, with "int" and "long"! -- ___ Python tracker <http://bugs.python.org/issue11457> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] Expose nanosecond precision from system calls
Larry Hastings added the comment: I think a pair of integers is a poor API. It ties the value of the fractional part to nanoseconds. What happens when a future filesystem implements picosecond resolution? And then later goes to femtoseconds? Or some platform chooses another divisor (2**32)? This should all be abstracted away by the API, as the current API does. Otherwise you start sprinkling magic values in your code (ie 1e9). Suggesting that other representations (float128, Decimal) can be built on top of the (int,int) interface is irrelevant; obviously, any representation can be built on top of any other. I think Decimal is reasonable, except that it breaks existing code. One cannot perform computation between a Decimal and a float, so almost any existing manipulations of atime/utime would start throw exceptions. I suggest that a float128 type would solve the problem on all counts--nobody would have to change their code, and it would handle nanosecond (or I think even zeptosecond!) resolution. And when we run out of resolution, we can switch to float256. (Or an arbitrary-precision float if Python has one by then.) os.stat added support for float atime/mtime in 2.3, specifically in October 2002: http://hg.python.org/cpython/rev/0bbea4dcd9be This predates both the inclusion of Decimal in Python (2.4) and nanosecond resolution in the utime API (2008). I could find no discussion of the change, so I don't know what other representations were considered. It's hard to say what the author of the code might have done if Decimal had existed back then, or if he foresaw nanosecond resolution. However, that author is already present in this thread ;-) Martin? -- ___ Python tracker <http://bugs.python.org/issue11457> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] Expose nanosecond precision from system calls
Larry Hastings added the comment: > To support higher resolution > will need to supply 3 fields in st_xtimesuperspec: tv_sec and tv_nsec > packed in st_xtimespec (say tv_timespec) and new tv_psec field. > st_xtime will now be st_xtimesuperspec.tv_timespec.tv_sec and > st_xtimespec will be a new macro expanding to > st_xtimesuperspec.tv_timespec. How is this superior to using either Decimal or float128? -- ___ Python tracker <http://bugs.python.org/issue11457> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] Expose nanosecond precision from system calls
Larry Hastings added the comment: I've drawn an ASCII table summarizing the proposals so far. If I've made any factual errors I trust you'll let me know. = means os.stat().st_mtime is changed to that type. + means os.stat() grows a new field using that type, and the current behavior of st_mtime is unchanged. ___ [ UPSIDES ]=(int,int) +(int,int) =Decimal +Decimal =float128 [ yes is good! ] all existing code gets no nono no yes more accurate for free some existing code gets no noyes no yes more accurate for free guaranteed future-proof against no noyes yes no* new representations very very future-proof against no noyes yes yes* new representations * float128 could handle representations finer than yoctosecond resolution, 10**-24, but not another 10**-3. fwiw, yocto is currently the smallest defined prefix. ___ [ DOWNSIDES ] =(int,int) +(int,int) =Decimal +Decimal =float128 [ yes is bad! ] breaks existing code yes noyes no no requires new code in order to take advantage yes* yes yes* yes no of added precision requires implementing a no nono no yes complicated new type * Since this option breaks existing code, obviously people will have to write new code in order to cope. ___ My take on the above: if we're willing to put people through the pain of changing their code to use the new accuracy, then Decimal is the obvious winner. I see no advantage to any of the pair-of-floats proposals over Decimal. If we want all existing code to continue working and get more accurate automatically, the only viable option is float128 (or a multiple-precision float). -- ___ Python tracker <http://bugs.python.org/issue11457> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] Expose nanosecond precision from system calls
Larry Hastings added the comment: s/pair-of-floats/pair-of-ints/ Also, to be clear: yocto is the smallest defined SI prefix. And what I meant when I referred to 10**-3 was, float128 could handle 10**-24 but not 10**-27. According to my back-of-the-envelope calculations, float128 could accurately represent timestamps with yoctosecond resolution for another 650 years to come. -- ___ Python tracker <http://bugs.python.org/issue11457> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] Expose nanosecond precision from system calls
Larry Hastings added the comment: Mark Dickinson: > > I realize a new float type would be a major undertaking > That's an understatement and a half. The only way this could ever > be viable is if float128 support becomes widespread enough that we > don't have to write our own algorithms for basic float128 operations As I mentioned earlier in this thread, GCC has supported __float128 since 4.3, Clang added support within the last year, and Intel has a _Quad type. All are purported to be IEEE 754-2008 quad-precision floats. Glibc added "quadmath.h" recently (maybe in 4.6), which defines sinq() / tanq() / etc. Is that not sufficient? The Windows compilers don't seem to support a float128 yet. But Windows only supports 100ns resolution for mtime/atime anyway. The bad news: according to my back-of-the-envelope calcuations, doubles will stop accurately representing 100ns timestamps in less than a year; they'll lose another bit of precision somewhere around 2019. Alexander Belopolsky > > How is this superior to using either Decimal or float128? > It is explicit about the units of time used. Why is that a feature? I'd rather that was abstracted away for me, like the os module currently does. > And familiarity with C API is expected from users of os module, IMO. As you say, that's your opinion. But I've never heard of that as an official policy. Therefore it is irrelevant as a design constraint for the API. > > I've drawn an ASCII table summarizing the proposals so far. > You did not mention "closely matches C API" as an upside. By "C API", you mean "the POSIX-2008 compatible C API". This would not match * POSIX platforms that don't meet the 2008 spec * Windows * Java * CLR all of which are supported platforms for Python. The documentation for the os module states: "This module provides a portable way of using operating system dependent functionality. [...] The design of all built-in operating system dependent modules of Python is such that as long as the same functionality is available, it uses the same interface" Since "the most recent modification time / access time of a file" is available on all platforms Python supports, it follows that Python should use the same interface to represent it on all those platforms. Tying the representation to that of one particular platform is therefore poor API design, particularly when there are representations that abstract away such details within easy reach. So I don't consider it a compelling upside--in fact I consider it a disadvantage. -- ___ Python tracker <http://bugs.python.org/issue11457> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11457] Expose nanosecond precision from system calls
Larry Hastings added the comment: Victor STINNER: > Python is compiled using Visual Studio 2008 on Windows. Portability > does matter on Python. If a type is not available on *all* platforms > (including some old platforms, e.g. FreeBSD 6 or Windows XP), we > cannot use it by default. The quad-precision float would be highly portable, but not 100% guaranteed. The idea is that st_mtime would be a float128 on a recent Linux, but still a double on Windows. Python's "duck typing", combined with a judicious implementation of float128, would permit code that performed simple math on a timestamp to run unchanged. That should be sufficient for almost all existing code that deals with these timestamps. But there are obvious, plausible scenarios where this would break. For example: pickling a float128 mtime on one platform and attempting to unpickle it on Windows. I can imagine legitimate reasons why one would want to ship ctime/atime/mtime across platforms. If that's an unacceptable level of portability, then for the proposal to remain viable we'd have to implement our own portable quad-precision floating point type. That's a staggering amount of work, and I doubt it would happen. But short of some quad-precision type, there's no way to preserve existing code and have it get more precise automatically. If float128 isn't viable then the best remaining option is Decimal. But changing st_mtime to Decimal would be an even more violent change than changing it to float was. I propose adding the Decimal fields "ctime", "atime", and "mtime" to the named tuple returned by os.stat(). -- ___ Python tracker <http://bugs.python.org/issue11457> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11670] configparser read_file now iterates over f, docs still say it calls readline
New submission from Larry Hastings : The documentation for configparser.RawConfigParser.read_file() states: "Read and parse configuration data from the file or file-like object in f (only the readline() method is used)." This was true in Python 3.1 and before. However in 3.2 the implementation of read_file changed. It no longer calls readline(). Instead, it iterates over the file object. Whoever made this change neglected to * fix the docs, or apparently * tell anybody. I've got the commit bit, so I'd be happy to fix this. I'd like to add it to the What's New In Python 3.2 as well; there's a section about configparser changes that would be perfect. Is it permissible to change the What's New document ex post facto like this? (Adding rhettinger so he can answer this.) -- assignee: docs@python components: Documentation messages: 132075 nosy: docs@python, larry, rhettinger priority: low severity: normal stage: needs patch status: open title: configparser read_file now iterates over f, docs still say it calls readline type: behavior versions: Python 3.2 ___ Python tracker <http://bugs.python.org/issue11670> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11670] configparser read_file now iterates over f, docs still say it calls readline
Larry Hastings added the comment: By the same token, readfp is now deprecate in favor of the new spelling read_file. That change *is* mentioned in configparser. If I'm touching What's New In Python 3.2, mind if I add a mention of that too? -- ___ Python tracker <http://bugs.python.org/issue11670> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7924] test_capi crashes with misleading datetime ImportError when _curses is not built
Larry Hastings added the comment: "To reproduce: add a typo on "_curses" on line 1666." Of what file? -- ___ Python tracker <http://bugs.python.org/issue7924> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7924] test_capi crashes with misleading datetime ImportError when _curses is not built
Larry Hastings added the comment: Patch to make testcapsule clear import errors when testing. Sorry about the whitespace changes, but I outdented a bunch of code--I think it's clearer this way. -- Added file: http://bugs.python.org/file16223/lch.patch.trunk.py3k.r78149 ___ Python tracker <http://bugs.python.org/issue7924> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5939] Ensure that PyCapsule_GetPointer calls in ctypes handle errors appropriately
Larry Hastings added the comment: I finally reviewed this, and I think it does need additional armor against attack. I think a user could insert a different object into the thread local dict with the hard-coded name and get CPython to crash. This patch fixes the vulnerability: http://codereview.appspot.com/217092/show If this goes in, I'll add it to the backport for 2.7. -- ___ Python tracker <http://bugs.python.org/issue5939> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7992] Backport capsule object
Larry Hastings added the comment: This is also (naturally) being discussed on python-dev. I've posted a long message there about it; you can find that message here: http://mail.python.org/pipermail/python-dev/2010-March/098904.html I don't know whether it's best to conduct such a discussion here, in Python-Dev, or both. I assumed Python-Dev is superior because more people will see it. If it makes sense to move the discussion here we can move it. -- ___ Python tracker <http://bugs.python.org/issue7992> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7992] Backport capsule object
Larry Hastings added the comment: Florent: that's because the Windows build projects don't build capsule.c. I don't have an appropriate Windows development environment; I'll try to fix them by hand tonight, but if that doesn't work we'll have to call for help from some Windows core developer. -- ___ Python tracker <http://bugs.python.org/issue7992> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Larry Hastings : -- nosy: +larry ___ Python tracker <http://bugs.python.org/issue7946> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8235] Support FreeBSD's SO_SETFIB in socketmodule.c
Larry Hastings added the comment: Kyle asked me to look at this; I think it looks ready for checkin, so unless someone else jumps in with something to say I'll check it in in a day or two. It's not going in to 2.6 however, just 2.7 trunk and probably 3.2. -- nosy: +larry versions: -Python 2.6 ___ Python tracker <http://bugs.python.org/issue8235> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8235] Support FreeBSD's SO_SETFIB in socketmodule.c
Larry Hastings added the comment: Committed in Python 2.7 as revision r79592 Committed in Python 3.2 as revision r79594 Thanks for the patch! -- assignee: -> larry resolution: -> accepted status: open -> closed versions: +Python 3.2 ___ Python tracker <http://bugs.python.org/issue8235> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7992] Backport capsule object
Larry Hastings added the comment: Yes. I was told it was inappropriate for me to change the bsddb module, as it's independently maintained by Jesus Cea. I sent Mr. Cea a patch last night; I hope he chooses to merge it soon. Since CObjects are marked Pending Deprecation, and bsddb is using a CObject, this is correct behavior. Were you mentioning it simply to bring it my attention, or am I missing something important? -- ___ Python tracker <http://bugs.python.org/issue7992> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7992] Backport capsule object
Larry Hastings added the comment: Oh, and, last night I checked in r79590. This is the checkin that ameliorates the backwards compatibility issues. Specifically, I changed four things: * PyCObject_AsVoidPtr() can now open capsules. This addresses most of the remaining backwards-compatibility concerns about the conversion of Python 2.7 from CObjects to capsules. * CObjects were marked Pending Deprecation. * Documentation about this pending deprecation was added to cobject.h. * The capsule source files were added to the legacy PC build processes. (Someone had already added them to the current PC build process, a day or two after I committed the previous checkin.) Sorry for forgetting to update the issue last night, but I was in a bit of a hurry to get this done before Benjamin cut 2.7b1. -- ___ Python tracker <http://bugs.python.org/issue7992> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7992] Backport capsule object
Larry Hastings added the comment: The patch is a bit more involved than that. Capsules didn't exist in 3.0, and the bsddb module published a CObject in 3.1. So bsddb must continue to use CObject for those two releases. Therefore the patch to the line you were addressing looks like this: -#if (PY_VERSION_HEX < 0x0302) +#if (PY_VERSION_HEX < ((PY_MAJOR_VERSION < 3) ? 0x0207 : 0x0302)) However, Jesus's use of capsules has a bug. The capsule API requires that the string passed in to PyCapsule_New() must "outlive" the capsule; he's passing in a stack buffer. So my patch also fixes that. Finally my bsddb patch adds the appropriate text to the header file where it describes how to import the _bsddb C API object (CObject vs capsule). FWIW, I want to close this issue soon. How about I close it after 2.7b1 has been out for a week or so--assuming there aren't any new concerns regarding the capsule backport. -- ___ Python tracker <http://bugs.python.org/issue7992> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7992] Backport capsule object
Larry Hastings added the comment: Marking closed, as promised; 2.7b1 has been out for nine days and there hasn't been a peep. If new problems crop up from the capsule backport, please create a new issue and assign it to me. For posterity: 2.7b1 shipped with a bsddb that still publishes a CObject. This is the only CObject left in 2.7b1, and Python doesn't itself use that CObject for anything. Therefore there are no CObject exploits left in pure Python 2.7b1. I don't know if bsddb will switch to a capsule in a later Python 2.7 beta; that's up to Jesus and Benjamin and co. But I suspect it won't. -- resolution: -> accepted status: open -> closed ___ Python tracker <http://bugs.python.org/issue7992> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Larry Hastings added the comment: > In Windows the high-precision counter might return different results > on different cores in some hardware configurations (older multi-core > processors). More specifically: some older multi-core processors where the HAL implements QueryPerformanceCounter using the TSC from the CPU, and the HAL doesn't keep the cores in sync and QPC doesn't otherwise account for it. This is rare; frequently QPC is implemented using another source of time. But it's true: QPC is not 100% reliable. QPC can unfortunately jump backwards (when using TSC and you switch cores), jump forwards (when using TSC and you switch cores, or when using the PCI bus timer on P3-era machines with a specific buggy PCI south bridge controller), speed up or slow down (when using TSC and not accounting for changing CPU speed via SpeedStep &c). The simple solution: give up QPC and use timeGetTime() with timeBeginPeriod(1), which is totally reliable but only has millisecond accuracy at best. http://www.virtualdub.org/blog/pivot/entry.php?id=106 http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q274323&; -- ___ Python tracker <http://bugs.python.org/issue7946> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment:
This isn't a CPython bug. It's a change in CPython behavior that wrapt needs
to accommodate. In particular, it isn't 'causing a regression for C subclasses
of heap types when the parent class has an "__annotations__" descriptor', nor
do child classes have any difficulty inheriting the descriptor of their parent
classes. That's unsurprising; after all, if I had broken child classes
inheriting the descriptors of their parent classes, a lot more would have
broken than just wrapt.
The problem is in WraptObjectProxy_setattro(). (Which--just to drive my point
home--*is* getting called when you set __annotations__ on one of wrapt's
various proxy objects.) WraptObjectProxy_setattro() proxies setattr calls for
wrapped objects to the original object--if "o" is a wrapt proxy object wrapping
"fn", and you run "o.__annotations__ = x", it should actually execute
"fn.__annotations__ = x" under the covers.
Except WraptObjectProxy_setattro() executes *this* code first, starting at line
1531 in my copy of _wrapped.c:
if (PyObject_HasAttr((PyObject *)Py_TYPE(self), name))
return PyObject_GenericSetAttr((PyObject *)self, name, value);
If the *type* has the attribute, then it doesn't proxy the setattr to the
wrapped object. Instead it does a "generic setattr" on the object itself.
PyObject_HasAttr works by attempting a getattr on the type. If that getattr
call succeeds, PyObject_HasAttr returns true. The type here is FunctionWrapper
(WraptFunctionWrapper_Type). Since we're now looking it up on this type
object, we use the type of the type object, which is "type", to access the
attribute. And getting the "__annotations__" attribute from an object of type
"type" means calling type_get_annotations(), a new descriptor which ensures
that the annotations dict always exists, which means the HasAttr call succeeds
and returns true. In short, this change to the semantics of the
"__annotations__" attribute means wrapt no longer proxies the setattr to the
underlying wrapped object when setting the "__annotations__" attribute on *any*
of its objects.
In my opinion, wrapt needs to accommodate this new behavior. In my testing I
changed the above code to this:
if (!annotations_str) {
annotations_str = PyUnicode_InternFromString("__annotations__");
}
if (PyObject_RichCompareBool(name, annotations_str, Py_NE)
&& PyObject_HasAttr((PyObject *)Py_TYPE(self), name))
return PyObject_GenericSetAttr((PyObject *)self, name, value);
I also declared
static PyObject *annotations_str = NULL;
at the top of the function. With that change in place, the tests now passed.
My hunch is, this approach is more or less what wrapt should do. It *might* be
undersophisticated; it's possible that there are classes out there playing
their own weird descriptor tricks with the "__annotations__" attribute.
Perhaps the fix needs to be on a case-by-case basis, based on the type of the
wrapped object. Anyway this is obviously up to Graham, which is for the best
anyway--he has far more experience than I do with this sort of object proxying
wizardry.
--
resolution: -> third party
stage: test needed -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue45319>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment: (It's also possible your workaround where wrapt explicitly defines an "__annotations__" getset on every child of ObjectProxy is the right fix. I don't know; I don't know the fine points of attribute access, like when does it access getsets on the type, vs getsets on base types, vs setattro, etc.) -- ___ Python tracker <https://bugs.python.org/issue45319> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment: Graham: I'm happy to help. I can write a more elaborate explanation of what's happening, or answer questions, whatever you like. I'm a fan of wrapt so I'd certainly like to see this issue resolved. And, seeing as you're the author and maintainer of wrapt, I assure you you've got more battle-won experience with object proxying than most developers--myself included. You're *absolutely* the right person to ameliorate this issue! -- ___ Python tracker <https://bugs.python.org/issue45319> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45319] Possible regression in __annotations__ descr for heap type subclasses
Larry Hastings added the comment: I only did my experiments with the _wrappers.c Christian gave me. If that's not the shipping version of wrapt, and wrapt doesn't exhibit this behavior in 3.10, then that's good. I agree you can wait to address this behavior until it affects code you actually plan to ship. -- ___ Python tracker <https://bugs.python.org/issue45319> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: Nobody I've nosied on this issue recently has expressed any opinion on the matter. I'm gonna try one more person: Graham Dumpleton, the maintainer of "wrapt", Python's premier function-wrapping. Graham, care to express any opinions about this issue? Can we fix it without causing widespread wailing and gnashing of teeth? Do you think people are depending on the current how-can-you-describe-it-as-anything-but-broken behavior? -- nosy: +grahamd ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: You make a good point. I filed a separate bug (#46846) suggesting that partial objects should set their own annotations and signature. I agree that objects performing such magic should take care of these details themselves, rather than requiring the inspect module to have workarounds based on deep knowledge of these other modules' inner workings. -- ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Jack: I've updated the PR, improving compatibility with the "blake3" package on PyPI. I took your notes, and also looked at the C module you wrote. The resulting commit is here: https://github.com/python/cpython/pull/31686/commits/37ce72b0444ad63fd1989ad36be5f7790e51f4f1 Specifically: * derive_key_context is now a string, which I internally encode into UTF-8. * I added the AUTO member to the module, set to -1. * max_threads may not be zero; it can be >= 1 or AUTO. * I added the reset() method. Some additional thoughts, both on what I did and on what you did: * In your new() method, your error string says "keys must be 32 bytes". I went with "key must be exactly 32 bytes"; the name of the parameter is "key", and I think "exactly" makes the message clearer. * In my new() method, I complain if the derive_key_context is zero-length. In your opinion, is that a good idea, or is that overly fussy? * In your copy() method, you hard-code Blake3Type. It's considered good form to use type(self) here, in case the user is calling copy on a user-created subclass of Blake3Type. * In your copy() method, if the original has a lock created, you create a lock in the copy too. I'm not sure why you bother; I leave the lock member uninitialized, and let the existing logic create the lock in the copy on demand. * In the Blake3_methods array, you list the "update" method twice. I suspect this is totally harmless, but it's unnecessary. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: > Given that I don't want to see us gain new vendored copies of > significant but non-critical third party hash code in our tree > (Modules/_blake3/impl/ in PR 31686) for anything but a known > fixed term need (ex: the sha2 libtomcrypt code is gone from > our tree as was clearly going to happen from the start), > the only way I think we should include blake3 support is if > there is already a plan for that code to leave our tree in > the future with a high probability of success. You've said what you want, but not why. It sounds like you are against merging the BLAKE3 PR containing its own impl. Why? -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: Ok, I give up. -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: The Rust version is already quite "lean". And it can be much faster than the C version, because it supports internal multithreading. Even without multithreading I bet it's at least a hair faster. Also, Jack has independently written a Python package based around the C version: https://github.com/oconnor663/blake3-py/tree/master/c_impl so my making one would be redundant. I have no interest in building standalone BLAKE3 PyPI packages for Raspberry Pi or Android. My goal was for BLAKE3 to be one of the "included batteries" in Python--which would have meant it would, eventually, be available on the Raspberry Pi and Android builds that way. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: I can't answer why the Rust one is so much larger--that's a question for Jack. But the blake3-py you built might (should?) have support for SIMD extensions. See the setup.py for how that works; it appears to at least try to use the SIMD extensions on x86 POSIX (32- and 64-bit), x86_64 Windows, and 64-bit ARM POSIX. If you were really curious, you could run some quick benchmarks, then hack your local setup.py to not attempt adding support for those (see "portable code only" in setup.py) and do a build, and run your benchmarks again. If BLAKE3 got a lot slower, yup, you (initially) built it with SIMD extension support. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39298] add BLAKE3 to hashlib
Larry Hastings added the comment: > Performance wise... The SHA series have hardware acceleration on > modern CPUs and SoCs. External libraries such as OpenSSL are in a > position to provide implementations that make use of that. Same with > the Linux Kernel CryptoAPI (https://bugs.python.org/issue47102). > > Hardware accelerated SHAs are likely faster than blake3 single core. > And certainly more efficient in terms of watt-secs/byte. I don't know if OpenSSL currently uses the Intel SHA1 extensions. A quick google suggests they added support in 2017. And: * I'm using a recent CPU that AFAICT supports those extensions. (AMD 5950X) * My Python build with BLAKE3 support is using the OpenSSL implementation of SHA1 (_hashlib.openssl_sha1), which I believe is using the OpenSSL provided by the OS. (I haven't built my own OpenSSL or anything.) * I'm using a recent operating system release (Pop!_OS 21.10), which currently has OpenSSL version 1.1.1l-1ubuntu1.1 installed. * My Python build with BLAKE3 doesn't support multithreaded hashing. * In that Python build, BLAKE3 is roughly twice as fast as SHA1 for non-trivial workloads. -- ___ Python tracker <https://bugs.python.org/issue39298> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: I heard back from both Samuel Colvin (Pydantic) and Sebastián Ramírez (FastAPI). They said neither of them use update_wrapper with partial objects. They also took a peek in a bunch of other projects (FastAPI, Typer, SQLModel, Asyncer, SQLAlchemy, Trio, and AnyIO) and nobody was doing it. So honestly it seems like nobody (but me!) calls update_wrapper on partial objects, and we can just fix it. Graham, any final thoughts before we start pulling levers and merging PRs? For now I just want to fix this bug. I'm in favor of re-engineering the relevant objects so they write their own __signature__ objects, so inspect.Signature doesn't have to understand the internals of objects from other modules. But maybe we save that for another day. -- ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46761] functools.update_wrapper breaks the signature of functools.partial objects
Larry Hastings added the comment: Ooh, good one. I don't know anybody in the Django project to contact though. Anybody have any leads? -- ___ Python tracker <https://bugs.python.org/issue46761> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
New submission from Larry Hastings : I've maintained my own topological sort class for a while now. The API I came up with (or was it Tim and I?) was adopted for graphlib.TopologicalSort(). Some of my method names are slightly different, but the APIs are so similar, I can do a little renaming and run Lib/test/test_graphlib using my implementation. (For clarity I'm going to write the rest of this issue using the names from graphlib.) Recently I made an improvement to my version: it no longer has the modal behavior where there's a "before prepare" phase where you add nodes and an "after prepare" phase where you can call get_ready and done. Instead, in my new and improved version the graph is always maintained in a "ready" state. You can call add, get_ready, and done in any order, as much as you like. prepare doesn't do anything besides the cycle check--but read on! This approach required tweaking some semantics behind the scenes. My graph object now maintains an internal "dirty" flag. It's set to True after any edges are added, and only gets cleared after checking for cycles. prepare runs this cycle check, but get_ready *also* runs this cycle check. (Though in both cases, only when the dirty flag is set, of course.) In theory this means you no longer need the prepare call. However, it's still useful, because you can call it to check for a CycleError. So, on the one hand, this means that get_ready can throw a CycleError, which it never did before. On the other hand, it won't throw for existing users of the library, because they never add edges after their prepare call. So this semantic change shouldn't break existing users, assuming they're not misusing the existing API. Another wrinkle: adding a dependency on a node that's already been handed out by get_ready (but hasn't been returned by done) is ignored. Again, not something existing users of graphlib can do, so this shouldn't break code. There's one final semantic change I made worth knowing about: when you return a node via done, the graph simply forgets about it. This means you could add it a second time (!) and it could go for a complete ride through the graph again, wh! This also means that when all nodes have been returned by done, the graph is essentially returned to its initial pristine state. This too seems completely harmless, because with the existing library it's illegal to add nodes to a graph after calling prepare, so nobody's doing it, so it won't break any code. I'm happy to contribute my version. But I was lazy and didn't worry about speed or memory implementation; it's strewn with sets and things. I figured I could always optimize it later. But "later" could become "now" if we were interested in trying to merge this behavior into Python's graphlib. Interested? -- assignee: larry components: Library (Lib) messages: 416182 nosy: eric.smith, larry, pablogsal, tim.peters priority: normal severity: normal stage: needs patch status: open title: Improve graphlib.TopologicalSort by removing the prepare step type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: I'm using my graph library to manage a list of tasks that need doing in some sort of proper order. One task may spawn other tasks at runtime, and we don't necessarily know what the tasks will be until runtime. It's way more convenient to simply add such tasks on demand, rather than trying to preemptively pre-add all such possible tasks before preparing the graph, or creating additional graphs. For example, consider a tool that downloads and digests zip files full of music from an online store. Your initial tasks might represent "download the zip file", then "decompress and examine the contents of the zip file". You could then iterate over the contents of the zip file, adding different tasks based on what you find--one pipeline of tasks for media files (FLAC/MP3/OGG/etc), another for the playlist, a third if you don't *find* a playlist, a fourth for image files, etc. (Not that you'd necessarily write such a tool this way, but it's at least plausible.) The new nodes needn't be connected to the existing nodes for this to still be useful. You could reuse the same graph object for all your tasks. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: > Assuming we do want to be able to add() after a get_ready(), is there > a reason that "forgetting" already-produced nodes is the correct > behavior, as opposed to remembering all nodes ever added, and > raising iff the addition creates a cycle among all nodes ever > added or depends on an already-yielded node? I'm not sure "correct" applies here, because I don't have a sense that one behavior is conceptually more correct than the other. But in implementing my API change, forgetting about the done nodes seemed more practical. The "benefit" to remembering done nodes: the library can ignore dependencies to them in the future, forever. Adding a dependency to a node that's already been marked as "done" doesn't make much conceptual sense to me, but as a practical thing maybe it's useful? I'm not sure, but it doesn't seem all that useful. I can only come up with a marginal reason why remembering done nodes is useful. Let's say all your tasks fan out from some fundamental task, like "download master list of work" or something. One of your tasks might discover additional tasks that need to run, and conceptually those tasks might depend on your "download master list of work" task. If the graph remembers the done list forever, then adding that dependency is harmless. If the graph forgets about done nodes, then adding that dependency could re-introduce that task to the graph, which could goof things up. So maybe it's a little bit of a footgun? But on the other hand: you already know you're running, and you're a task that was dependent on the master list of work, which means you implicitly know that dependency has been met. So just skip adding the redundant dependency and you're fine. On the other hand, forgetting about the nodes has a definite practical benefit: the graph consumes less memory. If you use a graph object for a long time, the list of done nodes it's holding references to would continue to grow and grow and grow. If we forget about done nodes, we free up all that memory, and done membership testing maybe gets faster. I guess I'm not married to the behavior. If someone had a great conceptual or practical reason why remembering the done nodes forever was better, I'd be willing to listen to reason. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: Having slept on it, I think the footgun is slightly bigger than I gave it credit for. Also the problem of nodes lingering forever is easily solved: give the user control. My next iteration will keep the done nodes around, but I'll also add a forget() method that compels the graph to forget all done nodes. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: I'm not sure I follow you. What do you suggest graphlib is guessing at? I thought we were discussing what graphlib's (documented, intentional) behavior should be; if there was any guessing going on, it was us doing it, guessing at what behavior the user would prefer. I appreciate you corresponding on the issue, but I'm having difficulty understanding what light you're shining on the problem. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: I agree that the API should have as few surprises as possible. AFAICT you haven't made any terminal pronouncements like "it's impossible to add this feature without too many unacceptable surprises", so I'll proceed assuming we can find an API (and semantics) that we're all happy with. I've come around on the idea that forgetting "done" nodes is too surprising. The least surprising behavior is: once we've added a node to the graph, the graph remembers it. Now I'll propose a second simple rule, which you've already touched on: you can't add a dependency after a node has been returned by get_ready(). Attempting this always throws an exception; which exception specifically TBD. "Tough beans". I think those two rules are easy enough to remember and to reason about. It meaningfully expands the utility of the library with minimal surprise. The library behaves identically for users of the existing API but permits new use cases too. Adding this functionality would therefore mean fewer users would discover too late their use case isn't supported. As far as my "long-lived graph objects will consume more and more memory over time" caveat, there's a better solution than "forget": graph.remove(*nodes). (My version already has a "remove" method, and I forgot that graphlib's doesn't.) Allowing the user to remove a node from the graph gives them explicit control, and the semantics should be obvious and unsurprising; if you--the user--remove a node, and later you--the user--re-add that node to the graph, it behaves identically to any other node the graph has never seen before. Removing a node intuitively removes all edges to that node. Two notes on "remove" if we decide to go that route. First, I'd ensure you can remove a node at any time. Nodes have three externally visible states wrt TopologicalSort: 1) added but not published by get_ready, 2) published by get_ready but not returned using done, and 3) done. You should be able to remove a node in any of those three states. Removing a node in 2) should be equivalent to calling done before calling remove; that is, if you're removing the node anyway, you don't need to call done. Second, the current underlying implementation would make remove really slow. Nodes don't remember their predecessors, only their successors, so removing a node would be O(n). If we expected remove to get a lot of use, we'd probably want to change how the graph is stored to speed up this operation. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47145] Improve graphlib.TopologicalSort by removing the prepare step
Larry Hastings added the comment: One final aside. Let me preface this by saying: I'm not proposing the following for graphlib.TopologicalSort. It's obviously too late to change that object this much. It's just something I'm thinking about--maybe I'll use this in my own library. Where we are now, the graphlib.TopologicalSort object is simultaneously a static representation of a graph--which the object doesn't provide a public API for you to examine!--and a stateful single-use iterator over that graph. My proposed change to the API seems to increase the tension between these two sets of semantics. Perhaps a better set of semantics, that more successfully maps my use case to the graph object, would be as follows: * you can add() nodes and edges at any time. * get_ready() always returns the current list of nodes with no prececessors. There is no internal "we already told you about this one" state. * replace done() with remove(), which removes the node and all edges from/to it from the graph. * static_order() is still fine. I think this would make it easy to reason about the object's behavior, and would be a better match to my use case where you're continually adding (and removing?) nodes, not just during an initial "populate the graph" phase. Again, not appropriate to consider for graphlib.TopologicalSort. -- ___ Python tracker <https://bugs.python.org/issue47145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34155] email.utils.parseaddr mistakenly parse an email
Larry Hastings added the comment: New changeset 063eba280a11d3c9a5dd9ee5abe4de640907951b by larryhastings (Abhilash Raj) in branch '3.5': [3.5] bpo-34155: Dont parse domains containing @ (GH-13079) (#15317) https://github.com/python/cpython/commit/063eba280a11d3c9a5dd9ee5abe4de640907951b -- nosy: +larry ___ Python tracker <https://bugs.python.org/issue34155> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34155] email.utils.parseaddr mistakenly parse an email
Larry Hastings added the comment: All PRs merged. Thanks, everybody! -- resolution: -> fixed status: open -> closed ___ Python tracker <https://bugs.python.org/issue34155> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36576] Some test_ssl and test_asyncio tests fail with OpenSSL 1.1.1 on Python 3.4 and 3.5
Larry Hastings added the comment: New changeset 4d1c2541c125fe9d211016193ebfd5899a8511aa by larryhastings (Victor Stinner) in branch '3.5': bpo-36576: Skip test_ssl and test_asyncio tests failing with OpenSSL 1.1.1 (#12694) https://github.com/python/cpython/commit/4d1c2541c125fe9d211016193ebfd5899a8511aa -- nosy: +larry ___ Python tracker <https://bugs.python.org/issue36576> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
