[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Change by Greg Price : -- pull_requests: +14969 pull_request: https://github.com/python/cpython/pull/15248 ___ Python tracker <https://bugs.python.org/issue37

[issue32771] merge the underlying data stores of unicodedata and the str type

Change by Greg Price : -- nosy: +Greg Price ___ Python tracker <https://bugs.python.org/issue32771> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Greg Price added the comment: > I like to run pyflakes time to time on the Python code base. Please avoid > "import *" since it prevents pyflakes (and other code analyzers) to find bugs. Ah fair enough, thanks! Pushed that change to the next/curr

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Greg Price added the comment: > BTW: Since when do we use type annotations in Python's stdlib ? Hmm, interesting question! At a quick grep, it's in a handful of places in the stdlib: asyncio, functools, importlib. The earliest it appeared was in 3.7.0a4. It's in more

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Greg Price added the comment: > What is the minimal Python version for developing CPython? The system Python > 3 on current Ubuntu LTS (18.04) is 3.6, so I think it should not be larger. Ah, I think my previous message had an ambiguous parse: the earliest that *uses* of the typing

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Greg Price added the comment: > This is good. But the title mentioned dataclasses, and they are 3.7+. Ahh, sorry, I think now I understand you. :-) Indeed, when I switch to the branch with that change (https://github.com/gnprice/cpython/commit/2b4aec4dd -- it comes after the patch tha

[issue32771] merge the underlying data stores of unicodedata and the str type

Greg Price added the comment: > Loading it dynamically reduces the memory footprint. Ah, this is a good question to ask! First, FWIW on my Debian buster desktop I get a smaller figure for `import unicodedata`: only 64 kiB. $ python Python 3.7.3 (default, Apr 3 2019, 05:39:12) [GCC 8.

[issue32771] merge the underlying data stores of unicodedata and the str type

Greg Price added the comment: Speaking of improving functionality: > Having unicodedata readily accessible to the str type would also permit > higher a fidelity unicode implementation. For example, implementing > language-tailored str.lower() requires having canonical combining c

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Greg Price added the comment: > From my perspective, the main problem with using type annotations is that > there's nothing checking them in CI. Yeah, fair concern. In fact I think I'm on video (from PyCon 2018) warning everyone not to do that in their codebases, because

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Change by Greg Price : -- pull_requests: +14985 pull_request: https://github.com/python/cpython/pull/15265 ___ Python tracker <https://bugs.python.org/issue37

[issue37848] More fully implement Unicode's case mappings

New submission from Greg Price : Splitting this out from #32771 for more specific discussion. Benjamin writes there that it would be good to: > implement the locale-specific case mappings of > https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt and §3.13 of > the U

[issue37848] More fully implement Unicode's case mappings

Greg Price added the comment: Another previous discussion is #4610. -- ___ Python tracker <https://bugs.python.org/issue37848> ___ ___ Python-bugs-list mailin

[issue32771] merge the underlying data stores of unicodedata and the str type

Greg Price added the comment: OK, I forked off the discussion of case-mapping as #37848. I think it's probably good to first sort out what we want, before returning to how to implement it (if it's agreed that changes are desired.) Are there other areas of functionality that would

[issue37848] More fully implement Unicode's case mappings

Greg Price added the comment: > I believe that all locale specific things should be in the locale module, not > in the str class. The locale module is all about doing things with the current process-global Unix locale. I don't think that'd be an appropriate interface for

[issue37848] More fully implement Unicode's case mappings

Greg Price added the comment: > Maintaining Python is already expensive [...] There are already enough bugs > waiting for you to be fixed ;-) BTW I basically agree with this. I think this is not a high-priority issue, and I have my eye on some of those bugs. :-) I think the fact tha

[issue37848] More fully implement Unicode's case mappings

Greg Price added the comment: (I should add that it was only after doing the reading that produced the OP that I had a clear idea what I thought the priority of the issue was -- before doing that work I didn't have a clear sense of the scope of what it affects. Based on

[issue36502] str.isspace() for U+00A0 and U+202F differs from document

Change by Greg Price : -- pull_requests: +15019 pull_request: https://github.com/python/cpython/pull/15296 ___ Python tracker <https://bugs.python.org/issue36

[issue32771] merge the underlying data stores of unicodedata and the str type

Greg Price added the comment: > About the RSS memory, I'm not sure how Linux accounts the Unicode databases > before they are accessed. Is it like read-only memory loaded on demand when > accessed? It stands for "resident set size", as in "resident in memory&q

[issue37758] unicodedata checksum-tests only test 1/17th of Unicode's codepoints

Change by Greg Price : -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue37758> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37864] Correct and deduplicate docs on "printable" characters

New submission from Greg Price : While working on #36502 and then #18236 about the definition and docs of str.isspace(), I looked closely also at its neighbor str.isprintable(). It turned out that we have the definition of what makes a character "printable" documented in three plac

[issue37864] Correct and deduplicate docs on "printable" characters

Change by Greg Price : -- keywords: +patch pull_requests: +15025 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15300 ___ Python tracker <https://bugs.python.org/issu

[issue36502] str.isspace() for U+00A0 and U+202F differs from document

Change by Greg Price : -- pull_requests: +15026 pull_request: https://github.com/python/cpython/pull/15301 ___ Python tracker <https://bugs.python.org/issue36

[issue37758] unicodedata checksum-tests only test 1/17th of Unicode's codepoints

Change by Greg Price : -- pull_requests: +15027 pull_request: https://github.com/python/cpython/pull/15302 ___ Python tracker <https://bugs.python.org/issue37

[issue37872] Move statics in Python/import.c to top of the file

Change by Greg Price : -- nosy: +Greg Price ___ Python tracker <https://bugs.python.org/issue37872> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37872] Move _Py_IDENTIFIER statics in Python/import.c to top of the file

Change by Greg Price : -- title: Move statics in Python/import.c to top of the file -> Move _Py_IDENTIFIER statics in Python/import.c to top of the file ___ Python tracker <https://bugs.python.org/issu

[issue37872] Move _Py_IDENTIFIER statics in Python/import.c to top of the file

Change by Greg Price : -- components: +Interpreter Core ___ Python tracker <https://bugs.python.org/issue37872> ___ ___ Python-bugs-list mailing list Unsub

[issue36502] str.isspace() for U+00A0 and U+202F differs from document

Greg Price added the comment: Thanks Victor for the reviews and merges! (Unmarking 2.7, because https://docs.python.org/2/library/stdtypes.html seems to not have this issue.) -- versions: -Python 2.7 ___ Python tracker <https://bugs.python.

[issue35518] test_timeout uses blackhole.snakebite.net domain which doesn't exist anymore

Change by Greg Price : -- keywords: +patch pull_requests: +15063 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15349 ___ Python tracker <https://bugs.python.org/issu

[issue35518] test_timeout uses blackhole.snakebite.net domain which doesn't exist anymore

Greg Price added the comment: I ran across this test when looking at especially slow files in the test suite: it turns out that not only is this service currently down, but the snakebite.net domain still exists, and as a result the test can end up waiting 20-30s before learning that the

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Greg Price added the comment: (A bit easy to miss in the way this thread gets displayed, so to highlight in a comment: GH-15265 is up, following the 5 other patches which have now all been merged. That's the one that replaces the length-18 tuples with a data

[issue36375] PEP 499 implementation: "python -m foo" binds the main module as both __main__ and foo in sys.modules

Change by Greg Price : -- nosy: +Greg Price ___ Python tracker <https://bugs.python.org/issue36375> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Change by Greg Price : -- pull_requests: +15140 pull_request: https://github.com/python/cpython/pull/15448 ___ Python tracker <https://bugs.python.org/issue37

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Greg Price added the comment: Thanks, Raymond, for the review on GH-15216! Shortly after posting this issue, I noticed a very similar story in CHECK_BINOP. I've just posted GH-15448 to similarly make returns explicit there. It basically consists of a number of repetitio

[issue37936] gitignore file is too broad

New submission from Greg Price : There are a number of files that we track in the repo, but are nevertheless covered by `.gitignore`. This *mostly* doesn't change anything, because Git itself only cares what `.gitignore` has to say about files that aren't already tracked. But: *

[issue37936] gitignore file is too broad

Change by Greg Price : -- keywords: +patch pull_requests: +15143 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15451 ___ Python tracker <https://bugs.python.org/issu

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Greg Price added the comment: > May I suggest directing your efforts towards fixing known bugs or > implementing requested features. Well, I would certainly be grateful for a review on my fix to #18236. ;-) There's also a small docs bug at GH-15301. I do think there's sign

[issue37837] add internal _PyLong_FromUnsignedChar() function

Greg Price added the comment: Hmm, I'm a bit confused because: * Your patch at GH-15251 replaces a number of calls to PyLong_FromLong with calls to the new _PyLong_FromUnsignedChar. * That function, in turn, just calls PyLong_FromSize_t. * And that function begins: PyO

[issue37837] add internal _PyLong_FromUnsignedChar() function

Greg Price added the comment: Oh also: * What compiler, and what compilation flags, are you using in your benchmarking? That seems relevant :) -- ___ Python tracker <https://bugs.python.org/issue37

[issue37837] add internal _PyLong_FromUnsignedChar() function

Greg Price added the comment: > Is there a particular reason to specifically call PyLong_FromSize_t? Seems > like PyLong_FromLong is the natural default (and what we default to in the > rest of the code), and it's what this ends up calling anyway. Ah I see, the patch is mean

[issue18236] str.isspace should use the Unicode White_Space property

Greg Price added the comment: > I've gone and made a patch for this change Update: * The preparatory changes in #37760 are now almost all merged; GH-15265 is the one piece remaining, and I'd be grateful for a review. It's a generally straightforward and boring change that

[issue37837] add internal _PyLong_FromUnsignedChar() function

Greg Price added the comment: Very interesting, thanks! It looks like with LTO enabled, this optimization has no effect at all. This change adds significant complexity, and it seems like the hoped-for payoff is entirely in terms of performance on rather narrowly-focused microbenchmarks

[issue37936] gitignore file is too broad

Change by Greg Price : -- pull_requests: +15219 pull_request: https://github.com/python/cpython/pull/15542 ___ Python tracker <https://bugs.python.org/issue37

[issue37936] gitignore file is too broad

Greg Price added the comment: > I have a minimal fix which takes care of all the files above. I'll post that > shortly, and I may also write up a more thorough fix that tries to make it > easy not to fall into the same Git pitfall again. Both now done. * GH-15451 is that minim

[issue37966] is_normalized is much slower than the standard's algorithm

New submission from Greg Price : In 3.8 we add a new function `unicodedata.is_normalized`. The result is equivalent to `str == unicodedata.normalize(form, str)`, but the implementation uses a version of the "quick check" algorithm from UAX #15 as an optimization to try to avoid

[issue37966] is_normalized is much slower than the standard's algorithm

Change by Greg Price : -- keywords: +patch pull_requests: +15231 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15558 ___ Python tracker <https://bugs.python.org/issu

[issue37966] is_normalized is much slower at "no" than the standard's algorithm

Greg Price added the comment: Fix posted, as GH-15558. Adding cc's for the folks in the thread on #32285, where this function was originally added. -- components: +Unicode nosy: +Maxime Belanger, benjamin.peterson, ezio.melotti, steven.daprano, vstinner title: is_normalized is

[issue6331] Add unicode script info to the unicode database

Change by Greg Price : -- nosy: +Greg Price ___ Python tracker <https://bugs.python.org/issue6331> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37837] add internal _PyLong_FromUnsignedChar() function

Greg Price added the comment: Ah OK, that makes sense of it then :) > But the most important thing is that using PyLong_FromUnsignedLong() instead > of _PyLong_FromUnsignedChar() on top of GH-15192 is producing the same > results: striter_next() uses small_ints[] directly. Howev

[issue38043] small cleanups in Unicode normalization code

New submission from Greg Price : Benjamin noticed in reviewing GH-15558 (for #37966) several points where the existing code around Unicode normalization can be improved: * on the `QuickcheckResult` enum: > Maybe `makeunicodedata.py` should output this enum (with better name namespac

[issue38043] small cleanups in Unicode normalization code

Change by Greg Price : -- keywords: +patch pull_requests: +15366 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15711 ___ Python tracker <https://bugs.python.org/issu

[issue38043] small cleanups in Unicode normalization code

Change by Greg Price : -- pull_requests: +15367 pull_request: https://github.com/python/cpython/pull/15712 ___ Python tracker <https://bugs.python.org/issue38

[issue38043] small cleanups in Unicode normalization code

Change by Greg Price : -- pull_requests: +15368 pull_request: https://github.com/python/cpython/pull/15558 ___ Python tracker <https://bugs.python.org/issue38

[issue38015] inline function generates slightly inefficient machine code

Greg Price added the comment: (Just to help keep discussions together: some earlier discussion was on GH-15216 .) Because is_small_int / IS_SMALL_INT is so small, there's not much cost in the source code to making it a macro (as GH-15710 did). But I think it'd be a mistake t

[issue38079] _PyObject_VAR_SIZE should avoid arithmetic overflow

New submission from Greg Price : Currently `_PyObject_VAR_SIZE` effectively has a precondition that it must not be passed arguments which would overflow in its arithmetic. If that's violated, it overflows... which is undefined behavior already, and in fact the likely next thing that ha

[issue38079] _PyObject_VAR_SIZE should avoid arithmetic overflow

Greg Price added the comment: (The tracker just linked GH-14838 to this issue because I mentioned it in a comment there, but it's not for this issue -- it's that recent fix for an 11-year-old bug in a callsite's overflow check.) -- ___

[issue37936] gitignore file is too broad

Change by Greg Price : -- pull_requests: +15472 pull_request: https://github.com/python/cpython/pull/15823 ___ Python tracker <https://bugs.python.org/issue37

[issue37936] gitignore file is too broad

Greg Price added the comment: Thanks @zach.ware for the review and merge of GH-15451! That was the "minimal" fix, fixing rules that apply to files we have in the repo now. So `rg` will no longer ignore `PC/pyconfig.h`. :-) I've just sent GH-15823 with the "more thorough

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Greg Price added the comment: Thanks Victor for linking that issue back here. > A first change converted a macro to a static inline function. The second > change converted the static inline fnuction to a macro Not quite. The first change converted a macro `CHECK_SMALL_INT` to an equi

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Greg Price added the comment: > if using a static inline function is causing issues Separately from whether there was or wasn't such an issue here, I think it's interesting to note that the build failure bpo-38205 is an example of exactly the opposite! It was caused by a comb

[issue38015] inline function generates slightly inefficient machine code

Greg Price added the comment: See followup at https://bugs.python.org/issue38205 and https://bugs.python.org/issue37812#msg352670 . The patch in GH-15710 had a side effect of introducing a call to `Py_UNREACHABLE` inside a comma-expression. A subsequent commit 3ab61473b changed

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Greg Price added the comment: > We're wasted a lot of dev time on something that never promised much value in > the first place. So, I'll revert and close this tracker issue OK, so be it. I'll certainly agree that this series of threads consumed a lot more time than I

[issue18236] str.isspace should use the Unicode White_Space property

Change by Greg Price : -- keywords: +patch pull_requests: +15849 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/16254 ___ Python tracker <https://bugs.python.org/issu

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Greg Price added the comment: Thanks Benjamin for reviewing and merging this series! -- ___ Python tracker <https://bugs.python.org/issue37760> ___ ___ Pytho

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Greg Price added the comment: I hesitate to come back to this thread, because as Raymond says it's consumed a lot of time already. But I think this point is of broader interest than the specific few lines we've been discussing: > When small integer are disabled at com

[issue16684] Unicode property value abbreviated names and long names

Greg Price added the comment: I've gone and implemented a version of this that's integrated into Tools/unicode/makeunicodedata.py , and into the unicodedata module. Patch attached. Demo: >>> import unicodedata, pprint >>> pprint.pprint(unicodedata.property_va

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Greg Price added the comment: > labeling long-stable code as "evil". Let me apologize about this bit -- I was thinking of the term in quotes (referring to the earlier comment), but I didn't write it clearly that way. I don't think any of this code is evil, past o

[issue1524081] logging using the SysLog handler fails if locale is set

Greg Price added the comment: For the record because this issue is mentioned in a comment in logging/handlers.py and people are sometimes confused by it today: > This happens because in that particular locale, > "INFO".lower() != "info" Since Python 3, this no

[issue36502] str.isspace() for U+00A0 and U+202F differs from document

Change by Greg Price : -- keywords: +patch pull_requests: +14836 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15019 ___ Python tracker <https://bugs.python.org/issu

[issue36502] str.isspace() for U+00A0 and U+202F differs from document

Greg Price added the comment: The actual behavior turns out to match that comment. See attached PR, which adds a test confirming that and also corrects the documentation. (A related issue is #18236 -- we should probably adjust the definition to match the one Unicode now provides. But

[issue37758] unicodedata checksum-tests only test 1/17th of Unicode's codepoints

New submission from Greg Price : The unicodedata module has two test cases which run through the database and make a hash of its visible outputs for all codepoints, comparing the hash against a checksum. These are helpful regression tests for making sure the behavior isn't chang

[issue37758] unicodedata checksum-tests only test 1/17th of Unicode's codepoints

Change by Greg Price : -- keywords: +patch pull_requests: +14865 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15125 ___ Python tracker <https://bugs.python.org/issu

[issue37758] unicodedata checksum-tests only test 1/17th of Unicode's codepoints

Change by Greg Price : -- pull_requests: +14866 pull_request: https://github.com/python/cpython/pull/15126 ___ Python tracker <https://bugs.python.org/issue37

[issue37758] unicodedata checksum-tests only test 1/17th of Unicode's codepoints

Greg Price added the comment: Sent two small PRs! The first one, GH-15125, makes the substantive test change I described above. The second one, GH-15126, is a small pure refactor to that test file, just cleaning out some bits that made sense when it was first written (as a script) but are

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

New submission from Greg Price : I spent some time yesterday on #18236, and I have a patch for it. Most of that work happens in the script Tools/unicode/makeunicode.py , and along the way I made several changes there that I found made it somewhat nicer to work on, and I think will help other

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Change by Greg Price : -- keywords: +patch pull_requests: +14868 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15128 ___ Python tracker <https://bugs.python.org/issu

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Change by Greg Price : -- pull_requests: +14869 pull_request: https://github.com/python/cpython/pull/15129 ___ Python tracker <https://bugs.python.org/issue37

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Change by Greg Price : -- pull_requests: +14870 pull_request: https://github.com/python/cpython/pull/15130 ___ Python tracker <https://bugs.python.org/issue37

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Greg Price added the comment: Just posted three PRs: * GH-15128 and GH-15129 are both quite small * GH-15130 is the first of two patches factoring out common parsing logic. Two remaining patches go on top of GH-15130. Here are drafts, in case they're helpful for reference: * Patc

[issue18236] str.isspace should use the Unicode White_Space property

Greg Price added the comment: I've gone and made a patch for this change: https://github.com/gnprice/cpython/commit/7dab9d879 Most of the work happens in the script Tools/unicode/makeunicode.py , and along the way I made several changes there that I found made it somewhat nicer to wo

[issue37760] Refactor makeunicodedata.py: dedupe parsing, use dataclass

Change by Greg Price : -- pull_requests: +14903 pull_request: https://github.com/python/cpython/pull/15171 ___ Python tracker <https://bugs.python.org/issue37

[issue18236] str.isspace should use the Unicode White_Space property

Greg Price added the comment: Good question! With the patch: >>> import re >>> re.match(r'\s', '\x1e') >>> In other words, the definition of the regexp r'\s' follows along. Good to know. -- ___

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

New submission from Greg Price : In longobject.c we have the following usage a few times: PyObject * PyLong_FromLong(long ival) { PyLongObject *v; // ... more locals ... CHECK_SMALL_INT(ival); if (ival < 0) { /* negate: can't write this as abs_ival = -ival si

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Change by Greg Price : -- keywords: +patch pull_requests: +14932 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15203 ___ Python tracker <https://bugs.python.org/issu

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Greg Price added the comment: I've just sent GH-15203 which is the first of two patches for this. It's quite small. The second patch, which completes the change, is also small: https://github.com/gnprice/cpython/commit/c6b905104 It depends on the first one, so I think the easiest

[issue37802] micro-optimization of PyLong_FromSize_t()

Change by Greg Price : -- nosy: +Greg Price ___ Python tracker <https://bugs.python.org/issue37802> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Greg Price added the comment: > Sometimes for testing we turn it off in order to identify identity test bugs. Interesting! Well, if the option is useful for testing, that's certainly a good reason to keep it. > Also, eveonline was controlling this to save memory. Also interest

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Change by Greg Price : -- pull_requests: +14944 pull_request: https://github.com/python/cpython/pull/15216 ___ Python tracker <https://bugs.python.org/issue37

[issue37812] Make implicit returns explicit in longobject.c (in CHECK_SMALL_INT)

Greg Price added the comment: > On the other hand, if it is tricky and requires something more than minor > surgery, that would be a hint that it isn't worth it. There is some value in > code that is stable and known to be working just fine. Definitely true! I think this cha

[issue26452] Wrong line number attributed to comprehension expressions

New submission from Greg Price: In a multi-line list comprehension (or dict or set comprehension), the code for the main expression of the comprehension is wrongly attributed to the *last* line of the comprehension, which might be several lines later. This makes for quite baffling tracebacks