Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
> * Add new GIL-free (no need to hold the GIL) memory allocator functions: > > - ``void* PyMem_RawMalloc(size_t size)`` > - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)`` > - ``void PyMem_RawFree(void *ptr)`` > - the behaviour of requesting zero bytes is not defined: return *NULL* > or a distinct non-*NULL* pointer depending on the platform. > (...) > * Add new functions to get and set internal functions of > ``PyMem_Malloc()``, ``PyMem_Realloc()`` and ``PyMem_Free()``: > > - ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` > - ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` > - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return > *NULL*: it would be treated as an error. > - default allocator: ``malloc()``, ``realloc()``, ``free()``; > ``PyMem_Malloc(0)`` calls ``malloc(1)`` > and ``PyMem_Realloc(NULL, 0)`` calls ``realloc(NULL, 1)`` Oh, one more question: PyMem_RawMalloc(0) has an undefined behaviour, whereas PyMem_Malloc(0) has a well defined behaviour (don't return NULL). Adding "if (size == 1) size = 0;" in the default implementation of PyMem_RawMalloc(0) should not have a visible overhead, but it gives the same guarantee than PyMem_Malloc(0) (don't return NULL). Do you agree to add the test? I chose to implement "if (size > (size_t)PY_SSIZE_T_MAX) return NULL;" in Py*_Malloc(), whereas "if (size == 0) size =1;" is implemented in the inner function (_PyMem_Malloc). An application may use an allocator which has already a well defined behaviour (a "malloc(0)" that don't return NULL) and I expect malloc(1) to allocate "more" memory than malloc(0) (malloc(0) may create a singleton) :-) Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
Oh, please don't misunderstand. I'm not making any demands or requirements, what I'm trying to do is to make recommendations based on experience that I have had with embedding. This sound altogether too much like I'm trying to push things one way or the other :) The api as laid out certainly seems to work, and be adequate for the purpose. I can add here as a point of information that since we work on windows, there was no need to pass in the "size" argument to the "munmap" callback. VirtualFree(address, NULL) will release the entire chunk of memory that was initially allocated at that place. Therefor in our implementation we can reuse the same allocator structo for those arenas. But I understand that munmap doesn't have this feature, so passing in the size is prudent. K > -Original Message- > From: Victor Stinner [mailto:[email protected]] > Sent: 19. júní 2013 15:59 > To: Kristján Valur Jónsson > Cc: Python Dev > Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python > memory allocators > > Is PyMemMappingAllocator complete enough for your usage at CCP Games? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round)
Hi, I changed the PEP 445 according to the discussing on python-dev. Read it online: http://www.python.org/dev/peps/pep-0445/ Changes: * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or PYALLOC_PYOBJECT * rename: - PyMemBlockAllocator structure => PyMemAllocator - PyMemMappingAllocator structure => PyObjectArenaAllocator - PyMem_GetMappingAllocator() => PyObject_GetArenaAllocator() - PyMem_SetMappingAllocator() => PyObject_SetArenaAllocator() * group get/set functions to only keep 2 functions: PyMem_GetAllocator() and PyMem_SetAllocator() * PyMem_RawMalloc(0) now calls malloc(1) to have a well defined behaviour * PYALLOC_PYMEM_RAW and PYALLOC_PYMEM are now using exactly the same allocator * Add more references for external libraries As expected, most changes occurred in the Proposal section. Full diff: http://hg.python.org/peps/rev/a17ebebe52ca I also updated the implementation attached to: http://bugs.python.org/issue3329 Who is going give the final decision on this PEP? Guido? Another candidate? Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round)
> I also updated the implementation attached to: > http://bugs.python.org/issue3329 I excluded new enums, structures and functions from the stable ABI, except PyMem_Raw*() functions. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round)
Le Thu, 20 Jun 2013 13:26:52 +0200, Victor Stinner a écrit : > Hi, > > I changed the PEP 445 according to the discussing on python-dev. > > Read it online: > http://www.python.org/dev/peps/pep-0445/ > > Changes: > > * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or > PYALLOC_PYOBJECT PYMEM_DOMAIN_RAW? > * rename: > >- PyMemBlockAllocator structure => PyMemAllocator >- PyMemMappingAllocator structure => PyObjectArenaAllocator >- PyMem_GetMappingAllocator() => PyObject_GetArenaAllocator() >- PyMem_SetMappingAllocator() => PyObject_SetArenaAllocator() > > * group get/set functions to only keep 2 functions: >PyMem_GetAllocator() and PyMem_SetAllocator() > * PyMem_RawMalloc(0) now calls malloc(1) to have a well defined > behaviour > * PYALLOC_PYMEM_RAW and PYALLOC_PYMEM are now using exactly the same > allocator > * Add more references for external libraries > > As expected, most changes occurred in the Proposal section. > > Full diff: > http://hg.python.org/peps/rev/a17ebebe52ca > > I also updated the implementation attached to: > http://bugs.python.org/issue3329 > > Who is going give the final decision on this PEP? Guido? Another > candidate? > > Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
On 20 June 2013 15:37, Victor Stinner wrote: > Le jeudi 20 juin 2013, Nick Coghlan a écrit : >> >> > Is PyMemMappingAllocator complete enough for your usage at CCP Games? >> >> Can we go back to calling this the "Arena" allocator? Or at least >> "Mapped"? When I see "Mapping" in the context of Python I think of the >> container API, not a memory allocation API. > > This function is written to be able to use mmap() and VirtualAlloc(). There > is no Python function to use directly this allocator yet, but I chose > "memory mapping" name because it is very different than the heap and it may > be useful for other functions than pymalloc. > > If I change the name, it would be called PyObject_SetArenaAllocator() with a > PyObjectArenaAllocator structure. I'm not sure that PyMemMappingAllocator > API is future-proof, so I'm fine to call it "arena" again. Yeah, I think making that API specifically about pymalloc is a good idea. It also makes it clearer that if you're bypassing pymalloc entirely (by replacing the object allocators), then you shouldn't need to worry about those. >> > I hope that the PEP 445 is flexible enough to allow you to decide >> > which functions are hooked and replaced, and which functions will be >> > leaved unchanged. That's why I'm not in favor of the "Make >> > PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative. >> >> It's also why I'm in favour of the "domain" API rather than separate >> functions. >> >> 1. In the initial iteration, just have the three basic domains (raw, >> interpreter, objects). Replacing allocators for third party libraries is the >> responsibility of embedding applications. >> >> 2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs >> so that extension modules can register new domains for wrapped libraries. >> Replacing allocators is still the responsibility of embedding applications, >> but there's a consistent API to do it. >> >> (Alternatively, we could do both now) > > How would you use an allocator of a new domain? PyMemBlockAllocator > structure is not convinient, and if Py_GetAllocator() only once, you may > loose a hook installed later. Say that, for the current PEP, we assume we configure standard library extension modules to use the PyMem or PyMem_Raw APIs (depending on GIL usage), thus allowing those to be redirected automatically to an externally configured allocator when the new PEP 445 APIs are used. The notion I had an mind as a possible future change is that extension modules could register a "set_allocator" callback with the interpreter so they will be automatically notified if the allocator they're interested in changes. However, I also realised that this would actually be independent of the APIs in the current PEP. You could do something like: typedef void (*PyMem_AllocatorSetter)(PyMemAllocator *allocator); void PyMem_AddExternalAllocator( PyMemAllocatorDomain domain, PyMemAllocatorSetter set_allocator ); Then, whenever the allocator for the specified domain was changed, the "set_allocator" callback would be invoked to set the allocator in the extension module as well. The setter would also be called immediately on registration, using the currently defined allocator. We don't have to do this right away (and we should give the basic API a chance to establish itself first). I just like it as something that the "memory domain" model may allow us to pursue in the future. (That said, we may end up wanting something like this internally anyway, even just for the standard library extension modules that do memory allocations) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] stat module in C -- what to do with stat.py?
Hello, I have re-implemented the entire stat module in C. [1] It's a necessary step to fix portability issues. For most constants POSIX standards dictate only the name of the constant and its meaning but not its value. Only the values of permission bits (0644 == rw-r--r--) have fixed values. For example S_IFDIR is usually 0o4 but it might be 0o2 on some platforms. Common file types seem to have the same value on all important platforms. It's the only reason we were able to get away with stat.py. But uncommon file types like door, event port and whiteout don't have sensible default values. The C implementation is able to pick up the platform's values easily. What shall I do about stat.py? statmodule.c implements all features of stat.py so there is no point in using it in CPython. It's one Python file less to load on every startup. However it might still be useful to alternative implementations of Python such as Jython or PyPy. 1) keep the file stat.py but let it be shadowed by the builtin stat module. Antoine loathes my hack... 2) rename stat.py to _stat.py 3) remove stat.py Opinions? Christian [1] http://bugs.python.org/issue11016 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
we already have "_pyio.py", we could have "_pystat.py". my 2c -- Florent Xicluna 2013/6/20 Christian Heimes : > Hello, > > I have re-implemented the entire stat module in C. [1] It's a necessary > step to fix portability issues. For most constants POSIX standards > dictate only the name of the constant and its meaning but not its value. > Only the values of permission bits (0644 == rw-r--r--) have fixed values. > > For example S_IFDIR is usually 0o4 but it might be 0o2 on some > platforms. Common file types seem to have the same value on all > important platforms. It's the only reason we were able to get away with > stat.py. But uncommon file types like door, event port and whiteout > don't have sensible default values. The C implementation is able to pick > up the platform's values easily. > > What shall I do about stat.py? statmodule.c implements all features of > stat.py so there is no point in using it in CPython. It's one Python > file less to load on every startup. However it might still be useful to > alternative implementations of Python such as Jython or PyPy. > > 1) keep the file stat.py but let it be shadowed by the builtin stat > module. Antoine loathes my hack... > 2) rename stat.py to _stat.py > 3) remove stat.py > > Opinions? > > Christian > > [1] http://bugs.python.org/issue11016 > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/florent.xicluna%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
Am 20.06.2013 15:21, schrieb Florent: > we already have "_pyio.py", we could have "_pystat.py". Works for me. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page.
On Wed, Jun 19, 2013 at 11:20 PM, senthil.kumaran < [email protected]> wrote: > http://hg.python.org/cpython/rev/dfead0696a71 > changeset: 84224:dfead0696a71 > branch: 3.3 > parent: 84221:0113247f894b > user:Senthil Kumaran > date:Wed Jun 19 22:19:46 2013 -0500 > summary: > Add -b and -X options to python man page. > Patch contributed by Corey Brune. > > files: > Misc/python.man | 22 ++ > 1 files changed, 18 insertions(+), 4 deletions(-) > > > diff --git a/Misc/python.man b/Misc/python.man > --- a/Misc/python.man > +++ b/Misc/python.man > @@ -11,6 +11,9 @@ > .B \-B > ] > [ > +.B \-b > +] > +[ > .B \-d > ] > [ > @@ -23,14 +26,14 @@ > .B \-i > ] > [ > -.B \-m > +.B \-m > .I module-name > ] > -[ > -.B \-q > -] > .br > [ > +.B \-q > +] > +[ > .B \-O > ] > [ > @@ -60,6 +63,10 @@ > .B \-x > ] > [ > +[ > +.B \-X > +.I option > +] > .B \-? > ] > .br > @@ -105,6 +112,10 @@ > .I .py[co] > files on import. See also PYTHONDONTWRITEBYTECODE. > .TP > +.B \-b > +Issue warnings about str(bytes_instance), str(bytearray_instance) > +and comparing bytes/bytearray with str. (-bb: issue errors) > +.TP > .BI "\-c " command > Specify the command to execute (see next section). > This terminates the option list (following options are passed as > @@ -243,6 +254,9 @@ > field matches the line number, where zero matches all line numbers and > is thus equivalent to an omitted line number. > .TP > +.BI "\-X " option > +Set implementation specific option. > Should probably be "Set the implementation-specific option." > +.TP > .B \-x > Skip the first line of the source. This is intended for a DOS > specific hack only. Warning: the line numbers in error messages will > > -- > Repository URL: http://hg.python.org/cpython > > ___ > Python-checkins mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-checkins > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round)
On 20 June 2013 22:16, Antoine Pitrou wrote: > Le Thu, 20 Jun 2013 13:26:52 +0200, > Victor Stinner a écrit : > >> Hi, >> >> I changed the PEP 445 according to the discussing on python-dev. >> >> Read it online: >> http://www.python.org/dev/peps/pep-0445/ >> >> Changes: >> >> * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or >> PYALLOC_PYOBJECT > > PYMEM_DOMAIN_RAW? The challenge there is coming up with a good name for PYALLOC_PYMEM. I think Victor's names for the domains work well: - "PYALLOC_" prefix to say "this is an allocation domain" - "PYMEM_RAW" suffix for PyMem_RawMalloc/Realloc/Free - "PYMEM" suffix for PyMem_Malloc/Realloc/Free - "PYOBJECT" suffix for PyObject_Malloc/Realloc/Free Although, there's a copy-and-paste error in the domain definitions (2 *Reallocs in each entry and no *Frees). If Guido doesn't want to do it, I'm happy to. Antoine would be a good choice too, since he's the one who suggested taking the time to thrash out the API details in a PEP (which I agree has substantially improved the end result). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
On 20 June 2013 23:29, Christian Heimes wrote: > Am 20.06.2013 15:21, schrieb Florent: >> we already have "_pyio.py", we could have "_pystat.py". > > Works for me. I suggest following the guidelines in PEP 399 for cross implementation compatibility of the standard library: http://www.python.org/dev/peps/pep-0399/#details 1. Keep stat.py 2. Make the C version "_stat" 3. Add the following to the end of stat.py: try: from _stat import * except ImportError: pass Run the tests with and without the C module in the test suite (again, as per PEP 399). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
On 6/20/2013 9:40 AM, Nick Coghlan wrote: > On 20 June 2013 23:29, Christian Heimes wrote: >> Am 20.06.2013 15:21, schrieb Florent: >>> we already have "_pyio.py", we could have "_pystat.py". >> >> Works for me. > > I suggest following the guidelines in PEP 399 for cross implementation > compatibility of the standard library: > http://www.python.org/dev/peps/pep-0399/#details > > 1. Keep stat.py > 2. Make the C version "_stat" > 3. Add the following to the end of stat.py: > > try: > from _stat import * > except ImportError: > pass > > Run the tests with and without the C module in the test suite (again, > as per PEP 399). Agreed with the above. But isn't the real problem with this module in Python the fact that the constants might be wrong? I'm not sure what, if anything, we can do about that. -- Eric. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon wrote: > > > > On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith wrote: > >> On 6/20/2013 9:40 AM, Nick Coghlan wrote: >> > On 20 June 2013 23:29, Christian Heimes wrote: >> >> Am 20.06.2013 15:21, schrieb Florent: >> >>> we already have "_pyio.py", we could have "_pystat.py". >> >> >> >> Works for me. >> > >> > I suggest following the guidelines in PEP 399 for cross implementation >> > compatibility of the standard library: >> > http://www.python.org/dev/peps/pep-0399/#details >> > >> > 1. Keep stat.py >> > 2. Make the C version "_stat" >> > 3. Add the following to the end of stat.py: >> > >> > try: >> > from _stat import * >> > except ImportError: >> > pass >> > >> > Run the tests with and without the C module in the test suite (again, >> > as per PEP 399). >> >> Agreed with the above. >> >> But isn't the real problem with this module in Python the fact that the >> constants might be wrong? I'm not sure what, if anything, we can do >> about that. >> > > There isn't anything we can do beyond at least trying to provide > reasonable defaults when something better isn't available (which is what > the stats module has done all this time). It might make testing difficult > for the Python code when the C code has the right values, but I don't think > it's necessarily worth tossing out the Python code as backup. > If the .py file is going to be wrong or incomplete, why would we want to keep it -- or use it as fallback -- at all? If we're dead set on having a .py file instead of requiring it to be part of the interpreter (whichever that is, however it was built), it should be generated as part of the build process. Personally, I don't see the value in it; other implementations will need to do *something* special to use it anyway. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
2013/6/20 Thomas Wouters : > If the .py file is going to be wrong or incomplete, why would we want to > keep it -- or use it as fallback -- at all? If we're dead set on having a > .py file instead of requiring it to be part of the interpreter (whichever > that is, however it was built), it should be generated as part of the build > process. Personally, I don't see the value in it; other implementations will > need to do *something* special to use it anyway. That's exactly my rationale for pushing for removal. cf ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
cffi makes this kind of constant-grabbing very easy. However this is a tiny module so no problem with having a C version. On Thu, Jun 20, 2013 at 11:04 AM, Thomas Wouters wrote: > > > > On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon wrote: >> >> >> >> >> On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith >> wrote: >>> >>> On 6/20/2013 9:40 AM, Nick Coghlan wrote: >>> > On 20 June 2013 23:29, Christian Heimes wrote: >>> >> Am 20.06.2013 15:21, schrieb Florent: >>> >>> we already have "_pyio.py", we could have "_pystat.py". >>> >> >>> >> Works for me. >>> > >>> > I suggest following the guidelines in PEP 399 for cross implementation >>> > compatibility of the standard library: >>> > http://www.python.org/dev/peps/pep-0399/#details >>> > >>> > 1. Keep stat.py >>> > 2. Make the C version "_stat" >>> > 3. Add the following to the end of stat.py: >>> > >>> > try: >>> > from _stat import * >>> > except ImportError: >>> > pass >>> > >>> > Run the tests with and without the C module in the test suite (again, >>> > as per PEP 399). >>> >>> Agreed with the above. >>> >>> But isn't the real problem with this module in Python the fact that the >>> constants might be wrong? I'm not sure what, if anything, we can do >>> about that. >> >> >> There isn't anything we can do beyond at least trying to provide >> reasonable defaults when something better isn't available (which is what the >> stats module has done all this time). It might make testing difficult for >> the Python code when the C code has the right values, but I don't think it's >> necessarily worth tossing out the Python code as backup. > > > If the .py file is going to be wrong or incomplete, why would we want to > keep it -- or use it as fallback -- at all? If we're dead set on having a > .py file instead of requiring it to be part of the interpreter (whichever > that is, however it was built), it should be generated as part of the build > process. Personally, I don't see the value in it; other implementations will > need to do *something* special to use it anyway. > > -- > Thomas Wouters > > Hi! I'm an email virus! Think twice before sending your email to help me > spread! > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith wrote: > On 6/20/2013 9:40 AM, Nick Coghlan wrote: > > On 20 June 2013 23:29, Christian Heimes wrote: > >> Am 20.06.2013 15:21, schrieb Florent: > >>> we already have "_pyio.py", we could have "_pystat.py". > >> > >> Works for me. > > > > I suggest following the guidelines in PEP 399 for cross implementation > > compatibility of the standard library: > > http://www.python.org/dev/peps/pep-0399/#details > > > > 1. Keep stat.py > > 2. Make the C version "_stat" > > 3. Add the following to the end of stat.py: > > > > try: > > from _stat import * > > except ImportError: > > pass > > > > Run the tests with and without the C module in the test suite (again, > > as per PEP 399). > > Agreed with the above. > > But isn't the real problem with this module in Python the fact that the > constants might be wrong? I'm not sure what, if anything, we can do > about that. > There isn't anything we can do beyond at least trying to provide reasonable defaults when something better isn't available (which is what the stats module has done all this time). It might make testing difficult for the Python code when the C code has the right values, but I don't think it's necessarily worth tossing out the Python code as backup. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
2013/6/20 Charles-François Natali : > 2013/6/20 Thomas Wouters : >> If the .py file is going to be wrong or incomplete, why would we want to >> keep it -- or use it as fallback -- at all? If we're dead set on having a >> .py file instead of requiring it to be part of the interpreter (whichever >> that is, however it was built), it should be generated as part of the build >> process. Personally, I don't see the value in it; other implementations will >> need to do *something* special to use it anyway. > > That's exactly my rationale for pushing for removal. +1 to nixing it. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
Am 20.06.2013 17:35, schrieb Benjamin Peterson: > 2013/6/20 Charles-François Natali : >> 2013/6/20 Thomas Wouters : >>> If the .py file is going to be wrong or incomplete, why would we want to >>> keep it -- or use it as fallback -- at all? If we're dead set on having a >>> .py file instead of requiring it to be part of the interpreter (whichever >>> that is, however it was built), it should be generated as part of the build >>> process. Personally, I don't see the value in it; other implementations will >>> need to do *something* special to use it anyway. >> >> That's exactly my rationale for pushing for removal. > > +1 to nixing it. I'm +0 for removal. The stat module contains merely constants, wrappers for macros and one simple functions. Somebody just say the word. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
20.06.13 16:05, Christian Heimes написав(ла): I have re-implemented the entire stat module in C. [1] It's a necessary step to fix portability issues. For most constants POSIX standards dictate only the name of the constant and its meaning but not its value. Only the values of permission bits (0644 == rw-r--r--) have fixed values. For example S_IFDIR is usually 0o4 but it might be 0o2 on some platforms. Common file types seem to have the same value on all important platforms. It's the only reason we were able to get away with stat.py. But uncommon file types like door, event port and whiteout don't have sensible default values. The C implementation is able to pick up the platform's values easily. What shall I do about stat.py? statmodule.c implements all features of stat.py so there is no point in using it in CPython. It's one Python file less to load on every startup. However it might still be useful to alternative implementations of Python such as Jython or PyPy. 1) keep the file stat.py but let it be shadowed by the builtin stat module. Antoine loathes my hack... 2) rename stat.py to _stat.py 3) remove stat.py Opinions? Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] How about we extend C3 to support ABCs?
Review and comments on the ticket needed, please. http://bugs.python.org/issue18244#msg191535 -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
2013/6/20 Serhiy Storchaka > Now with enumerations in the stdlib the stat module constants are > candidates for flag enumerations. How easy will be implement it on C? Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
But aren't most of these masks? enum,IntEnum doesn't handle those very gracefully (the | operator returns a plain int). On Thu, Jun 20, 2013 at 1:18 PM, Amaury Forgeot d'Arc wrote: > 2013/6/20 Serhiy Storchaka >> >> Now with enumerations in the stdlib the stat module constants are >> candidates for flag enumerations. How easy will be implement it on C? > > > Aha. Should an internal C module fetch the value of the constants, and a > public stat.py nicely wrap them in enums? > > > -- > Amaury Forgeot d'Arc > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
On 06/20/2013 01:18 PM, Amaury Forgeot d'Arc wrote: 2013/6/20 Serhiy Storchaka wrote: Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C? Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums? That's pretty much the approach I'm thinking about for sockets. -- ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] stat module in C -- what to do with stat.py?
2013/6/20 Eric V. Smith >: > But isn't the real problem with this module in Python the fact that the > constants might be wrong? I'm not sure what, if anything, we can do > about that. Python is providing a stat module implemented in Python since 10 years, or maybe 20 years, and I don't remember that someone complained that constants are wrong. At the same time, Python distributes IN.py which contains things like that: LONG_MAX = 9223372036854775807 LONG_MAX = 2147483647 LONG_MIN = (-LONG_MAX - 1) In my opinion, the situation of plat-*/*.py modules is worse than stat.py :-) Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round)
2013/6/20 Antoine Pitrou >: > Victor Stinner > a écrit : >> Changes: >> >> * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or >> PYALLOC_PYOBJECT > > PYMEM_DOMAIN_RAW? I prefer your suggestion because it shares the PYMEM/PyMem prefix with PyMem_SetAllocator(). So the whole list would be: - PYMEM_DOMAIN_RAW - PYMEM_DOMAIN_MEM - PYMEM_DOMAIN_OBJ Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
On 6/20/2013 4:53 PM, Victor Stinner wrote: > 2013/6/20 Eric V. Smith >: >> But isn't the real problem with this module in Python the fact that the >> constants might be wrong? I'm not sure what, if anything, we can do >> about that. > > Python is providing a stat module implemented in Python since 10 years, > or maybe 20 years, and I don't remember that someone complained that > constants are wrong. This is serious, not argumentative: If there's really no concern that the values be correct, then why implement it in C? Let's just leave the python version with the hard-coded, usually-correct values. My personal opinion is that having the correct values in C is the way to go, and delete stat.py. Eric. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
2013/6/20 Eric V. Smith : > This is serious, not argumentative: If there's really no concern that > the values be correct, then why implement it in C? I read again the issue. The problem is to add new flags. Current flags (type: block device/symlink/..., file mode) are well defined and portable, whereas new flags are usually specific to an OS and no standardize yet. Examples: - Solaris: "door" - Solaris: "event port" - OSX/Darwin/FreeBSD: "whiteout" The other problem is that values of the new flags are not portable. Martin v. Löwis wrote: "Looking in more detail: for the S_IFMT flags, OSX/Darwin/FreeBSD defines 0xe000 as S_IFWHT (whiteout), but Solaris defines it as S_IFPORT (event port)." A C module reusing existing S_ISxxx() macros is the obvious way to solve this issue. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
On 06/20/2013 01:27 PM, Guido van Rossum wrote: On Thu, Jun 20, 2013 at 1:18 PM, Amaury Forgeot d'Arc wrote: 2013/6/20 Serhiy Storchaka wrote: Now with enumerations in the stdlib the stat module constants are candidates for flag enumerations. How easy will be implement it on C? Aha. Should an internal C module fetch the value of the constants, and a public stat.py nicely wrap them in enums? But aren't most of these masks? enum.IntEnum doesn't handle those very gracefully (the | operator returns a plain int). As is, IntEnum would not be a great choice. We could either add a another Enum (BitMaskEnum?) that was also int-based, or flesh out IntEnum with enough smarts to handle |, &, and ^ and return an IntEnum when possible and fall back to a plain int when not possible. -- ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
2013/6/20 Serhiy Storchaka :
> Now with enumerations in the stdlib the stat module constants are candidates
> for flag enumerations. How easy will be implement it on C?
Numerical values are less important than S_ISxxx() macros. Example:
#define S_ISDOOR(mode) (((mode)&0xF000) == 0xd000)
0xd000 is (stat.S_IFSOCK + stat.S_IFIFO).
And how do you represent the file mode with enums? I don't think that
enum should be used in the stat module.
I would prefer a stat object with methods than having to calls
low-level functions. Something like:
os.stat("document.txt").st_mode.is_reg()
versus
stat.S_ISREG(os.stat("document.txt").st_mode)
The idea was discussed in http://bugs.python.org/issue11406 to solve a
real API design issue. How should os.scandir() return the "is a
directory" information with a portable API.
I'm not saying that stat.S_ISREG should go away. The two approaches
are complementary.
Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
On 21 June 2013 01:04, Thomas Wouters wrote: > If the .py file is going to be wrong or incomplete, why would we want to > keep it -- or use it as fallback -- at all? If we're dead set on having a > .py file instead of requiring it to be part of the interpreter (whichever > that is, however it was built), it should be generated as part of the build > process. Personally, I don't see the value in it; other implementations will > need to do *something* special to use it anyway. Because practicality beats purity. This "wrong" Python code has been good enough for all Python version up until 3.4, it makes sense to keep it as a fallback instead of throwing it away. As Daniel notes, it also means PyPy can just have a small cffi module that adds (or overwrites) the platform specific constants, instead of having to reimplement the whole module. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
