Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Antoine Pitrou 
On Sat, 24 Aug 2013 15:57:50 +1000
Nick Coghlan  wrote:
> On 24 August 2013 15:32, Stefan Behnel  wrote:
> > So, to put it more nicely, I think this feature was added without the
> > amount of review that it needs, and now that I've given it that review, I'm
> > asking for removal of the feature and a proper redesign that fits into the
> > existing library.
> 
> FWIW, it seems to me that this is something that could live in *tulip*
> as an adapter between the tulip data processing APIs and the existing
> ElementTree incremental parsing APIs, without needing to be added
> directly to xml.etree at all.

This is not an adapter, it's a new feature that ElementTree wasn't
providing before. The need to process data in a non-blocking way
(not merely incremental) didn't appear with Tulip, and the fact that the
API is *inspired* by Tulip doesn't make it Tulip-specific.

(I'm also curious why Tulip would start providing data-processing APIs:
until now, I thought it's supposed to be a networking library :-))

Furthermore, such a feature has to access implementation details of
ElementTree, so it's only natural that it be provided in ElementTree.

By the way, just know that Stefan tried to provide a patch that would
better suit his API desires, and failed because ElementTree's current
implementation makes it difficult to do so.

Someone can take the whole thing over if they want to, change the API
and make it more shiny or different, tweak the implementation to suit
it better to their own aesthetic sensibilities, but please don't revert
an useful feature unless it's based on concrete, serious issues rather
than a platonic disagreement about design.

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan 
On 24 August 2013 15:51, Nick Coghlan  wrote:
> My current plan is to create an experimental prototype of this
> approach this weekend. That will include stdlib test cases, so it will
> also show how it looks from the extension developer's point of view.

I prototyped as much as I could without PEP 451's ModuleSpec support here:


https://bitbucket.org/ncoghlan/cpython_sandbox/commits/branch/new_extension_imports

On systems that use dynload_shlib (at least Linux & the BSDs), this
branch allows extension modules to be imported if they provide a
PyImportExec_NAME hook. The new hook is preferred to the existing
PyInit_NAME hook, so extension modules using the stable ABI can
provide both and degrade to the legacy initialisation API on older
versions of Python.

The PyImportExec hook is called with a pre-created module object that
the hook is then expected to populate. To aid in this task, I added
two new APIs:

PyModule_SetDocString
PyModule_AddFunctions

These cover setting the docstring and adding module level functions,
tasks that are handled through the PyModule_Create API when using the
PyInit_NAME style hook.

The _testimportexec.c module was derived from the existing example
xxlimited.c module, with a few name changes. The main functional
difference is that _testimportexec uses the new API, so the module
object is created externally and passed in to the API, rather than
being created by the extension module. The effect of this can be seen
in the test suite, where ImportExecTests.test_fresh_module shows that
loading the module twice will create two *different* modules, unlike
the legacy API.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Antoine Pitrou 
On Sat, 24 Aug 2013 21:36:51 +1000
Nick Coghlan  wrote:
> On 24 August 2013 15:51, Nick Coghlan  wrote:
> > My current plan is to create an experimental prototype of this
> > approach this weekend. That will include stdlib test cases, so it will
> > also show how it looks from the extension developer's point of view.
> 
> I prototyped as much as I could without PEP 451's ModuleSpec support here:
> 
> 
> https://bitbucket.org/ncoghlan/cpython_sandbox/commits/branch/new_extension_imports
> 
> On systems that use dynload_shlib (at least Linux & the BSDs), this
> branch allows extension modules to be imported if they provide a
> PyImportExec_NAME hook. The new hook is preferred to the existing
> PyInit_NAME hook, so extension modules using the stable ABI can
> provide both and degrade to the legacy initialisation API on older
> versions of Python.
> 
> The PyImportExec hook is called with a pre-created module object that
> the hook is then expected to populate. To aid in this task, I added
> two new APIs:
> 
> PyModule_SetDocString
> PyModule_AddFunctions

I was thinking about something like PyType_FromSpec, only specialized
for module subclasses to make it easier to declare them (e.g.
PyModuleType_FromSpec).

This would also imply extension module have to be subclasses of the
built-in module type. They can't be arbitrary objects like Stefan
proposed. I'm not sure what the latter enables, but it would probably
make things more difficult internally.

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Stefan Behnel 
Antoine Pitrou, 24.08.2013 12:58:
> By the way, just know that Stefan tried to provide a patch that would
> better suit his API desires, and failed because ElementTree's current
> implementation makes it difficult to do so.

Absolutely. I agree that your current implementation is a hack that works
around these issues. That doesn't mean that they go away, though.

And yes, I even provided a half-finished implementation, even though I'm
certainly not interested enough in this feature to make much use of it. Why
don't you just take a look at my patch and finish it up?

Given that you are apparently the most ambitious supporter of this feature,
I would expect you to provide an appropriate implementation and have it
reviewed. And with "reviewed" I also mean "accept criticism". Just because
you managed to sneak in a hack doesn't mean it has to stay there once it's
uncovered.


> Someone can take the whole thing over if they want to, change the API
> and make it more shiny or different, tweak the implementation to suit
> it better to their own aesthetic sensibilities, but please don't revert
> an useful feature unless it's based on concrete, serious issues rather
> than a platonic disagreement about design.

As I said, the only reason why the current implementation is there is
"because it's there". The problems of the current iterparse implementation
should not be taken as a reason for a design decision of a new feature.
Instead, they should be fixed and the feature should be based on these fixes.

Yes, that's more work than adding a hack. But I'm sure that cleaning up
first will pay off quite quickly. I already gave lots of reasons for that.

Stefan


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel 
Antoine Pitrou, 24.08.2013 13:53:
> This would also imply extension module have to be subclasses of the
> built-in module type. They can't be arbitrary objects like Stefan
> proposed. I'm not sure what the latter enables, but it would probably
> make things more difficult internally.

My line of thought was more like: if Python code can stick anything into
sys.modules and the runtime doesn't care, why can't extension modules stick
anything into sys.modules as well?

I can't really see the advantage of requiring a subtype here. Or even just
a type, as I said.

I guess we'll have to start using this in real code to see if it makes any
difference.

Stefan


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Antoine Pitrou 
On Sat, 24 Aug 2013 14:46:32 +0200
Stefan Behnel  wrote:
> 
> As I said, the only reason why the current implementation is there is
> "because it's there".

No. It works, it's functional, it fills an use case, and it doesn't seem
to have any concrete issues.

Get over it, Stefan, and stop trolling us.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Antoine Pitrou 
On Sat, 24 Aug 2013 14:51:42 +0200
Stefan Behnel  wrote:
> Antoine Pitrou, 24.08.2013 13:53:
> > This would also imply extension module have to be subclasses of the
> > built-in module type. They can't be arbitrary objects like Stefan
> > proposed. I'm not sure what the latter enables, but it would probably
> > make things more difficult internally.
> 
> My line of thought was more like: if Python code can stick anything into
> sys.modules and the runtime doesn't care, why can't extension modules stick
> anything into sys.modules as well?
> 
> I can't really see the advantage of requiring a subtype here. Or even just
> a type, as I said.

sys.modules doesn't care indeed. There's still the whole
extension-specific code, though, i.e. the eternal PyModuleDef store
and the state management routines. How much of it would remain with
your proposal?

Regards

Antoine.


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel 
Antoine Pitrou, 24.08.2013 15:00:
> On Sat, 24 Aug 2013 14:51:42 +0200
> Stefan Behnel wrote:
>> Antoine Pitrou, 24.08.2013 13:53:
>>> This would also imply extension module have to be subclasses of the
>>> built-in module type. They can't be arbitrary objects like Stefan
>>> proposed. I'm not sure what the latter enables, but it would probably
>>> make things more difficult internally.
>>
>> My line of thought was more like: if Python code can stick anything into
>> sys.modules and the runtime doesn't care, why can't extension modules stick
>> anything into sys.modules as well?
>>
>> I can't really see the advantage of requiring a subtype here. Or even just
>> a type, as I said.
> 
> sys.modules doesn't care indeed. There's still the whole
> extension-specific code, though, i.e. the eternal PyModuleDef store
> and the state management routines. How much of it would remain with
> your proposal?

PEP 3121 would no longer be necessary. Extension types can do all we need.
No more special casing of modules, that was the idea.

Stefan


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Stefan Behnel 
Antoine Pitrou, 24.08.2013 14:53:
> it doesn't seem to have any concrete issues.

I don't consider closing your eyes and ignoring the obvious a good strategy
for software design.

Stefan


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread R. David Murray 
On Sat, 24 Aug 2013 14:53:13 +0200, Antoine Pitrou  wrote:
> On Sat, 24 Aug 2013 14:46:32 +0200
> Stefan Behnel  wrote:
> > 
> > As I said, the only reason why the current implementation is there is
> > "because it's there".
> 
> No. It works, it's functional, it fills an use case, and it doesn't seem
> to have any concrete issues.
> 
> Get over it, Stefan, and stop trolling us.

Stefan is not trolling.  He's raising objections that you disagree with.
It costs nothing to keep the discussion civil.

--David
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel 
Nick Coghlan, 24.08.2013 13:36:
> On 24 August 2013 15:51, Nick Coghlan wrote:
>> My current plan is to create an experimental prototype of this
>> approach this weekend. That will include stdlib test cases, so it will
>> also show how it looks from the extension developer's point of view.
> 
> I prototyped as much as I could without PEP 451's ModuleSpec support here:
> 
> 
> https://bitbucket.org/ncoghlan/cpython_sandbox/commits/branch/new_extension_imports

Cool. I'll take a look.


> On systems that use dynload_shlib (at least Linux & the BSDs), this
> branch allows extension modules to be imported if they provide a
> PyImportExec_NAME hook. The new hook is preferred to the existing
> PyInit_NAME hook, so extension modules using the stable ABI can
> provide both and degrade to the legacy initialisation API on older
> versions of Python.

Hmm, right, good call. Since both init schemes have to be part of the
stable ABI, we can's rely on people compiling out one or the other. So
using the old one as a fallback should work. However, only actual usage in
code will tell us how it feels on user side. Supporting both in the same
binary will most likely complicate things quite a bit.


> The PyImportExec hook is called with a pre-created module object that
> the hook is then expected to populate. To aid in this task, I added
> two new APIs:
> 
> PyModule_SetDocString
> PyModule_AddFunctions
> 
> These cover setting the docstring and adding module level functions,
> tasks that are handled through the PyModule_Create API when using the
> PyInit_NAME style hook.

What are those needed for? If you subtype the module type, or provide an
arbitrary extension type as implementation, you'd get these for free,
wouldn't you? It's in no way different from setting up an extension type.


> The _testimportexec.c module

Where can I find that module?


> was derived from the existing example
> xxlimited.c module, with a few name changes. The main functional
> difference is that _testimportexec uses the new API, so the module
> object is created externally and passed in to the API, rather than
> being created by the extension module. The effect of this can be seen
> in the test suite, where ImportExecTests.test_fresh_module shows that
> loading the module twice will create two *different* modules, unlike
> the legacy API.

Stefan


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Nick Coghlan 
On 24 August 2013 20:58, Antoine Pitrou  wrote:
> Someone can take the whole thing over if they want to, change the API
> and make it more shiny or different, tweak the implementation to suit
> it better to their own aesthetic sensibilities, but please don't revert
> an useful feature unless it's based on concrete, serious issues rather
> than a platonic disagreement about design.

While "It's a useful feature" is a necessary criterion for adding
something to the standard library, it has never been a *sufficient*
criterion. There's a lot more to take into account when judging the
latter, and one of the big ones if "There should be one obvious way to
do it".

Looking at the current documentation of ElementTree sets of alarm
bells on that front, as it contains the following method descriptions
for XMLParser:

close()
Finishes feeding data to the parser. Returns an element structure.

feed(data)
Feeds data to the parser. data is encoded data.

And these for IncrementalParser:

data_received(data)

Feed the given bytes data to the incremental parser.

eof_received()

Signal the incremental parser that the data stream is terminated.

events()

Iterate over the events which have been encountered in the
data fed to the parser. This method yields (event, elem) pairs, where
event is a string representing the type of event (e.g. "end") and elem
is the encountered Element object. Events provided in a previous call
to events() will not be yielded again.

It is thoroughly unclear to me as a user of the module how and why one
would use the new IncrementalParser API over the existing incremental
XMLParser API. If there is some defect in the XMLParser API that
prevents it from interoperating correctly with asynchronous code, then
*that is a bug to be fixed*, rather than avoided by adding a whole new
parallel API.

If Stefan's "please revert this" as lxml.etree maintainer isn't
enough, then I'm happy to add a "please revert this" as a core
committer that is confused about how and when the new tulip-inspired
incremental parsing API should be used in preference to the existing
incremental parsing API, and believes this needs to be clearly
resolved before adding a second way to do it (especially if there's a
possibility of using a different implementation strategy that avoids
adding the second way).

Regards,
Nick.

--
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Antoine Pitrou 
On Sun, 25 Aug 2013 00:03:01 +1000
Nick Coghlan  wrote:
> If Stefan's "please revert this" as lxml.etree maintainer isn't
> enough, then I'm happy to add a "please revert this" as a core
> committer that is confused about how and when the new tulip-inspired
> incremental parsing API should be used in preference to the existing
> incremental parsing API, and believes this needs to be clearly
> resolved before adding a second way to do it
> (especially if there's a
> possibility of using a different implementation strategy that avoids
> adding the second way).

To be clear, again: anyone who wants to "see it resolved" can take over
the issue and handle it by themselves. I'm done with it.

Regards

Antoine.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan 
On 24 August 2013 23:19, Stefan Behnel  wrote:
> Nick Coghlan, 24.08.2013 13:36:
>> On 24 August 2013 15:51, Nick Coghlan wrote:
>>> My current plan is to create an experimental prototype of this
>>> approach this weekend. That will include stdlib test cases, so it will
>>> also show how it looks from the extension developer's point of view.
>>
>> I prototyped as much as I could without PEP 451's ModuleSpec support here:
>>
>> 
>> https://bitbucket.org/ncoghlan/cpython_sandbox/commits/branch/new_extension_imports
>
> Cool. I'll take a look.

The new _PyImport_CreateAndExecExtensionModule function does the heavy lifting:


https://bitbucket.org/ncoghlan/cpython_sandbox/src/081f8f7e3ee27dc309463b48e6c67cf4880fca12/Python/importdl.c?at=new_extension_imports#cl-65

One key point to note is that it *doesn't* call
_PyImport_FixupExtensionObject, which is the API that handles all the
PEP 3121 per-module state stuff. Instead, the idea will be for modules
that don't need additional C level state to just implement
PyImportExec_NAME, while those that *do* need C level state implement
PyImportCreate_NAME and return a custom object (which may or may not
be a module subtype). Such modules can still support reloading (e.g.
to pick up reloaded or removed module dependencies) by providing
PyImportExec_NAME as well.

(in a PEP 451 world, this would likely be split up as two separate
functions, one for create, one for exec)

>> On systems that use dynload_shlib (at least Linux & the BSDs), this
>> branch allows extension modules to be imported if they provide a
>> PyImportExec_NAME hook. The new hook is preferred to the existing
>> PyInit_NAME hook, so extension modules using the stable ABI can
>> provide both and degrade to the legacy initialisation API on older
>> versions of Python.
>
> Hmm, right, good call. Since both init schemes have to be part of the
> stable ABI, we can's rely on people compiling out one or the other. So
> using the old one as a fallback should work. However, only actual usage in
> code will tell us how it feels on user side. Supporting both in the same
> binary will most likely complicate things quite a bit.

It shouldn't be too bad - the PyInit_NAME fallback would just need to
do the equivalent of calling PyImportCreate_NAME (or PyModule_Create
if not using a custom object), call PyImportExec_NAME on it, and then
return the result.

Modules that genuinely *needed* the new behaviour wouldn't be able to
provide a sensible fallback, and would thus be limited to Python 3.4+

>> The PyImportExec hook is called with a pre-created module object that
>> the hook is then expected to populate. To aid in this task, I added
>> two new APIs:
>>
>> PyModule_SetDocString
>> PyModule_AddFunctions
>>
>> These cover setting the docstring and adding module level functions,
>> tasks that are handled through the PyModule_Create API when using the
>> PyInit_NAME style hook.
>
> What are those needed for? If you subtype the module type, or provide an
> arbitrary extension type as implementation, you'd get these for free,
> wouldn't you? It's in no way different from setting up an extension type.

The idea is to let people use an import system provided module object
if they don't define a custom PyImportCreate_NAME hook.

Setting the docstring and adding module level functions were the two
things that PyModule_Create previously handled neatly through the
Py_ModuleDef struct. The two new API functions just break out those
subsets as separate operations to call on the import system provided
module.

>> The _testimportexec.c module
>
> Where can I find that module?

Oops, forgot to add it to the repo. Uploaded now:

https://bitbucket.org/ncoghlan/cpython_sandbox/src/081f8f7e3ee27dc309463b48e6c67cf4880fca12/Modules/_testimportexec.c?at=new_extension_imports

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Nick Coghlan 
On 25 August 2013 00:13, Antoine Pitrou  wrote:
> On Sun, 25 Aug 2013 00:03:01 +1000
> Nick Coghlan  wrote:
>> If Stefan's "please revert this" as lxml.etree maintainer isn't
>> enough, then I'm happy to add a "please revert this" as a core
>> committer that is confused about how and when the new tulip-inspired
>> incremental parsing API should be used in preference to the existing
>> incremental parsing API, and believes this needs to be clearly
>> resolved before adding a second way to do it
>> (especially if there's a
>> possibility of using a different implementation strategy that avoids
>> adding the second way).
>
> To be clear, again: anyone who wants to "see it resolved" can take over
> the issue and handle it by themselves. I'm done with it.

OK, I'll revert it for now, then. If someone else steps up to resolve
the API duplication problem, cool, otherwise we can continue to live
without this as a standard library feature.

Regards,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Nick Coghlan 
On 25 August 2013 00:26, Nick Coghlan  wrote:
> On 25 August 2013 00:13, Antoine Pitrou  wrote:
>> On Sun, 25 Aug 2013 00:03:01 +1000
>> Nick Coghlan  wrote:
>>> If Stefan's "please revert this" as lxml.etree maintainer isn't
>>> enough, then I'm happy to add a "please revert this" as a core
>>> committer that is confused about how and when the new tulip-inspired
>>> incremental parsing API should be used in preference to the existing
>>> incremental parsing API, and believes this needs to be clearly
>>> resolved before adding a second way to do it
>>> (especially if there's a
>>> possibility of using a different implementation strategy that avoids
>>> adding the second way).
>>
>> To be clear, again: anyone who wants to "see it resolved" can take over
>> the issue and handle it by themselves. I'm done with it.
>
> OK, I'll revert it for now, then. If someone else steps up to resolve
> the API duplication problem, cool, otherwise we can continue to live
> without this as a standard library feature.

On the other hand... because other changes have been made to the
module since the original commit, a simple "hg backout" is no longer
possible :(

Stefan - if you'd like this reverted, you're going to have to either
make the alternative solution work correctly, or else craft the commit
to undo the API addition.

However, I have at least reopened http://bugs.python.org/issue17741

Regards,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Eli Bendersky 
On Sat, Aug 24, 2013 at 7:33 AM, Nick Coghlan  wrote:

> On 25 August 2013 00:26, Nick Coghlan  wrote:
> > On 25 August 2013 00:13, Antoine Pitrou  wrote:
> >> On Sun, 25 Aug 2013 00:03:01 +1000
> >> Nick Coghlan  wrote:
> >>> If Stefan's "please revert this" as lxml.etree maintainer isn't
> >>> enough, then I'm happy to add a "please revert this" as a core
> >>> committer that is confused about how and when the new tulip-inspired
> >>> incremental parsing API should be used in preference to the existing
> >>> incremental parsing API, and believes this needs to be clearly
> >>> resolved before adding a second way to do it
> >>> (especially if there's a
> >>> possibility of using a different implementation strategy that avoids
> >>> adding the second way).
> >>
> >> To be clear, again: anyone who wants to "see it resolved" can take over
> >> the issue and handle it by themselves. I'm done with it.
> >
> > OK, I'll revert it for now, then. If someone else steps up to resolve
> > the API duplication problem, cool, otherwise we can continue to live
> > without this as a standard library feature.
>
> On the other hand... because other changes have been made to the
> module since the original commit, a simple "hg backout" is no longer
> possible :(
>
> Stefan - if you'd like this reverted, you're going to have to either
> make the alternative solution work correctly, or else craft the commit
> to undo the API addition.
>
> However, I have at least reopened http://bugs.python.org/issue17741
>

Let's please keep the discussion calm and civil, everyone, and keep things
in proportion. This is precisely what alpha releases are for - we have time
before beta (Nov 24) to tweak the API. It's a fairly minor feature that
*does* appear useful. I agree it would be nice to find an API that's
acceptable for more developers.

I'll try to find time to review this again, and others are free to do so
too.

Eli
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel 
Nick Coghlan, 24.08.2013 16:22:
> On 24 August 2013 23:19, Stefan Behnel wrote:
>> Nick Coghlan, 24.08.2013 13:36:
>>> On 24 August 2013 15:51, Nick Coghlan wrote:
 My current plan is to create an experimental prototype of this
 approach this weekend. That will include stdlib test cases, so it will
 also show how it looks from the extension developer's point of view.
>>>
>>> I prototyped as much as I could without PEP 451's ModuleSpec support here:
>>>
>>> 
>>> https://bitbucket.org/ncoghlan/cpython_sandbox/commits/branch/new_extension_imports
>>
>> Cool. I'll take a look.
> 
> The new _PyImport_CreateAndExecExtensionModule function does the heavy 
> lifting:
> 
> 
> https://bitbucket.org/ncoghlan/cpython_sandbox/src/081f8f7e3ee27dc309463b48e6c67cf4880fca12/Python/importdl.c?at=new_extension_imports#cl-65
> 
> One key point to note is that it *doesn't* call
> _PyImport_FixupExtensionObject, which is the API that handles all the
> PEP 3121 per-module state stuff. Instead, the idea will be for modules
> that don't need additional C level state to just implement
> PyImportExec_NAME, while those that *do* need C level state implement
> PyImportCreate_NAME and return a custom object (which may or may not
> be a module subtype).

Is it really a common case for an extension module not to need any C level
state at all? I mean, this might work for very simple accelerator modules
with only a few stand-alone functions. But anything non-trivial will almost
certainly have some kind of global state, cache, external library, etc.,
and that state is best stored at the C level for safety reasons.


> Such modules can still support reloading (e.g.
> to pick up reloaded or removed module dependencies) by providing
> PyImportExec_NAME as well.
> 
> (in a PEP 451 world, this would likely be split up as two separate
> functions, one for create, one for exec)

Can't we just always require extension modules to implement their own type?
Sure, it's a lot of boiler plate code, but that could be handled by a
simple C code generator or maybe even a copy&paste example in the docs. I
would like to avoid making it too easy for users in the future to get
anything wrong with reloading or sub-interpreters. Most people won't test
these things for their own code and the harder it is to make them not work,
the more likely it is that a given set of dependencies will properly work
in a sub-interpreter.

If users are required to implement their own type, I think it would be more
obvious where to put global module state, how to define functions (i.e.
module methods), how to handle garbage collection at the global module
level, etc.


>>> On systems that use dynload_shlib (at least Linux & the BSDs), this
>>> branch allows extension modules to be imported if they provide a
>>> PyImportExec_NAME hook. The new hook is preferred to the existing
>>> PyInit_NAME hook, so extension modules using the stable ABI can
>>> provide both and degrade to the legacy initialisation API on older
>>> versions of Python.
>>
>> Hmm, right, good call. Since both init schemes have to be part of the
>> stable ABI, we can's rely on people compiling out one or the other. So
>> using the old one as a fallback should work. However, only actual usage in
>> code will tell us how it feels on user side. Supporting both in the same
>> binary will most likely complicate things quite a bit.
> 
> It shouldn't be too bad - the PyInit_NAME fallback would just need to
> do the equivalent of calling PyImportCreate_NAME (or PyModule_Create
> if not using a custom object), call PyImportExec_NAME on it, and then
> return the result.
> 
> Modules that genuinely *needed* the new behaviour wouldn't be able to
> provide a sensible fallback, and would thus be limited to Python 3.4+

Right. I only saw it from the POV of Cython, which *will* have to support
both, and *will* use the new feature in Py3.4+. No idea how that is going
to work, but we've found so many tricks and work-arounds in the past that
I'm sure it'll work somehow. Module level properties are just way too
tempting not to make use of them.

Stefan


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Terry Reedy 

On 8/24/2013 10:03 AM, Nick Coghlan wrote:

I have not used ET or equivalent, but I do have opinions on function names.


Looking at the current documentation of ElementTree sets of alarm
bells on that front, as it contains the following method descriptions
for XMLParser:

 close()
 Finishes feeding data to the parser. Returns an element structure.

 feed(data)
 Feeds data to the parser. data is encoded data.


These are short active verbs, reused from other Python contexts.


And these for IncrementalParser:

 data_received(data)
 Feed the given bytes data to the incremental parser.


Longer, awkward, and to me ugly in comparison to 'feed'. Since it seems 
to mean more or less the same thing, why not reuse 'feed' and continue 
to build on people prior knowledge of Python? Is this the 'tulip 
inspired' part? If so, I hope the names are not set in stone yet.



 eof_received()
 Signal the incremental parser that the data stream is terminated.


What is the incremental parser supposed to do with the information? 
Close ;-?



 events()
 Iterate over the events which have been encountered in the
data fed to the parser. This method yields (event, elem) pairs, where
event is a string representing the type of event (e.g. "end") and elem
is the encountered Element object. Events provided in a previous call
to events() will not be yielded again.


Plural nouns work well as iterator names: 'for event in events:'.

--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of 3.2 in Hg repository?

2013-08-24 Thread Georg Brandl 
Am 21.08.2013 21:26, schrieb Brett Cannon:
> 
> 
> 
> On Wed, Aug 21, 2013 at 2:22 PM, Tim Peters  > wrote:
> 
> [Tim, wondering why the 3.2 branch isn't "inactive"]
> >> ...
> >> So let's try a different question ;-)  Would anyone _object_ to
> >> completing the process described in the docs:  merge 3.2 into 3.3,
> >> then merge 3.3 into default?  I'd be happy to do that.  I'd throw away
> >> all the merge changes except for adding the v3,2.5 tag to .hgtags.
> >>
> >> The only active branches remaining would be `default` and 2.7, which
> >> is what I expected when I started this ;-)
> 
> [Brett Cannon]
> > While I would think Georg can object if he wants, I see no reason to 
> help
> > visibly shutter the 3.2 branch by doing null merges. It isn't like it 
> makes
> > using hg harder or the history harder to read.
> 
> Well, why do we _ever_ do a null merge?  Then why don't the reasons
> apply in this case?
> 
> 
> After reading that sentence I realize there is a key "not" missing: "I see no
> reason NOT to help visibly shutter the 3.2. branch ...". IOW I say do the null
> merge. Sorry about that.

FWIW I have no real objections, I just don't see the gain.

Georg

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Terry Reedy 

On 8/24/2013 8:51 AM, Stefan Behnel wrote:

Antoine Pitrou, 24.08.2013 13:53:

This would also imply extension module have to be subclasses of the
built-in module type. They can't be arbitrary objects like Stefan
proposed. I'm not sure what the latter enables, but it would probably
make things more difficult internally.


My line of thought was more like: if Python code can stick anything into
sys.modules and the runtime doesn't care, why can't extension modules stick
anything into sys.modules as well?


Being able to stick anything in sys.modules in CPython is an 
implementation artifact rather than language feature.


"sys.modules
This is a dictionary that maps module names to modules which have 
already been loaded."


This implies to me that an implementation could use a dict subclass (or 
subtype if you prefer) that checks that keys are names and values 
ModuleType instances (or None).


"This can be manipulated to force reloading of modules and other tricks."

I guess this refers to the undocumented (at least here) option of None 
as a signal value.



I can't really see the advantage of requiring a subtype here. Or even just
a type, as I said.


A 'module' has to work with the import machinery and user code. I would 
ask, "What is the advantage of loosening the current spec?" (Or 
reinterpreting 'module', if you prefer.)  Loosening is hard to undo once 
done.


--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Benjamin Peterson 
2013/8/24 Terry Reedy :
> On 8/24/2013 8:51 AM, Stefan Behnel wrote:
>>
>> Antoine Pitrou, 24.08.2013 13:53:
>>>
>>> This would also imply extension module have to be subclasses of the
>>> built-in module type. They can't be arbitrary objects like Stefan
>>> proposed. I'm not sure what the latter enables, but it would probably
>>> make things more difficult internally.
>>
>>
>> My line of thought was more like: if Python code can stick anything into
>> sys.modules and the runtime doesn't care, why can't extension modules
>> stick
>> anything into sys.modules as well?
>
>
> Being able to stick anything in sys.modules in CPython is an implementation
> artifact rather than language feature.

This is not really true. Many people use this feature to replace
modules as they are being imported with other things.

-- 
Regards,
Benjamin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of 3.2 in Hg repository?

2013-08-24 Thread Tim Peters 
[Tim, wondering why the 3.2 branch isn't "inactive"]

[Georg Brandl]
> FWIW I have no real objections, I just don't see the gain.

I'm glad it's OK!  Especially because it's already been done ;-)

Two gains:

1. "hg branches" output now matches what the developer docs imply it
should be.  It didn't before.

2. If a security fix needs to made to 3.2, it will be much easier to
forward-merge it to the 3.3 and default branches now (the merges won't
suck in a pile of ancient, and unwanted, irrelevant-to-the-fix
changes).
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of 3.2 in Hg repository?

2013-08-24 Thread Georg Brandl 
Am 24.08.2013 22:38, schrieb Tim Peters:
> [Tim, wondering why the 3.2 branch isn't "inactive"]
> 
> [Georg Brandl]
>> FWIW I have no real objections, I just don't see the gain.
> 
> I'm glad it's OK!  Especially because it's already been done ;-)
> 
> Two gains:
> 
> 1. "hg branches" output now matches what the developer docs imply it
> should be.  It didn't before.

Well, the dev docs are not dogma and could be changed :)

> 2. If a security fix needs to made to 3.2, it will be much easier to
> forward-merge it to the 3.3 and default branches now (the merges won't
> suck in a pile of ancient, and unwanted, irrelevant-to-the-fix
> changes).

It's unusual to develop a security fix on 3.2; usually the fix is done
in the active branches and then backported to security-only branches.

But I get the consistency argument (and especially the .hgtags
entry is nice to have in the newer branches).

cheers,
Georg

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan 
On 25 Aug 2013 05:19, "Benjamin Peterson"  wrote:
>
> 2013/8/24 Terry Reedy :
> > On 8/24/2013 8:51 AM, Stefan Behnel wrote:
> >>
> >> Antoine Pitrou, 24.08.2013 13:53:
> >>>
> >>> This would also imply extension module have to be subclasses of the
> >>> built-in module type. They can't be arbitrary objects like Stefan
> >>> proposed. I'm not sure what the latter enables, but it would probably
> >>> make things more difficult internally.
> >>
> >>
> >> My line of thought was more like: if Python code can stick anything
into
> >> sys.modules and the runtime doesn't care, why can't extension modules
> >> stick
> >> anything into sys.modules as well?
> >
> >
> > Being able to stick anything in sys.modules in CPython is an
implementation
> > artifact rather than language feature.
>
> This is not really true. Many people use this feature to replace
> modules as they are being imported with other things.

Right - arbitrary objects in sys.modules is definitely a supported feature
(e.g. most lazy import mechanisms rely on that). However, such objects
should really provide the module level attributes the import system expects
for ducktyping purposes, which is why I suggest the import system should
automatically take care of setting those.

Cheers,
Nick.

>
> --
> Regards,
> Benjamin
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan 
On 25 Aug 2013 01:44, "Stefan Behnel"  wrote:
>
> Nick Coghlan, 24.08.2013 16:22:
> > On 24 August 2013 23:19, Stefan Behnel wrote:
> >> Nick Coghlan, 24.08.2013 13:36:
> >>> On 24 August 2013 15:51, Nick Coghlan wrote:
>  My current plan is to create an experimental prototype of this
>  approach this weekend. That will include stdlib test cases, so it
will
>  also show how it looks from the extension developer's point of view.
> >>>
> >>> I prototyped as much as I could without PEP 451's ModuleSpec support
here:
> >>>
> >>>
https://bitbucket.org/ncoghlan/cpython_sandbox/commits/branch/new_extension_imports
> >>
> >> Cool. I'll take a look.
> >
> > The new _PyImport_CreateAndExecExtensionModule function does the heavy
lifting:
> >
> >
https://bitbucket.org/ncoghlan/cpython_sandbox/src/081f8f7e3ee27dc309463b48e6c67cf4880fca12/Python/importdl.c?at=new_extension_imports#cl-65
> >
> > One key point to note is that it *doesn't* call
> > _PyImport_FixupExtensionObject, which is the API that handles all the
> > PEP 3121 per-module state stuff. Instead, the idea will be for modules
> > that don't need additional C level state to just implement
> > PyImportExec_NAME, while those that *do* need C level state implement
> > PyImportCreate_NAME and return a custom object (which may or may not
> > be a module subtype).
>
> Is it really a common case for an extension module not to need any C level
> state at all? I mean, this might work for very simple accelerator modules
> with only a few stand-alone functions. But anything non-trivial will
almost
> certainly have some kind of global state, cache, external library, etc.,
> and that state is best stored at the C level for safety reasons.

I'd prefer to encourage people to put that state on an exported *type*
rather than directly in the module global state. So while I agree we need
to *support* C level module globals, I'd prefer to provide a simpler
alternative that avoids them.

We also need the create/exec split to properly support reloading. Reload
*must* reinitialize the object already in sys.modules instead of inserting
a different object or it completely misses the point of reloading modules
over deleting and reimporting them (i.e. implicitly affecting the
references from other modules that imported the original object).

> > Such modules can still support reloading (e.g.
> > to pick up reloaded or removed module dependencies) by providing
> > PyImportExec_NAME as well.
> >
> > (in a PEP 451 world, this would likely be split up as two separate
> > functions, one for create, one for exec)
>
> Can't we just always require extension modules to implement their own
type?
> Sure, it's a lot of boiler plate code, but that could be handled by a
> simple C code generator or maybe even a copy&paste example in the docs. I
> would like to avoid making it too easy for users in the future to get
> anything wrong with reloading or sub-interpreters. Most people won't test
> these things for their own code and the harder it is to make them not
work,
> the more likely it is that a given set of dependencies will properly work
> in a sub-interpreter.
>
> If users are required to implement their own type, I think it would be
more
> obvious where to put global module state, how to define functions (i.e.
> module methods), how to handle garbage collection at the global module
> level, etc.

Take a look at the current example - everything gets stored in the module
dict for the simple case with no C level global state. The module level
functions are still added through a Py_MethodDef array, the docstring still
comes from a C char pointer. I did have to fix the custom type's tp_new
method to use the type pointer passed in by the interpreter rather than a C
static global pointer, but that change would also have been needed if
defining a custom type.

Since Antoine fixed it, there's also nothing particularly quirky about
module destruction in 3.4+ - cyclic GC should "just work".

Cheers,
Nick.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Eli Bendersky 
On Sat, Aug 24, 2013 at 7:33 AM, Nick Coghlan  wrote:

> On 25 August 2013 00:26, Nick Coghlan  wrote:
> > On 25 August 2013 00:13, Antoine Pitrou  wrote:
> >> On Sun, 25 Aug 2013 00:03:01 +1000
> >> Nick Coghlan  wrote:
> >>> If Stefan's "please revert this" as lxml.etree maintainer isn't
> >>> enough, then I'm happy to add a "please revert this" as a core
> >>> committer that is confused about how and when the new tulip-inspired
> >>> incremental parsing API should be used in preference to the existing
> >>> incremental parsing API, and believes this needs to be clearly
> >>> resolved before adding a second way to do it
> >>> (especially if there's a
> >>> possibility of using a different implementation strategy that avoids
> >>> adding the second way).
> >>
> >> To be clear, again: anyone who wants to "see it resolved" can take over
> >> the issue and handle it by themselves. I'm done with it.
> >
> > OK, I'll revert it for now, then. If someone else steps up to resolve
> > the API duplication problem, cool, otherwise we can continue to live
> > without this as a standard library feature.
>
> On the other hand... because other changes have been made to the
> module since the original commit, a simple "hg backout" is no longer
> possible :(
>
> Stefan - if you'd like this reverted, you're going to have to either
> make the alternative solution work correctly, or else craft the commit
> to undo the API addition.
>


I'm strongly opposed to reverting because it cleaned up messy code
duplication and actually make the code size smaller. While I agree that the
API of incremental parsing should be given another look, IncrementalParser
can also be seen as an implementation detail of iterparse(). Thus, it's
probably OK to revert the documentation part of the commit to not mention
IncrementalParser at all, making it an undocumented internal implementation
detail (one of many in this module and elsewhere). However, since we're
still in alpha I don't see much point to doing this change now.

Let's keep discussing this in the issue. Anyone interested - please make
yourself nosy and any feedback on the API is welcome.

Eli
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Stephen J. Turnbull 
Eli Bendersky writes:

 > I'm strongly opposed to reverting [the change to ElementTree]
 > because it cleaned up messy code duplication and actually make the
 > code size smaller. While I agree that the API of incremental parsing
 > should be given another look, IncrementalParser can also be seen as
 > an implementation detail of iterparse().

Except that its API is familiar and cleaner.  Does any current
application depend on *not* doing whatever it is that the new API does
that IncrementalParser *does* do?  If not, why not keep the API of
IncrementalParser and shim the new code in under that?

 > Thus, it's probably OK to revert the documentation part of the
 > commit to not mention IncrementalParser at all,

FWIW, as somebody who can recall using ET exactly one,
IncrementalParser is what I used.

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2

2013-08-24 Thread Eli Bendersky 
On Sat, Aug 24, 2013 at 5:55 PM, Stephen J. Turnbull wrote:

> Eli Bendersky writes:
>
>  > I'm strongly opposed to reverting [the change to ElementTree]
>  > because it cleaned up messy code duplication and actually make the
>  > code size smaller. While I agree that the API of incremental parsing
>  > should be given another look, IncrementalParser can also be seen as
>  > an implementation detail of iterparse().
>
> Except that its API is familiar and cleaner.  Does any current
> application depend on *not* doing whatever it is that the new API does
> that IncrementalParser *does* do?  If not, why not keep the API of
> IncrementalParser and shim the new code in under that?
>

I'm having a difficulty parsing the above. Could you please re-phrase your
suggestion?


>
>  > Thus, it's probably OK to revert the documentation part of the
>  > commit to not mention IncrementalParser at all,
>
> FWIW, as somebody who can recall using ET exactly one,
> IncrementalParser is what I used.
>
>
Just to be on the safe side, I want to make sure that you indeed mean
IncrementalParser, which was committed 4 months ago into the Mercurial
default branch (3.4) and has only seen an alpha release?

Eli
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread PJ Eby 
On Fri, Aug 23, 2013 at 4:50 AM, Stefan Behnel  wrote:
> Reloading and Sub-Interpreters
> ==
>
> To "reload" an extension module, the module create function is executed
> again and returns a new module type. This type is then instantiated as by
> the original module loader and replaces the previous entry in sys.modules.
> Once the last references to the previous module and its type are gone, both
> will be subject to normal garbage collection.

I haven't had a chance to address this on the import-sig discussion
yet about ModuleSpec, but I would like to just mention that one
property of the existing module system that I'm not sure either this
proposal or the ModuleSpec proposal preserves is that it's possible to
implement lazy importing of modules using standard reload() semantics.

My "Importing" package offers lazy imports by creating module objects
in sys.modules that are a subtype of ModuleType, and use a
__getattribute__ hook so that trying to use them fires off a reload()
of the module.  Because the dummy module doesn't have __file__ or
anything else initialized, the import system searches for the module
and then loads it, reusing the existing module object, even though
it's actually only executing the module code for the first time.

That the existing object be reused is important, because once the
dummy is in sys.modules, it can also be imported by other modules, so
references to it can abound everywhere, and we wish only for it to be
loaded lazily, without needing to trace down and replace all instances
of it.  This also preserves other invariants of the module system.

Anyway, the reason I was asking why reloading is being handled as a
special case in the ModuleSpec proposal -- and the reason I'm curious
about certain provisions of this proposal -- is that making the
assumption you can only reload something with the same
spec/location/etc. it was originally loaded with, and/or that if you
are reloading a module then you previously had a chance to do things
to it, doesn't jibe with the way things work currently.

That is to say, in the pure PEP 302 world, there is no special status
for "reload" that is different from "load" -- the *only* thing that's
different is that there is already a module object to use, and there
is *no guarantee that it's a module object that was initialized by the
loader now being invoked*.

AFAICT both this proposal and the ModuleSpec one are making an invalid
assumption per PEP 302, and aren't explicitly proposing to change the
status quo: they just assume things that aren't actually assured by
the prior specs or implementations.

So, for example, this extension module proposal needs to cover what
happens if an extension module is reloaded and the module object is
not of the type or instance it's expecting.  Must it do its own
checking?  Error handling?  Will some other portion of the import
system be expected to handle it?

For that matter, what happens (in either proposal) if you reload() a
module which only has a __name__, and no other attributes?  I haven't
tested with importlib, but with earlier Pythons this results in a
standard module search being done by reload().  But the ModuleSpec
proposal and this one seem to assume that a reload()-ed module must
already be associated with a loader, location, and/or spec.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan 
On 25 August 2013 14:12, PJ Eby  wrote:
> That is to say, in the pure PEP 302 world, there is no special status
> for "reload" that is different from "load" -- the *only* thing that's
> different is that there is already a module object to use, and there
> is *no guarantee that it's a module object that was initialized by the
> loader now being invoked*.

Yeah, this is an aspect of why I'd like PEP 451 to use create & exec
for the new loader API components. That way, any loader which either
doesn't define the create method, or which returns NotImplemented from
the call (a subtlety needed to make this work for C extensions), can
be used with reload *and* with the -m switch via runpy (currently
runpy demands the ability to get hold of the code object).

> AFAICT both this proposal and the ModuleSpec one are making an invalid
> assumption per PEP 302, and aren't explicitly proposing to change the
> status quo: they just assume things that aren't actually assured by
> the prior specs or implementations.

Indeed.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com