[Python-Dev] Anyone else seeing a lack of caching in local docs builds?
Hi folks, I'm currently seeing a behaviour where every time I run "make html", all 474 source files get rebuilt. For now, I'm assuming I've messed something up with my local docs build setup, but figured I'd ask if anyone else was seeing this, in case it was actually broken at the build level (CI wouldn't pick this up, since it always builds from scratch anyway). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source of truth for C-API
On Thu, 24 Jan 2019 at 02:56, Victor Stinner wrote: > > I suggest you to add a new function and leaves the existing function > unchanged. "Just in case". > > You may deprecate the old functions at the same time using Py_DEPRECATED(). And potentially put an underscore in front of the new ones (or even hide them behind Py_BUILD_CORE). Some of these older APIs are only public because the modern conventions we use for hiding implementation details weren't established yet. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sub-interpreters: importing numpy causes hang
On Thu, 24 Jan 2019 at 05:45, Stephan Reiter wrote:
> If we create a fresh OS thread and make it call PyGILState_Ensure, it
> won't have a PyThreadState saved under autoTLSkey. That means it will
> create one using the main interpreter. I, as the developer embedding
> Python into my application and using multiple interpreters, have no
> control here. Maybe I know that under current conditions a certain
> other interpreter should be used.
>
> I'll try to provoke this situation and then introduce a callback from
> Python into my application that will allow me to specify which
> interpreter should be used, e.g. code as follows:
>
> PyInterpreter *pickAnInterpreter() {
> return activePlugin ? activePlugin->interpreter : nullptr; //
> nullptr maps to main interpreter
> }
>
> PyGILState_SetNewThreadInterpreterSelectionCallback(&pickAnInterpreter);
>
> Maybe rubbish. But I think a valuable experiment that will give me a
> better understanding.
That actually sounds like a pretty plausible approach to me, at least
for cases where the embedding application maintains some other state
that lets it know which interpreter a new thread should be associated
with. The best aspect of it is that it would let the embedding
application decide how to handle registration of previously unknown
threads with the Python runtime *without* requiring that all existing
extension modules switch to a new thread registration API first.
I'll pass the concept along to Graham Dumpleton (author of the
mod_wsgi module for Apache httpd) to see if an interface like this
might be enough to resolve some of the major compatibility issues
mod_wsgi currently encounters with subinterpreters.
Cheers,
Nick.
--
Nick Coghlan | [email protected] | Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add more SyntaxWarnings?
On Fri, 25 Jan 2019 at 10:15, Eric V. Smith wrote: > It would be a change if the code is never called. I'm not sure we care > about code that's never called, but it is a change. The biggest problem with converting runtime errors to compile time errors is that it means affected dead code goes from being a readability & maintainability problem to "Python X.Y breaks my project". SyntaxWarning splits the difference nicely, since the compiler complains about the same dead code that would confuse a human reader, but also ignores it and moves on. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sub-interpreters: importing numpy causes hang
Cool. Thanks, Nick!
I did experiments based on this idea (
https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622)
and haven't rejected it yet. :-)
Together with the other fix (
https://github.com/stephanreiter/cpython/commit/c1afa0c8cdfab862f409f1c7ff02b189f5191cbe),
numpy at least is happy in my Python-hosting app.
I will pursue the idea of swapping sys.modules in a single Interpreter now
because that wouldn't require patching Python and I might get the mileage
out of this approach I need.
Still interested in improving sub-interpreters, though. I just need to
balance short and long term solution. :-)
Stephan
Den søn. 27. jan. 2019, 15.17 skrev Nick Coghlan On Thu, 24 Jan 2019 at 05:45, Stephan Reiter
> wrote:
> > If we create a fresh OS thread and make it call PyGILState_Ensure, it
> > won't have a PyThreadState saved under autoTLSkey. That means it will
> > create one using the main interpreter. I, as the developer embedding
> > Python into my application and using multiple interpreters, have no
> > control here. Maybe I know that under current conditions a certain
> > other interpreter should be used.
> >
> > I'll try to provoke this situation and then introduce a callback from
> > Python into my application that will allow me to specify which
> > interpreter should be used, e.g. code as follows:
> >
> > PyInterpreter *pickAnInterpreter() {
> > return activePlugin ? activePlugin->interpreter : nullptr; //
> > nullptr maps to main interpreter
> > }
> >
> > PyGILState_SetNewThreadInterpreterSelectionCallback(&pickAnInterpreter);
> >
> > Maybe rubbish. But I think a valuable experiment that will give me a
> > better understanding.
>
> That actually sounds like a pretty plausible approach to me, at least
> for cases where the embedding application maintains some other state
> that lets it know which interpreter a new thread should be associated
> with. The best aspect of it is that it would let the embedding
> application decide how to handle registration of previously unknown
> threads with the Python runtime *without* requiring that all existing
> extension modules switch to a new thread registration API first.
>
> I'll pass the concept along to Graham Dumpleton (author of the
> mod_wsgi module for Apache httpd) to see if an interface like this
> might be enough to resolve some of the major compatibility issues
> mod_wsgi currently encounters with subinterpreters.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan | [email protected] | Brisbane, Australia
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Anyone else seeing a lack of caching in local docs builds?
On 1/27/2019 9:01 AM, Nick Coghlan wrote: Hi folks, I'm currently seeing a behaviour where every time I run "make html", all 474 source files get rebuilt. I just rebuilt, ditto, all 474. Caching only works when I rebuild 'soon' (at least within same day) after a complete rebuild. I just updated and only function.rst was changed and only function.html and the indexes were rebuilt. For now, I'm assuming I've messed something up with my local docs build setup, but figured I'd ask if anyone else was seeing this, in case it was actually broken at the build level (CI wouldn't pick this up, since it always builds from scratch anyway). I think something is broken. Caching used to work better. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sub-interpreters: importing numpy causes hang
On Sun, Jan 27, 2019, 06:34 Stephan Reiter Cool. Thanks, Nick! > > I did experiments based on this idea ( > https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622) > and haven't rejected it yet. :-) > > Together with the other fix ( > https://github.com/stephanreiter/cpython/commit/c1afa0c8cdfab862f409f1c7ff02b189f5191cbe), > numpy at least is happy in my Python-hosting app. > So again, just to make sure you're aware, even if it looks like it's working right now, there are definitely many subtle ways that numpy will break when used in a subinterpreter and this configuration is not supported by the numpy devs. If you discover later that there's some strange crash, or even that you've been getting incorrect results for months without noticing, then the numpy devs will be sympathetic but will probably close your bugs without further investigation. -n ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sub-interpreters: importing numpy causes hang
On Thu, Jan 24, 2019 at 1:25 PM Chris Barker - NOAA Federal via Python-Dev < [email protected]> wrote: > If your primary concern is module clashes between plugins, maybe you > can hack around that: > > 1) if the plugins are providing copies of any other modules, then you > can simply require them to put them in their own namespace — that is, > a plug-in is a single package, with however many sub modules as it may > need. > > 2) if plugins might require third party packages that need to be > isolated, then maybe you could use an import hook that > re-names/isolates the modules each plugin loads, so they are kept > separate. > > I haven’t thought through how to do any of this, but in principle, you > can have the same module loaded twice if it has a different name. > This is dangerous for extension modules. C is a single global space unrelated to Python module names that cannot be isolated without intentionally building and linking each desired extension module statically and configured not to export its own symbols (no-export-dynamic). Non trivial. Suggesting importing the same extension module multiple times under different Python sys.modules names is a recipe for disaster. Most extension module code is not written with that in mind. So while *some* things happen to "work", many others blow up in unexpected hard to debug ways. Not that sub interpreters aren’t cool and useful, but you can probably > handle module clashes in a simpler way. > They're a cool and useful theory... but I really do not recommend their use for code importing other libraries expecting to be isolated. CPython doesn't offer multiple isolated runtimes in a process today. -gps ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
