Re: [Python-Dev] libffi embedded in CPython
On Tue, Mar 24, 2015 at 11:31 PM, Paul Moore wrote: > On 12 March 2015 at 17:44, Paul Moore wrote: >> On 12 March 2015 at 17:26, Brett Cannon wrote: I'm all for ditching our 'libffi_msvc' in favor of adding libffi as another 'external' for the Windows build. I have managed to get _ctypes to build on Windows using vanilla libffi sources, prepared using their configure script from within Git Bash and built with our usual Windows build system (properly patched). Unfortunately, making things usable will take some work on ctypes itself, which I'm not qualified to do. I'm happy to pass on my procedure and patches for getting to the point of successful compilation to anyone who feels up to fixing the things that are broken. >>> >>> >>> So it seems possible to use upstream libffi but will require some work. >> >> I'd be willing to contemplate helping out on the Windows side of >> things, if nobody else steps up (with the proviso that I have little >> free time, and I'm saying this without much idea of what's involved >> :-)) If Zachary can give a bit more detail on what the work on ctypes >> is, and/or put what he has somewhere that I could have a look at, that >> might help. > > One thing that seems to be an issue. On Windows, ctypes detects if the > FFI call used the wrong number of arguments off the stack, and raises > a ValueError if it does. The tests rely on that behaviour. But it's > based on ffi_call() returning a value, which upstream libffi doesn't > do. As far as I can tell (not that the libffi docs are exactly > comprehensive...) there's no way of getting that information from > upstream libffi. > > What does Unix ctypes do when faced with a call being made with the > wrong number of arguments? On Windows, using upstream libffi and > omitting the existing check, it seems to crash the Python process, > which obviously isn't good. But the test that fails is > Windows-specific, and short of going through all the tests looking for > one that checks passing the wrong number of arguments and isn't > platform-specific, I don't know how Unix handles this. > > Can anyone on Unix tell me if a ctypes call with the wrong number of > arguments returns ValueError on Unix? Something like strcmp() (with no > args) should do as a test, I guess... > > If there's a way Unix handles this, I can see about replicating it on > Windows. But if there isn't, I fear we could always need a patched > libffi to maintain the interface we currently have... > > Thanks, > Paul Linux crashes. The mechanism for detecting the number of arguments is only available on windows (note that this is a band-aid anyway, since if your arguments are of the wrong kind you segfault anyway). We do have two copies of libffi one for windows one for unix anyway, don't we? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 25 March 2015 at 08:05, Maciej Fijalkowski wrote: > Linux crashes. The mechanism for detecting the number of arguments is > only available on windows (note that this is a band-aid anyway, since > if your arguments are of the wrong kind you segfault anyway). We do > have two copies of libffi one for windows one for unix anyway, don't > we? Yes, this is in relation to looking at whether we can just use the upstream libffi directly on Windows. It looks like we guarantee a ValueError for the wrong number of arguments on Windows and upstream libffi doesn't give a way to detect that, so there may be a backward compatibility issue in doing so. Thanks for checking for me. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons
On 24 Mar 2015 07:59, "Gregory P. Smith" wrote:
>
>
> On Wed, Mar 18, 2015 at 9:48 AM Barry Warsaw wrote:
>>
>> On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote:
>>
>> >Does it work to pass command line options to Python in the shebang?
>>
>> Yes, but only one "word", thus -Es or -I.
>>
>> We've often mused about whether it makes sense to have two Pythons on the
>> system. One for system scripts and another for users. System Python
>> ('/usr/bin/spython' perhaps) would be locked down and only extensible by
>> system packages. On Debuntu that might mean no /usr/local on sys.path.
It
>> would also have a much more limited set of installed packages, i.e. only
those
>> needed to support system functionality.
>
>
> I recommend this. It'd be nice to see a common OS distro actually do it.
>
> For a system-only Python lockdown you should remove distutils and pip to
prevent anyone from easily installing anything that isn't a distro package
into it. Possibly even removing its site-packages directory and sys.path
entry all together (things your distro needs to install could mix directly
with the stdlib packages)
The hard part of this is the complexity in getpath.c and the current
difficulty of testing the interaction of all the different configuration
settings properly. Completely overriding the standard path setup isn't too
bad, it's *partially* overriding it that's hard.
That's really one of the main goals of the proposed startup refactoring in
PEP 432 - separating things into distinct phases so the path configuration
is easier to customise and so the current behaviour becomes easier to test.
That's a fairly big project with high barriers to entry though, as even
just wrapping your head around the current accumulated complexity in the
way the default sys.path gets constructed is a time consuming task.
That approach has the virtue of not being a "system Python" specific
solution, it's a "make embedding the CPython runtime much easier" solution,
with "system Python with different default settings" as the reference use
case.
>> /usr/bin/python2 and /usr/bin/python3 then would be user tools, with all
the
>> goodness they currently have.
>>
>> It's never gotten much farther than musings, but protecting the system
against
>> the weird things people install would be a good thing. OTOH, this feels
a lot
>> like virtual environments so maybe there's something useful to be mined
there.
For distro level work it would be closer to the environment modules used in
high performance computing work (which are also the basis of
softwarecollections.org)
Where things get really messy is on the package management side. Either we
end up with yet another copy of modules that are often already duplicated
between the Python 2 & 3 stacks, or else the relevant user Python needs to
be customised to also be able to see the system level libraries for things
like Kerberos, interfacing with the package manager, dbus, etc.
The former approach = "ugh, no, just no" as far as I'm concerned, while the
latter would be a messy patch to try to carry given the current difficulty
of customising and testing the startup sequence.
>
>
> While people sometimes suggest virtualenv as a solution for this. It
isn't really the same thing. It isn't a hermetic clone of the original
interpreter. It copies the main binary but symlinks back to much of the
stdlib. So any existing virtualenv can be an out of date unsupported mix
of both after the original interpreter is updated. This has caused users
problems in the past with some minor version updates where an internal
non-public API used between some binary and a stdlib module was updated as
part of a bugfix. Suddenly they didn't match up for existing virtualenvs.
>
> virtualenv is an amazing hack that I promote to most anyone for their own
applications use with the occasional known caveats (solvable by regurly
rebuilding your virtualenvs)... But I wouldn't want to see it used for the
core OS itself even though it sounds better at first glance.
Right, this is why Fedora/RHEL/CentOS have software collections instead,
and those have their own "gotchas" that currently make them unsuitable for
use as the main system Python (for example: the Fedora installer & package
manager are both written in Python, so having the main system Python be an
SCL would bring SCL support into the minimal package set, which we wouldn't
want to do. The typical aim is to find ways to get things *out* of that
set, not add new ones)
Cheers,
Nick.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 25 Mar 2015 08:15:18 + Paul Moore wrote: > On 25 March 2015 at 08:05, Maciej Fijalkowski wrote: > > Linux crashes. The mechanism for detecting the number of arguments is > > only available on windows (note that this is a band-aid anyway, since > > if your arguments are of the wrong kind you segfault anyway). We do > > have two copies of libffi one for windows one for unix anyway, don't > > we? > > Yes, this is in relation to looking at whether we can just use the > upstream libffi directly on Windows. It looks like we guarantee a > ValueError for the wrong number of arguments on Windows and upstream > libffi doesn't give a way to detect that, so there may be a backward > compatibility issue in doing so. I'm not sure we guarantee anything. In any case, it's only a small proportion of the kind of crashes you can get by messing the signature. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 25 March 2015 at 09:09, Antoine Pitrou wrote: > I'm not sure we guarantee anything. In any case, it's only a small > proportion of the kind of crashes you can get by messing the signature. Fair point. I guess what I'm asking is, would it be OK to remove the code that checks for a stack size discrepancy and raises ValueError, and the tests that verify this behaviour, as part of switching to using upstream libffi directly? On a related note, is there any information available on how the "externals" repo is maintained? In particular, should things in there be exact copies of upstream, or is it OK to include extra data (in this case, the results of running "configure" for the Windows build)? It works for me either way, it's just a matter of how the build process would be structured and maintained. Thanks, Paul Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 25 Mar 2015 09:22:01 + Paul Moore wrote: > On 25 March 2015 at 09:09, Antoine Pitrou wrote: > > I'm not sure we guarantee anything. In any case, it's only a small > > proportion of the kind of crashes you can get by messing the signature. > > Fair point. I guess what I'm asking is, would it be OK to remove the > code that checks for a stack size discrepancy and raises ValueError, > and the tests that verify this behaviour, as part of switching to > using upstream libffi directly? IMHO yes. Of course it's still a behaviour change, but relying on it sounds seriously broken. > On a related note, is there any information available on how the > "externals" repo is maintained? In particular, should things in there > be exact copies of upstream, or is it OK to include extra data (in > this case, the results of running "configure" for the Windows build)? > It works for me either way, it's just a matter of how the build > process would be structured and maintained. Zachary or Steve could probably answer you on that one. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] super() does not work during class initialization
On 24 March 2015 at 08:22, Greg Ewing wrote: > Martin Teichmann wrote: >> >> maybe >> we could just change the compiler to leave the order in which things are >> defined >> in a class in the class namespace, say as a member __order__? Then we >> could >> use plain-old dicts for the class namespace, and we would not slow down >> class >> creation (not that it matters much), as determining the order would happen >> at >> compile time. > > > I don't think the compiler can determine the order in > all cases. Consider: > > class Spam: > > if moon_is_full: > alpha = 1 > beta = 2 > else: > beta = 2 > alpha = 1 This is also expected to work in class namespaces: locals()["alpha"] = 1 The language reference suggests it isn't, there's an open tracker issue I filed some time ago to suggest clarifying it but haven't found the time to actually sit down and come up with readable wording: http://bugs.python.org/issue17960 Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Mar 25, 2015 4:22 AM, "Paul Moore" wrote: > > On 25 March 2015 at 09:09, Antoine Pitrou wrote: > > I'm not sure we guarantee anything. In any case, it's only a small > > proportion of the kind of crashes you can get by messing the signature. > > Fair point. I guess what I'm asking is, would it be OK to remove the > code that checks for a stack size discrepancy and raises ValueError, > and the tests that verify this behaviour, as part of switching to > using upstream libffi directly? > > On a related note, is there any information available on how the > "externals" repo is maintained? In particular, should things in there > be exact copies of upstream, or is it OK to include extra data (in > this case, the results of running "configure" for the Windows build)? > It works for me either way, it's just a matter of how the build > process would be structured and maintained. Its not extremely clear how it's "supposed to be" done; look at the differences between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, though, we will store the configure output so that our build doesn't depend on any more than it absolutely has to. -- Zach On a phone ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] super() does not work during class initialization
>> I don't think the compiler can determine the order in
>> all cases. Consider:
>>
>> class Spam:
>>
>> if moon_is_full:
>> alpha = 1
>> beta = 2
>> else:
>> beta = 2
>> alpha = 1
>
> This is also expected to work in class namespaces:
>
> locals()["alpha"] = 1
>
> The language reference suggests it isn't, there's an open tracker
> issue I filed some time ago to suggest clarifying it but haven't found
> the time to actually sit down and come up with readable wording:
> http://bugs.python.org/issue17960
Well, for sure the compiler cannot deduce things happening at runtime,
but it still has an order in which things appear at compile time. And
for nearly all use cases that's by far enough. We cannot stop people
from doing weird things, but unless there is a use case for those
weird things, we don't need to support them.
And I think that tampering with locals() is not really a good idea. In
your issue you mention that you want that so that you can create
enums programatically. This is already doable by writing:
from enum import Enum
from types import new_class
def cb(ns):
ns.update({"a{}".format(i): i for i in range(100)})
return ns
Calc = new_class("Calc", (Enum,), None, cb)
I think this is simple enough that we don't need another way
of doing it.
Btw, going on-topic again, what should I do to get my patch
to make super() work during class initialization into python?
Greetings
Martin
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
Zachary Ware wrote: > On Mar 25, 2015 4:22 AM, "Paul Moore" wrote: >> On a related note, is there any information available on how the >> "externals" repo is maintained? In particular, should things in there >> be exact copies of upstream, or is it OK to include extra data (in >> this case, the results of running "configure" for the Windows build)? >> It works for me either way, it's just a matter of how the build >> process would be structured and maintained. > > Its not extremely clear how it's "supposed to be" done; look at the > differences > between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, > though, we will store the configure output so that our build doesn't depend on > any more than it absolutely has to. It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :) As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system. Cheers, Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 26 March 2015 at 01:57, Steve Dower wrote: > Zachary Ware wrote: >> On Mar 25, 2015 4:22 AM, "Paul Moore" wrote: >>> On a related note, is there any information available on how the >>> "externals" repo is maintained? In particular, should things in there >>> be exact copies of upstream, or is it OK to include extra data (in >>> this case, the results of running "configure" for the Windows build)? >>> It works for me either way, it's just a matter of how the build >>> process would be structured and maintained. >> >> Its not extremely clear how it's "supposed to be" done; look at the >> differences >> between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, >> though, we will store the configure output so that our build doesn't depend >> on >> any more than it absolutely has to. > > It would be nice for at least one other person to know how to do it. I can't > see the SVN logs, but I feel like you're the only person who's touched it in > a while :) > > As well as the configure output, we sometimes include very light patching > (for example, Tcl/Tk currently has a patch to their makefile to work around a > VC14 bug). I'm not sure whether that's considered good practice or not, but I > prefer only needing to do it once vs. building it into the build system. It likely makes sense to try to at least write down the status quo regarding the externals repo as a new top level section in the Developer's Guide. That way you'll at least have a shared understanding of the starting point for any further improvements. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Mar 25, 2015 9:28 PM, "Nick Coghlan" wrote: > > On 26 March 2015 at 01:57, Steve Dower wrote: > > Zachary Ware wrote: > >> On Mar 25, 2015 4:22 AM, "Paul Moore" wrote: > >>> On a related note, is there any information available on how the > >>> "externals" repo is maintained? In particular, should things in there > >>> be exact copies of upstream, or is it OK to include extra data (in > >>> this case, the results of running "configure" for the Windows build)? > >>> It works for me either way, it's just a matter of how the build > >>> process would be structured and maintained. > >> > >> Its not extremely clear how it's "supposed to be" done; look at the differences > >> between how we handle OpenSSL and Tcl/Tk, for example. One way or the other, > >> though, we will store the configure output so that our build doesn't depend on > >> any more than it absolutely has to. > > > > It would be nice for at least one other person to know how to do it. I can't see the SVN logs, but I feel like you're the only person who's touched it in a while :) > > > > As well as the configure output, we sometimes include very light patching (for example, Tcl/Tk currently has a patch to their makefile to work around a VC14 bug). I'm not sure whether that's considered good practice or not, but I prefer only needing to do it once vs. building it into the build system. > > It likely makes sense to try to at least write down the status quo > regarding the externals repo as a new top level section in the > Developer's Guide. That way you'll at least have a shared > understanding of the starting point for any further improvements. I'll try to get something thrown together at least by the time I leave PyCon. Unfortunately I can't guarantee anything sooner than that. -- Zach On a phone ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 26 March 2015 at 12:40, Zachary Ware wrote: > On Mar 25, 2015 9:28 PM, "Nick Coghlan" wrote: >> >> On 26 March 2015 at 01:57, Steve Dower wrote: >> > Zachary Ware wrote: >> >> On Mar 25, 2015 4:22 AM, "Paul Moore" wrote: >> >>> On a related note, is there any information available on how the >> >>> "externals" repo is maintained? In particular, should things in there >> >>> be exact copies of upstream, or is it OK to include extra data (in >> >>> this case, the results of running "configure" for the Windows build)? >> >>> It works for me either way, it's just a matter of how the build >> >>> process would be structured and maintained. >> >> >> >> Its not extremely clear how it's "supposed to be" done; look at the >> >> differences >> >> between how we handle OpenSSL and Tcl/Tk, for example. One way or the >> >> other, >> >> though, we will store the configure output so that our build doesn't >> >> depend on >> >> any more than it absolutely has to. >> > >> > It would be nice for at least one other person to know how to do it. I >> > can't see the SVN logs, but I feel like you're the only person who's >> > touched >> > it in a while :) >> > >> > As well as the configure output, we sometimes include very light >> > patching (for example, Tcl/Tk currently has a patch to their makefile to >> > work around a VC14 bug). I'm not sure whether that's considered good >> > practice or not, but I prefer only needing to do it once vs. building it >> > into the build system. >> >> It likely makes sense to try to at least write down the status quo >> regarding the externals repo as a new top level section in the >> Developer's Guide. That way you'll at least have a shared >> understanding of the starting point for any further improvements. > > I'll try to get something thrown together at least by the time I leave > PyCon. Unfortunately I can't guarantee anything sooner than that. That sounds plenty soon enough to me - my own list of "I should really do something about that some day" items stretches back years, as the issue tracker occasionally reminds me :) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
