Hi Hartmut! I was hoping you'd join this thread! Hartmut Goebel <h.goe...@crazy-compilers.com> writes:
> Am 06.01.2017 um 19:41 schrieb Maxim Cournoyer:
>> One thing which I discovered while testing pip on Guix was that
>> depending on how you use Python on Guix the PYTHONPATH differs:
>
> I analysed this problem and found set we have another serious problem
> here. But good news: There is an easy solution for both problems.
>
> Result of the analysis (as explained below): We must not extend
> PYTHONPATH for adding the profile's site-packages directory.
Right! site.py will put prepend any previously set PYTHONPATH to the one
it forms. If a site-packages is present there it will appear before the
user site location, which is not good. I had missed this particular
problem before by failing to restart my user session after installing
python (the custom PYTHONPATH we set was not yet in effect).
>
> Proposed solution: The standard library module "site" already contains a
> place where we can hook in:
>
> # Prefixes for site-packages; add additional prefixes like /usr/local
> here
> PREFIXES = [sys.prefix, sys.exec_prefix]
>
> So we simply prepend $GUIX_ENVIRONMENT to this list:
>
> PREFIXES = [os.path.expandvar(GUIX_ENVIRONMENT), sys.prefix,
> sys.exec_prefix]
>
I think your approach is correct! Any directory we add to PREFIXES will
get picked by the `addsitepackages` function in site.py, and this
function is called *following* the `addusersitepackages` function in
site.main(), which means those entries will appear after the user site
location in sys.path, which is what we want.
What is currently responsible for exporting PYTHONPATH to
".guix-profile/lib/python2.7/site-packages" ? It looks like it might be
the `native-search-paths' fields of our python-2.7 package?
(native-search-paths
(list (search-path-specification
(variable "PYTHONPATH")
(files '("lib/python2.7/site-packages")))))
This should rather be added with a patch adding it to the PREFIXES
variable of the site.py module, as you suggested.
This is the value of PYTHONPATH in my profile, after installing python@2
and re-login:
$ echo $PYTHONPATH
/home/maxim/.guix-profile/lib/python2.7/site-packages
Also, if we no longer rely on manipulating the PYTHONPATH directly, we
should also remove the "python-2.7-site-prefixes.patch" patch, which
adds /gnu/store entries found in PYTHONPATH to the PREFIXES variable.
> Prepending to speed up searches, since within an environment almost all
> site-packages will end in GUIX_ENVIRONMENT. There is no need to check if
> $GUIX_ENVIRONMENT is set, since if it is not, it will be left unchanged
> and the non-existing path will be skipped out automatically
>
> This would solve both the problem Maxim described and the problem
> described below, since the Guix site-packages behave exactly like any
> other global site-packages directory.
>
> When following this proposal, we *may* even remove some "wrapping" stuff
> in the python-build-system. But this has to be investigated first.
>
Yes, it will be interesting to see if the changes proposed here would
remove the need for the wrap phase. I hope it does :).
>
> Analysis (proof see below):
>
> The python interpreter was two command line options, which are rarely
> used, though:
>
> -E ignores environment variables, esp. PYTHONPATH
> -S Disable the import of the module site and the site-dependent
> manipulations of sys.path.
>
> This means:
> a) When passing -E, the Guix environment's site-packages are ignored
> (together with PYTHONPATH), while they should not.
> b) When passing -S, the Guix environment's site-packages should be
> ignored, but is not (since it is specified in $PYTHONPATH, which is not
> ignored)
>
> Conclusion: We should must not use PYTHONPATH to specify the Guix's
> environment's site-package directory.
>
Right. In general, we can generalize the PYTHONPATH rule to "we must not use
PYTHONPATH" ;)
>
> Analysis details:
>
> (guix)$ which python3
> /gnu/store/zcnb…-profile/bin/python3
>
>
> Without any options:
>
> -> /home/USER/.local/lib/python3.5/site-packages and
> /gnu/store/zcnb…-profile/lib/python3.5/site-packages
> should be in sys.path
>
> (base)$ python3 -c 'import sys ; print("\n".join(sys.path))'
>
> /usr/lib64/python34.zip
> /usr/lib64/python3.4
> /usr/lib64/python3.4/plat-linux
> /usr/lib64/python3.4/lib-dynload
> /home/USER/.local/lib/python3.4/site-packages
> /usr/lib64/python3.4/site-packages
> /usr/lib/python3.4/site-packages
>
> (guix)$ python3 -c 'import sys ; print("\n".join(sys.path))'
>
> /gnu/store/zcnb…-profile/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python35.zip
> /gnu/store/alk9…-python-3.5.2/lib/python3.5
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
> /home/USER/.local/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages
>
>
> -E Ignore environment variables like PYTHONPATH and PYTHONHOME
> that modify the behavior of the interpreter.
>
> -> /home/USER/.local/lib/python3.5/site-packages and
> /gnu/store/zcnb…-profile/lib/python3.5/site-packages
> should be in sys.path
>
> (base)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'
>
> /usr/lib64/python34.zip
> /usr/lib64/python3.4
> /usr/lib64/python3.4/plat-linux
> /usr/lib64/python3.4/lib-dynload
> /home/USER/.local/lib/python3.4/site-packages
> /usr/lib64/python3.4/site-packages
> /usr/lib/python3.4/site-packages
>
> (guix)$ python3 -E -c 'import sys ; print("\n".join(sys.path))'
>
> /gnu/store/alk9…-python-3.5.2/lib/python35.zip
> /gnu/store/alk9…-python-3.5.2/lib/python3.5
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
> /home/USER/.local/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/site-packages
>
>
>
> -S = Disable the import of the module site and the site-dependent
> manipulations of sys.path that it entails.
>
> -> /home/USER/.local/lib/python3.5/site-packages and
> /gnu/store/zcnb…-profile/lib/python3.5/site-packages
> sould *not* be in sys.path
>
> (base)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'
>
> /usr/lib64/python34.zip
> /usr/lib64/python3.4/
> /usr/lib64/python3.4/plat-linux
> /usr/lib64/python3.4/lib-dynload
>
>
> (guix)$ python3 -S -c 'import sys ; print("\n".join(sys.path))'
>
> /gnu/store/zcnb…-profile/lib/python3.5/site-packages
> /gnu/store/alk9…-python-3.5.2/lib/python35.zip
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/plat-linux
> /gnu/store/alk9…-python-3.5.2/lib/python3.5/lib-dynload
Thanks for the feedback and detailed analysis!
Here's what I will attempt from now:
In the python-2.7 variable:
1. Remove the current "python-2.7-site-prefixes" patch.
2. Apply a new patch to site.py to add "$HOME/.guix-profile"
as well as "$GUIX_ENVIRONMENT".
3. Remove the 'native-search-paths' field.
As a bonus, now that we won't be using PYTHONPATH anymore, there
shouldn't be any crosstalk between Python 2 & Python 3, so we ought to
be able to install both in the same profile/environment at the same time
(currently installing Python 3 "updates" Python 2).
Maxim
signature.asc
Description: PGP signature
