Hi Marek,
Marek Felšöci <ma...@felsoci.sk> skribis:
> Recently, I have come around this issue. I created a Singularity
> container using the following `guix pack` command:
>
> `guix pack -f squashfs bash coreutils python python-numpy`
>
> Then, I run a bash shell within the container:
>
> `singularity exec test.gz.squashfs bash`
>
> Now, if I want to use numpy,
>
> `python3 -m numpy`
>
> it gives me the following error and the numpy module IS NOT found:
>
> Error in sitecustomize; set PYTHONVERBOSE for traceback:
> ValueError:
> '/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages'
>
> is not in list
> /gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/python3: No
> module named numpy
I can kinda reproduce via a relocatable pack (relocatable packs and
Singularity packs have in common that they use relative symlinks in
their profile, to allow for relocation):
guix pack -RR python python-numpy -S /bin=bin -S /etc=etc
Then I unpack the thing:
mkdir /tmp/pack; cd /tmp/pack; tar xf /gnu/store/…-pack.tar.gz
And from there I use the same trick as in
‘tests/guix-pack-relocatable.sh’ to “hide” /gnu/store:
--8<---------------cut here---------------start------------->8---
$ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store; .
/tmp/pack/etc/profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 '
/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages
Python 3.9.6 (default, Jan 1 1970, 00:00:01)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>> import os
>>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages")
['setuptools-56.0.0.dist-info', 'distutils-precedence.pth', '_distutils_hack',
'setuptools', 'pip-21.1.3.dist-info', 'sitecustomize.py', 'README.txt', 'pip',
'numpy-1.20.3-py3.9.egg-info', 'numpy', 'pkg_resources', '__pycache__']
>>> os.listdir("/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages/numpy")
['__init__.pyi', 'doc', 'matlib.py', 'ctypeslib.pyi', 'linalg', 'ctypeslib.py',
'compat', 'typing', '__init__.pxd', 'lib', 'char.pyi', 'version.py',
'__config__.py', 'core', '_distributor_init.py', 'distutils', 'tests',
'emath.pyi', '_pytesttester.py', 'fft', '__init__.cython-30.pxd',
'conftest.py', '__init__.py', 'py.typed', '_globals.py', 'rec.pyi', 'testing',
'LICENSE.txt', 'polynomial', 'f2py', 'matrixlib', 'random', 'ma', 'dual.py',
'setup.py', '__pycache__']
--8<---------------cut here---------------end--------------->8---
So in this case ‘GUIX_PYTHONPATH’ is correctly set, but it looks as
though ‘sitecustomize.py’ wasn’t loaded. (I don’t get the “Error in
sitecustomize” message that Marek sees though, weird!)
>From Marek’s message, it would seem that somewhere file names are
constructed by appending “..”. Given Unix’s weird dot-dot semantics,
relative symlinks could be interfering:
--8<---------------cut here---------------start------------->8---
$ ls -l
/tmp/pack/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages/
totalo 48
lrwxrwxrwx 1 ludo users 102 Jan 1 1970 _distutils_hack ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/_distutils_hack
lrwxrwxrwx 1 ludo users 111 Jan 1 1970 distutils-precedence.pth ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/distutils-precedence.pth
lrwxrwxrwx 1 ludo users 99 Jan 1 1970 numpy ->
../../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy
lrwxrwxrwx 1 ludo users 121 Jan 1 1970 numpy-1.20.3-py3.9.egg-info ->
../../../../mlccgh05bf8cdinq0ilpvpdmsspq36pv-python-numpy-1.20.3R/lib/python3.9/site-packages/numpy-1.20.3-py3.9.egg-info
lrwxrwxrwx 1 ludo users 90 Jan 1 1970 pip ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip
lrwxrwxrwx 1 ludo users 107 Jan 1 1970 pip-21.1.3.dist-info ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pip-21.1.3.dist-info
lrwxrwxrwx 1 ludo users 100 Jan 1 1970 pkg_resources ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/pkg_resources
lrwxrwxrwx 1 ludo users 98 Jan 1 1970 __pycache__ ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/__pycache__
lrwxrwxrwx 1 ludo users 97 Jan 1 1970 README.txt ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/README.txt
lrwxrwxrwx 1 ludo users 97 Jan 1 1970 setuptools ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/setuptools
lrwxrwxrwx 1 ludo users 114 Jan 1 1970 setuptools-56.0.0.dist-info ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/setuptools-56.0.0.dist-info
lrwxrwxrwx 1 ludo users 103 Jan 1 1970 sitecustomize.py ->
../../../../niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages/sitecustomize.py
--8<---------------cut here---------------end--------------->8---
Now look below how dot-dot resolution whereas “lexical” dot-dot
resolution dose the right thing:
--8<---------------cut here---------------start------------->8---
$ ls -l
/tmp/pack/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages
ls: ne eblas atingi
'/tmp/pack/gnu/store/2dyd6rlcc9m68k3mvkw86k8dygkbg025-profile/bin/../../p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages':
Dosiero aŭ dosierujo ne ekzistas
$ ls -l
/tmp/pack/gnu/store/p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/lib/python3.9/site-packages
totalo 40
dr-xr-xr-x 3 ludo users 4096 Jan 1 1970 _distutils_hack/
-r--r--r-- 1 ludo users 152 Jan 1 1970 distutils-precedence.pth
dr-xr-xr-x 5 ludo users 4096 Jan 1 1970 pip/
dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 pip-21.1.3.dist-info/
dr-xr-xr-x 6 ludo users 4096 Jan 1 1970 pkg_resources/
dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 __pycache__/
-r--r--r-- 1 ludo users 119 Jan 1 1970 README.txt
dr-xr-xr-x 7 ludo users 4096 Jan 1 1970 setuptools/
dr-xr-xr-x 2 ludo users 4096 Jan 1 1970 setuptools-56.0.0.dist-info/
-r--r--r-- 1 ludo users 2172 Jan 1 1970 sitecustomize.py
--8<---------------cut here---------------end--------------->8---
In Python’s ‘site.py’, the ‘addsitedir’ method indirectly calls
‘os.path.abspath’, not sure if that is at fault.
I also see that ‘sys.path’ is in fact completely bogus, with “pack”
missing from every directory name:
--8<---------------cut here---------------start------------->8---
$ unshare -mrf sh -c 'mount -t tmpfs -o ro none /gnu/store; .
/tmp/pack/etc/profile; echo $GUIX_PYTHONPATH ; /tmp/pack/bin/python3 '
/gnu/store/w5cz9gbqbja99x0636zsg2fzksmfig4n-profile/lib/python3.9/site-packages
Python 3.9.6 (default, Jan 1 1970, 00:00:01)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python39.zip',
'/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9',
'/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/lib-dynload',
'/tmp/niddlj0qbk5vzk1mdn5im9y4x6l8a2k9-python-3.9.6R/lib/python3.9/site-packages']
--8<---------------cut here---------------end--------------->8---
So I think something somewhere is manipulating file names and gets
confused.
Does that ring a bell, Maxim?
Thanks,
Ludo’.
