[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: Merged my PR, but I want to leave this open in commit review for now - I'm not sure it deals with all the issues here, and probably not everything from the Discourse thread linked by Victor (though it might come close). -- stage: patch review -> commit

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: New changeset 7407fe4c25ba0308d49e3e88e4a107ef32251cdc by Steve Dower in branch 'main': bpo-46028: Calculate base_executable by resolving symlinks in a venv (GH-30144) https://github.com/python/cpython/commit/7407fe4c25ba0308d49e3e88e4a107ef32251cdc --

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

STINNER Victor added the comment: See also: https://discuss.python.org/t/virtual-environments-vs-nix-python-upgrades/12588 "Virtual environments vs. *nix Python upgrades". -- nosy: +vstinner ___ Python tracker

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Ned Batchelder added the comment: Thanks, your change looks good. The exact symlinks in the nested venv's are different for 3.10.1 and your 3.11, but they both work: $ python3.10 -c "import sys; print(sys.version)" 3.10.1 (main, Dec 14 2021, 08:30:13) [Clang 12.0.0 (clang-1200.0.32.29)] $ p

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: This PR *might* be a fix, but I think it's only partial. It isn't going to work if you've made a venv and copied "python3"->"python", for example. It still might be best solved by writing base_executable into pyvenv.cfg, and then simply reading it out. ---

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Change by Steve Dower : -- keywords: +patch pull_requests: +28362 stage: -> patch review pull_request: https://github.com/python/cpython/pull/30144 ___ Python tracker ___

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Ned Batchelder added the comment: Here's the experiment again with 3.10.1 and 3.11.0a3, and more ls's: $ python3.10 -V Python 3.10.1 $ python3.10 -m venv v310 $ ls -al v310/bin total 72 drwxr-xr-x 12 nedbatchelder wheel 384 Dec 16 06:42 ./ drwxr-xr-x 6 nedbatchelder wheel 192 Dec 1

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: Does the first venv's 'python' link to python3[.11]? If so, maybe _base_executable should be based on real_executable's filename rather than executable (that is, *after* resolving symlinks rather than before). I don't *think* that will cause any issues, and it

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Ned Batchelder added the comment: > I assume /private/tmp/bpo-46028/311-nested/bin/python3.11 exists though? Yes, that file exists, but it's a symlink to a non-existent file: $ ls -al 311-nested/bin total 0 drwxr-xr-x 5 nedbatchelder wheel 160 Dec 13 18:04 ./ drwxr-xr-x 6 nedbatchelder

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: Or possibly that error is coming from the attempt to copy it? And since both executable and base_executable don't have the 3/3.x suffix, the copy is failing because the "real" binary does have the suffix. This could be corrected in getpath.py with a platform-sp

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: > $ v311/bin/python -m venv 311-nested > Error: [Errno 2] No such file or directory: > '/private/tmp/bpo-46028/311-nested/bin/python' I assume /private/tmp/bpo-46028/311-nested/bin/python3.11 exists though? Probably that's the issue here - I don't know how else

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Ned Batchelder added the comment: Demonstration of a problem with only stdlib, and no undocumented features. This is on a Mac: This works: $ python3.10 -V Python 3.10.0 $ python3.10 -m venv v310 $ v310/bin/python -m venv v310-nested $ v310-nested/bin/python -V Python 3.10.0 This does not

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: > Because we never actually use the executable referenced by the 'home' path in > a pyvenv.cfg. This is actually not true on Windows or (I believe) some situations on macOS, where we need to use a redirecting launcher to actually launch the binary specified in

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: > But the value as it's calculated now seems to give a file that doesn't exist > - how can that be correct? Because we never actually use the executable referenced by the 'home' path in a pyvenv.cfg. It's only used as a starting point to locate a couple of entr

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: Is sys._base_executable correct without a venv? It should be the same as sys.executable in that case. venv calculates 'home' here: Lib/venv/__init__.py#L117 executable = sys._base_executable dirname, exename = os.path.split(os.path.abspath(execu

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Ned Batchelder added the comment: This started because a set of tests in the coverage.py test suite fail with 3.11.0a3. They attempt to create a venv in the test, and the test is already running inside a virtualenv. The venv creation fails, with a reference to a non-existent file. I wrote

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Vinay Sajip added the comment: > This is the intended behaviour, and yes it's changed from previous versions > ... The previous value was incorrect, hence it was marked as an internal > field. But the value as it's calculated now seems to give a file that doesn't exist - how can that be cor

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: I'm downgrading this from release blocker. If Vinay thinks there's a venv-related release blocker here he's welcome to raise the priority again, but I only see an intentional change to an internal value. Tools relying on internal fields will have to adapt for 3

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: > $ cat venv_a3/pyvenv.cfg > home = /home/ss/.pyenv/versions/3.11.0a3/bin > $ venv_a3/bin/python -c "import sys,os.path; print(e := > sys._base_executable); print(os.path.exists(e))" > /home/ss/.pyenv/versions/3.11.0a3/bin/python > True This is the intended beh

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Saaket Prakash added the comment: But on windows with the python.org installer, the behavior is same for both both a2 and a3. # With a3 installed > py -m venv venv_a3 > venv_a3/Scripts/python -c "import sys,os.path; print(e := > sys._base_executable); print(os.path.exists(e))" C:\Users\ss\A

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Saaket Prakash added the comment: I tried the same stuff as nedbat on WSL2, and I see similar change in the path of sys._base_executable (though I get a different "base" path on a3, so the path exists even there). $ ~/.pyenv/versions/3.11.0a2/bin/python -m venv venv_a2 $ ~/.pyenv/versions/3.

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Ned Batchelder added the comment: The two venvs seem analogous: $ cat venv_a2/pyvenv.cfg home = /usr/local/bin include-system-site-packages = false version = 3.11.0 $ ls -al venv_a2/bin total 72 drwxr-xr-x 13 nedbatchelder wheel 416 Dec 11 10:43 ./ drwxr-xr-x 6 nedbatchelder wheel 1

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: What's the contents of the pyvenv.cfg in these cases? It looks like the first case is definitely wrong, because the base executable should not be in "venv_a2" (that's sys.executable), but I don't know where it should be on your system. --

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Ned Batchelder added the comment: Tox isn't needed, just venv from the stdlib: $ python3.11.0a2 -m venv venv_a2 $ venv_a2/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))" /private/tmp/venv_a2/bin/python True $ python3.11.0a3 -m venv venv_a3 $

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Steve Dower added the comment: I'm going to need a decent amount of time to learn all of these components, because I never use this OS, Tox, nor virtualenv :) I'll try and get to it, but don't hold your breath. Luckily, Modules/getpath.py is much easier to follow and modify than the old syst

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Pablo Galindo Salgado added the comment: Indeed, seems my original hunch is correct. Steve, could you take a look when you have some time? -- ___ Python tracker ___ _

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Ned Batchelder added the comment: git bisect also identifies that commit as the first bad: 99fcf1505218464c489d419d4500f126b6d6dc28 is the first bad commit commit 99fcf1505218464c489d419d4500f126b6d6dc28 Author: Steve Dower Date: Fri Dec 3 00:08:42 2021 + bpo-45582: Port getpath[p

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Christian Heimes added the comment: Commit 9f2f7e42269db74a89fc8cd74d82a875787f01d7 has correct _base_executable $ venv/bin/python Python 3.11.0a2+ (heads/bpo-45847-simple-115-g9f2f7e42269:9f2f7e42269, Dec 10 2021, 10:09:54) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux Type "help", "copy

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Pablo Galindo Salgado added the comment: Ned, are you able to bisect this or provide a simpler reproducer that doesn't involve tox? -- ___ Python tracker ___

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Change by Pablo Galindo Salgado : -- priority: normal -> release blocker ___ Python tracker ___ ___ Python-bugs-list mailing list Un

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

Pablo Galindo Salgado added the comment: Steve, could this be related to the changes in getpath? -- nosy: +steve.dower ___ Python tracker ___ _

[issue46028] 3.11.0a3: under tox, sys._base_executable is wrong

New submission from Ned Batchelder : Under tox, sys._base_executable is not an actual file for 3.11.0a3. It was fine in 3.11.0a2. To reproduce: --- 8< # tox.ini [tox] envlist = py{310,311a2,311} skipsdist = True [testenv] commands = python -c "import sys,os.path; pr