[issue42041] venv subprocess call to python resolves to wrong interpreter

Change by Eryk Sun : -- resolution: -> fixed stage: -> resolved status: open -> closed ___ Python tracker ___ ___ Python-bugs-list

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: New changeset 5ab27cc518f614a0b954ff3eb125290f264242d5 by Paul Moore in branch 'master': bpo-42041: Clarify how subprocess searches for the executable (GH-22715) https://github.com/python/cpython/commit/5ab27cc518f614a0b954ff3eb125290f264242d5 -- _

[issue42041] venv subprocess call to python resolves to wrong interpreter

Mark Keller added the comment: +1 to what Steve said -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: I like that, I've updated the PR accordingly. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue42041] venv subprocess call to python resolves to wrong interpreter

Steve Dower added the comment: Proposed alternative based on Eryk's: For maximum reliability, use a fully-qualified path for the executable. To search for an unqualified name on :envvar:`PATH`, use :meth:`shutil.which`. On all platforms, passing :data:`sys.executable` i

[issue42041] venv subprocess call to python resolves to wrong interpreter

Eryk Sun added the comment: > My biggest concern with the suggested wording ... is that it > reintroduces the issue with the redirector. I thought I addressed that in second paragraph by recommending sys.executable. It could be emphasized that running "python[x][.y]" is unreliable, without

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: My biggest concern with the suggested wording (which in broad terms looks OK to me) is that it reintroduces the issue with the redirector. By specifically saying that the behaviour in `CreateProcess` applies, we lead the user to the statement that the search pat

[issue42041] venv subprocess call to python resolves to wrong interpreter

Eryk Sun added the comment: > I'd read it because I'm interested, but that is probably too much detail > for someone who is trying to get something done quickly. I did say that I don't know how much should be documented. I tend to dump a lot of information in issues so that people can make in

[issue42041] venv subprocess call to python resolves to wrong interpreter

Mark Keller added the comment: Paul explained precisely of what I was going through when I reached out to you with this: > then the "running executable" is "path/to/venv/scripts/python.exe", and so > the path containing the running executable is "path/to/venv/scripts", so a > search for "py

[issue42041] venv subprocess call to python resolves to wrong interpreter

Eryk Sun added the comment: > For platform semantics, I'd prefer a link to the CreateProcessW docs, > with advice to read about the lpApplicationName parameter with > respect to `executable` and lpCommandLine with respect to `args` > and the platform search semantics. For example, let's he

[issue42041] venv subprocess call to python resolves to wrong interpreter

Eryk Sun added the comment: > You can build a cross-platform wrapper on top of native behaviour > (witness `shutil.which`) but you can't do the opposite. Nothing would prevent adding a parameter to use the platform semantics, if that had been in the design from the outset. But it's not somet

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: Exactly. Why isn't the current directory searched? Why isn't foo/bar searched for on PATH? Why is it possible for the user to accidentally remove system commands from PATH and lose access to them? Any system is confusing and surprising to users only familiar wit

[issue42041] venv subprocess call to python resolves to wrong interpreter

Eryk Sun added the comment: > Also, why would we document the Windows rules, but not the POSIX > rules? They are arguably just as strange to someone who doesn't > know them. POSIX rules are simply to search PATH for the filename as passed, and if it has a slash in it, the path is resolved a

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: Well, I'm not convinced that using `shutil.which` is the right answer for anything other than "python". And even there, I'd recommend using `sys.executable` over `shutil.which` in pretty much every case. My view is that if we ignore the "weirdness" introduced by

[issue42041] venv subprocess call to python resolves to wrong interpreter

Eryk Sun added the comment: > I don't think we should document this level of detail But a lot of it -- most of it -- is also strange behavior that no one would expect without reading about it somewhere. Most users of subprocess.Popen() will never wade through the documentation of CreateProce

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: I don't think we should document this level of detail, both because it's basically Windows standard behaviour and therefore not up to us to document, and because it's *way* too overwhelming for the average user. -- __

[issue42041] venv subprocess call to python resolves to wrong interpreter

Eryk Sun added the comment: I don't know how much should be documented for subprocess.Popen, but here are the details for how searching works with shell=False (default) and shell=True. For shell=False, the search path used by WinAPI CreateProcessW checks %__APPDIR__%; %__CD__% (unless %NoDef

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: OK, PR updated as per discussion. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubsc

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: > We'd need Mark to weigh in (or possibly track down and ask a colleague), but > I expect the assumption is that subprocess.Popen("python") searches PATH > first. I'm sure it was, TBH. The executable directory side of things is really just on digging a bit deep

[issue42041] venv subprocess call to python resolves to wrong interpreter

Steve Dower added the comment: > I'm not asking that we guarantee any behaviour, or that we commit ourselves > to anything. Just that we note that people ought not to be making a specific > assumption Which is a fine intent, it's just the running assumption is that if it's in the documentati

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: > This is still true, though, as much as it's ever been. But it's *not*, if I understand the problem here. Windows resolves a bare "python" command by searching the directory containing the running executable first. I would reasonably assume that if I type

[issue42041] venv subprocess call to python resolves to wrong interpreter

Steve Dower added the comment: > The bit that I *do* think is a venv gotcha is that it's entirely reasonable > for a user to expect that if they run path\to\venv\Scripts\python.exe, then > their Python script will be run by that executable. This is still true, though, as much as it's ever be

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: Fair enough. I think we have to be careful here - people do rely on how the "internal" aspects of virtual environments work, like it or not, and we've definitely broken people's code in the past as a result of taking a hardline "it's not documented, so it's not

[issue42041] venv subprocess call to python resolves to wrong interpreter

Steve Dower added the comment: Thanks for doing the PR, Paul. Honestly, I'd rather document in subprocess that Popen("python") is not a good idea, and should use either sys.executable or shutil.which depending on what you mean. This is a platform-independent gotcha, as it's very easy to repr

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: Eryk - I suggest that we modify (or remove) the note in venv that my PR adds as part of such a change, rather than try to qualify the text now (which would probably only make it harder for people to understand the issue). --

[issue42041] venv subprocess call to python resolves to wrong interpreter

Mark Keller added the comment: This sounds good to me Paul, thank you again! -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue42041] venv subprocess call to python resolves to wrong interpreter

Eryk Sun added the comment: PR 22715 is accurate for the current implementation of subprocess. There has been discussion in past issues about implementing a PATH search that's similar to what the CMD shell does -- as in calling os.system() or Popen() with shell=True -- instead of relying on

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: I've created https://github.com/python/cpython/pull/22715 as a suggested documentation fix. Mark, does this cover what you would have needed to know? Steve, is what I've written technically accurate? -- stage: patch review -> __

[issue42041] venv subprocess call to python resolves to wrong interpreter

Change by Paul Moore : -- keywords: +patch pull_requests: +21683 stage: -> patch review pull_request: https://github.com/python/cpython/pull/22715 ___ Python tracker ___ _

[issue42041] venv subprocess call to python resolves to wrong interpreter

Mark Keller added the comment: Thank you so much for the prompt responses. I would say that this should be documented on the venv page (https://docs.python.org/3/library/venv.html) as I find the subprocess page too dense of information already, but I do agree that venv page is most likely no

[issue42041] venv subprocess call to python resolves to wrong interpreter

Bernat Gabor added the comment: Taking a look at the existing documentation this should probably be mentioned under the args part of https://docs.python.org/3/library/subprocess.html#popen-constructor, not? (that's where other similar gotchas are explained too) -- _

[issue42041] venv subprocess call to python resolves to wrong interpreter

Paul Moore added the comment: Agreed, this is just "normal Windows behaviour". It's definitely surprising and non-obvious (I saw this issue and it never even occurred to me that this was what was happening) so I'd support documenting it somewhere. My instinct is that the subprocess documenta

[issue42041] venv subprocess call to python resolves to wrong interpreter

Steve Dower added the comment: This is due to how Windows resolves relative paths when creating a new process. It *always* looks in the current application directory first, which with this setup will be the original Python executable rather than the venv redirector. The best fix is to pass s

[issue42041] venv subprocess call to python resolves to wrong interpreter

Bernat Gabor added the comment: I wonder if this is an interaction issue with the builtin Windows provided python executable? -- nosy: +eryksun, gaborjbernat ___ Python tracker _

[issue42041] venv subprocess call to python resolves to wrong interpreter

New submission from Mark Keller : Hey, Wanted to report this weird behavior I saw recently. Let me first explain how to reproduce and then talk about where I think the issue comes from. I attached a tar.gz file with 2 Python files in it. Here's how to reproduce: 0. Make sure that pytest is u