linrrzqqq opened a new pull request, #70786:
URL: https://github.com/apache/doris/pull/70786

   ### Background
   
   The backend discovers Python environments during startup. In venv mode, it 
uses the full version of each configured interpreter to resolve an environment 
directory. `SHOW PYTHON PACKAGES` asks each live backend to list the packages 
for the requested interpreter.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #59543 (introduced venv discovery), #60751 (introduced the 
package-list command)
   PR Type: Bugfix
   
   Problem Summary:
   
   Venv discovery provisioned another environment when the expected path was 
missing. With verbose logging enabled, pip's upgrade check appended debug 
output to the package list and Doris rejected the JSON. Discovery now reports 
the missing executable and its expected environment directory. Package listing 
disables the upgrade check.
   
   ### Details
   
   - **Frequency** — Venv discovery took the creation path whenever the 
versioned interpreter was missing. With pip 23.2.1 and `PIP_VERBOSE=2`, the 
enabled upgrade check adds debug output to stdout.
   - **Root cause** — `scan_from_venv_root_path` created the versioned 
directory and invoked the configured interpreter's venv module. 
`list_installed_packages` ran pip with its upgrade check enabled.
   - **Why existing tests missed it** — The previous venv unit test expected 
automatic provisioning. The package-list unit tests supply pip output, and the 
SQL suite checks the returned package rows.
   - **Fix** — Require the versioned interpreter to exist and pass 
`--disable-pip-version-check` on the JSON listing command.
   - **Other paths and branches** — Shared-nothing and cloud backends use the 
same scanner and package listing. Both original code paths appear in the local 
`origin/branch-4.1` and `origin/branch-4.2` refs.
   - **Behavior change** — Each configured venv must exist at the versioned 
path before BE startup.
   
   ### Testing
   
   The local comparison used Python 3.12.12, pip 23.2.1, `PIP_VERBOSE=2`, PyPI, 
and a fresh pip cache in each run. The BE used `python_env_mode = venv`, 
`python_venv_interpreter_paths = /usr/bin/python3`, and `python_venv_root_path` 
pointing to the directory shown as `$VENV_ROOT` below. The same SQL ran in each 
case.
   
   ```sql
   SHOW PYTHON PACKAGES IN '3.12.12';
   ```
   
   **Before the fix and an empty venv root (`6e7bea93c53`)**
   
   The BE created `$VENV_ROOT/python3.12.12` during startup. `SHOW PYTHON 
VERSIONS;` returned the environment it had created.
   
   ```text
   Version      EnvName EnvType BasePath        ExecutablePath
   3.12.12      python3.12.12   venv    $VENV_ROOT/python3.12.12        
$VENV_ROOT/python3.12.12/bin/python
   ```
   
   `SHOW PYTHON PACKAGES` returned this error. The BE also logged the JSON 
parse failure.
   
   ```text
   ERROR 1105 (HY000) at line 1: errCode = 2, detailMessage = [E6] Failed to 
parse pip list json output for python version: 3.12.12
   ```
   
   In a direct run of `python -m pip list --format=json`, the first line after 
the JSON array was `1 location(s) to search for versions of pip:`. `python -m 
json.tool` reported `Extra data: line 2 column 1`. With 
`--disable-pip-version-check`, the output parsed as JSON.
   
   **With the fix and an empty venv root (`5a0adb52cf2`)**
   
   The expected environment directory remained absent. The startup log reported 
the missing interpreter, with the local venv root abbreviated here.
   
   ```text
   Failed to initialize python version manager: [NOT_FOUND]Python virtual 
environment executable not found: $VENV_ROOT/python3.12.12/bin/python. Create 
the environment at $VENV_ROOT/python3.12.12 before starting BE.
   ```
   
   ```text
   ERROR 1105 (HY000) at line 1: errCode = 2, detailMessage = No alive backends 
found to get python packages
   ```
   
   **With the fix and a provisioned venv (`5a0adb52cf2`)**
   
   Provisioning the environment and restarting BE let the same SQL return these 
rows. The pip self-check cache remained absent.
   
   ```bash
   /usr/bin/python3 -m venv --system-site-packages "$VENV_ROOT/python3.12.12"
   ```
   
   ```text
   Package      Version
   pip  23.2.1
   setuptools   68.2.2
   ```
   
   **Other local checks**
   
   - The full FE and BE builds passed for both commits. clang-format 16, 
clang-tidy, and the backend build hygiene checks passed for the fix.
   - Both targeted `PythonEnvTest` cases passed with the coverage-enabled 
runner.
   - `test_show_python_packages_command` passed on the isolated cluster. The 
run executed one suite with zero failures and zero skips.
   
   ### Release note
   
   Provision every configured Python virtual environment under 
`python_venv_root_path/python<full-version>` before starting the backend. `SHOW 
PYTHON PACKAGES` skips pip's upgrade check.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No. The existing Python UDF documentation 
directs operators to create the versioned venv and install dependencies before 
use.
   - Pre-submission status: Ready
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: Backend changes, unit test changes, and this draft
   - Human review: not stated
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to