eldenmoon opened a new pull request, #70730: URL: https://github.com/apache/doris/pull/70730
### Background Extended DESCRIBE queries fetch inferred VARIANT subcolumns from sampled tablets on the backends. This applies to both the table and partition forms. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: A slow schema response can hold the DESCRIBE read lock long enough for a Stream Load commit on the same table to fail. Capture the catalog metadata under the read lock and fetch the remote schema after releasing it. On a shared-nothing cluster, set `describe_extend_variant_column = true`, pause the schema response for `DESC t` or `DESC t PARTITION (p1)`, and run Stream Load against `t` from a separate session. On the baseline both loads returned `get tableList write lock timeout`. With this fix both loads finished while their respective DESCRIBE statements were still waiting. ### Details - **Frequency:** Intermittent in practice. The controlled reproduction held the DESCRIBE read lock past the commit deadline and reproduced the timeout for both statement forms. Production frequency is unknown. - **Root cause:** The outer DescribeCommand read lock encloses both forms of the remote schema fetch. - **Why existing tests missed it:** `ShowExecutorTest` and `IndexSchemaProcNodeTest` cover statement behavior and schema rendering. `regression_test_variant_desc` checks result contents. None of these tests pause a backend schema response while a Stream Load commits to the same table. - **Fix:** Snapshot the base schema rows and tablet replica locations under the read lock. Fetch and append the inferred columns outside it. - **Other paths and branches:** The master fix covers both forms of DESCRIBE and shares the snapshot logic with SHOW PROC. The same outer lock is present on branch-4.1. Other release branches and cloud runtime were outside this validation. - **Behavior change:** Stream Load can commit while extended DESCRIBE is waiting for backend schema metadata. ### Testing On dev-x, the ASAN master baseline a5d5a52b3f reproduced the commit write-lock timeout with both forms of extended DESCRIBE. Both runs spent 10,001 ms in commit and publish. The candidate passed the same regression with successful loads while DESCRIBE was still paused. The commit and publish times were 29 ms for plain DESCRIBE and 18 ms for the partition form. The existing regression_test_variant_desc suite passed. IndexSchemaProcNodeTest and ShowExecutorTest finished with 19 passes and one skip. The affected-suite sweep covered 26 regression suites. Twenty-one passed on the first run. Three passed unchanged after enabling BE debug points. The two dotted-key suites passed with temporary test copies that read the invisible session variable through `SELECT @@default_variant_enable_nested_group`. The existing DESCRIBE assertions and goldens ran unchanged in those copies. ### Release note Extended DESCRIBE no longer holds the table lock while waiting for backend schemas, allowing concurrent Stream Load commits to proceed. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Implementation, regression suite, and verification notes. - Human review: No author review statement was provided. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
