hubgeter opened a new pull request, #70802:
URL: https://github.com/apache/doris/pull/70802

   ### Background
   
   Doris binds storage properties in file queries, exports and loads before 
opening a filesystem. For OUTFILE, the FE chooses the writer and passes a path 
and an optional suffix to the BE, which generates the output filename.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   
   Related PR: #24334 (introduced the OUTFILE suffix option)
   
   PR Type: Bugfix
   
   Problem Summary:
   
   A statement could override the Hadoop implementation or redirect a remote 
filesystem to local storage. OUTFILE also accepted path separators in its 
suffix. The FE now checks options supplied by the statement before binding 
storage, verifies that the provider matches the requested operation, and 
constrains the suffix to a single filename component.
   
   ### Details
   
   - **Change map:** 205 changed lines of FE code cover statement options, 
provider selection and output validation. The other 141 changed lines cover FE 
and SQL cases, including an existing HDFS export fixture.
   - **Frequency:** Confirmed by tracing statement analysis and writer 
selection in the source. Runtime frequency was not measured.
   - **Root cause:** HDFS bindings accepted broad Hadoop configuration 
namespaces. OUTFILE checked the URI prefix before properties selected the 
backend, and the writer appended the suffix verbatim.
   - **Why existing tests missed it:** OutFileTest checked the HDFS default URI 
and object-store destinations. The file-suffix and ViewFS regression cases 
exercised valid extensions and remote mounts. They lacked negative cases for 
provider selection, mount targets and filename boundaries.
   - **Fix:** Check raw SQL options before resource expansion and provider 
binding. Allow standard HA providers and direct remote ViewFS links. Check 
resource usage before reading its configuration. Match readers and writers to 
the resolved provider, including the filesystem used for pre-write deletion.
   - **Other paths and branches:** The checks cover file TVFs, INSERT INTO TVF, 
OUTFILE and the shared WITH storage clause for EXPORT and LOAD. Named broker 
destinations retain broker-side path handling. Source inspection found the 
original Hadoop passthrough and unchecked suffix on branch-3.1, branch-4.0, 
branch-4.1 and branch-4.2.
   - **Behavior change:** Unsupported per-statement Hadoop options produce an 
analysis error. Native local OUTFILE requires the canonical local URI and the 
FE local-output setting. Compound extensions such as `csv.gz` remain valid.
   
   ### Testing
   
   - Checkstyle 9.3 with the repository configuration passed on all 16 changed 
Java files. The FE metadata dependency check, Groovy syntax parsing of both 
changed suites, and `git diff --check` passed.
   - Added FE tests for accepted HA and ViewFS settings, rejected property 
overrides, storage-kind mismatches, and valid and invalid suffixes. Added SQL 
cases covering a file query, OUTFILE, INSERT INTO TVF and EXPORT.
   - No local build, FE unit test execution or cluster regression run was 
performed. The Maven validation phase could not resolve uncached dependency 
metadata offline. The style checker and metadata dependency check ran 
independently.
   
   ### Release note
   
   File statements reject unsupported Hadoop overrides and incompatible storage 
providers. Move custom Hadoop settings into operator-managed configuration. 
OUTFILE suffixes accept letters, digits, dots, underscores, plus signs and 
hyphens.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes. Update the file TVF and OUTFILE 
references with the statement options and allowed suffix characters.
   - Pre-submission status: Partially verified: static checks passed. FE unit 
and SQL runtime coverage await CI.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex (GPT-6)
   - Generated: Implementation, tests and PR description.
   - Human review: Pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to