Gabriel39 opened a new pull request, #66753: URL: https://github.com/apache/doris/pull/66753
### What problem does this PR solve? Issue Number: DORIS-27899 Problem Summary: The native Parquet schema validator rejected every group deeper than 100. That bound is below the physical depth produced by standard automatic Variant shredding: - Paimon 1.4.2 defaults `variant.shredding.maxSchemaDepth` to 50 ([source](https://github.com/apache/paimon/blob/release-1.4.2/paimon-api/src/main/java/org/apache/paimon/CoreOptions.java#L418-L424)). - Iceberg also caps automatic shredding at 50 logical levels ([source](https://github.com/apache/iceberg/blob/35889387c8c6ff0a3f57d17a304709dc1b7d9340/parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java#L79-L80)). - An object level adds a field wrapper and a `typed_value` wrapper, so 50 levels can reach group depth 101. - A nested array level adds `typed_value (LIST)`, repeated `list`, and `element` groups, so 50 levels can reach group depth 151. This PR raises the bound to 192. The value covers the 151-level nested-array case plus Doris's maximum nine enclosing logical type levels ([source](https://github.com/apache/doris/blob/branch-4.1/fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java#L52)), where list/map encodings may add up to three physical groups per level, and leaves a small compatibility margin. The limit remains a fixed reader-side safety bound. It is intentionally derived from the physical Parquet schema instead of a Paimon table option: the option controls schema inference, is not guaranteed to be present in the file footer, and explicit or third-party writers can produce the same layout. Validation still happens before recursive parsing or schema-driven allocation, and schemas deeper than 192 remain rejected. Two unit tests construct Paimon-style object and nested-array schemas at the default 50-level shredding depth. The existing boundary test also verifies that depth 192 is accepted and depth 193 is rejected. ### Release note Fix native Parquet reads of deeply shredded Variant schemas generated by standard writers. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Native Parquet readers now accept valid group nesting up to 192 instead of 100. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
