Mryange opened a new pull request, #70470: URL: https://github.com/apache/doris/pull/70470
### Background The JSON reader currently converts simdjson values to columns through a type-dispatch chain. JSONB objects and arrays are serialized to JSON text and parsed again before insertion. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Refactor Problem Summary: Move simdjson-to-column conversion into `DataTypeSerDe::deserialize_one_cell_from_simdjson`. Complex types recursively use their child SerDes, and JSONB objects and arrays are encoded directly from the parsed subtree. Scalar types retain the existing text conversion, including boolean coercion. ### Details - **Change map:** 500 changed production lines across the SerDe interface/implementations, reader integration, read context, and JSONB parser helpers; 324 added test/fixture/result lines; no generated source or documentation changes. - **Moves vs. logic changes:** Move Array/Map/Struct/Nullable dispatch into SerDes. Add a default text fallback and a direct JSONB subtree encoder. Keep the string cache and writer in reader-owned context for overlapping JSONPaths. - **Behavior change:** No intended user-visible semantic change; legacy scalar conversions and error propagation are retained. - **Compatibility:** No storage format, protocol, or configuration changes. Independent of parser-reuse changes. - **Performance impact:** Removes the JSON text/reparse round trip for JSONB objects and arrays. No benchmark claim; the common SerDe header now includes simdjson. - **Rollback:** Revert this PR. ### Testing Validation on the development machine, before the final adjustments: - ASAN BE/FE builds passed at `c44d740953a`. - `run-be-ut.sh --run -j48 --filter="SimdJsonSerDeTest.*:JsonReader*Test.*:JsonbParserTest.*:DataType*SerDeTest.*"`: 177 tests from 16 suites passed after rollback removal at `bc8929bca2c`. - Regression suites `test_simdjson_column_deserialization` and `test_load_json_with_jsonpath` passed on the earlier build. The new regression output was generated with the regression runner. - Clang-tidy passed after rollback removal. Final adjustment: clang-format 16 and `build-support/check-build-hygiene.sh` passed. Builds, unit tests, regression tests, and clang-tidy have not been rerun on final HEAD after the length-check removal, local struct lookup, numeric fallback, and wrapper removal. ### Release note None ### Check List (For Author) - Behavior changed: No intended user-visible change - Does this need documentation: No - Pre-submission status: Partially verified: final HEAD requires compilation and test validation. ### AI Disclosure - Tool: Codex - Generated: Implementation, tests, and PR description. - Human review: The author reviewed the main implementation and requested interface and compatibility adjustments; BE unit tests were not manually reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
