lemire opened a new pull request, #70789: URL: https://github.com/apache/doris/pull/70789
### Background simdjson is a JSON parser built from source under `thirdparty/` and linked into the BE. The BE uses its On-Demand API to load JSON files and streams (`format_v2/json/json_reader.cpp`), to convert JSON text to JSONB (`util/jsonb_parser_simd.h`) and in the JSON functions (`exprs/json_functions.cpp`), and its DOM API to parse JSON into VARIANT (`util/json/simd_json_parser.h`). At startup the BE selects simdjson's AVX2 (`haswell`) kernel (`init_simdjson_parser`). ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Release/Dependency Problem Summary: Doris pins simdjson 3.11.6. simdjson 5.2.0 parses faster, mostly when walking documents. With the kernel Doris uses, BE-level measurements show JSON load 1 to 12% faster, JSONB conversion 6 to 18% faster and VARIANT parsing 3 to 4% faster; one JSON path lookup is 12% slower (see Notes for Reviewers). This PR upgrades simdjson to 5.2.0, ports Doris's `BOOL` to `BOOLEAN` patch, and adds the one `default` case the new version needs. ### Details - **Versions:** simdjson 3.11.6 → 5.2.0, from the GitHub tag tarball, with `SIMDJSON_MD5SUM` updated in `thirdparty/vars.sh`. Doris renames `dom::element_type::BOOL` to `BOOLEAN` in its own patch, so `simdjson-3.0.1.patch` is replaced by `simdjson-5.2.0.patch`, the same rename on the 5.2.0 sources, and `download-thirdparty.sh` applies it. The build flags in `build-thirdparty.sh` are unchanged. `dist/LICENSE-dist.txt` listed simdjson 3.0.1 and now lists 5.2.0. - **Upstream changes:** The release notes are at https://github.com/simdjson/simdjson/releases. The changes that matter to Doris: - `ondemand::json_type` gains an `unknown` value, so the top-level `switch` in `JsonbParser::parse` needs a `default` case to build with `-Werror`. It returns `InvalidArgument`. - Since 5.0, an integer in [2^64, 10^20) is reported as `BIGINT_ERROR` instead of `NUMBER_ERROR`. `JsonbParser` already handles both through `parse_number_success`, and no other BE code checks these codes. - Faster value copies in On-Demand (simdjson#3061), faster stage 1, and fixes in APIs that Doris does not use. - The license is still Apache-2.0. 5.2.0 also offers MIT and adds `internal/fast_float.h` (Apache-2.0 OR MIT OR BSL-1.0). - **Compatibility:** Built and tested on Linux x86_64 with the `apache/doris:build-env-ldb-toolchain-latest` toolchain (clang 20.1.7), in RELEASE and ASAN unit-test builds. ARM64, the GCC toolchain and macOS were not built. No stored data changes: simdjson only parses input. The BE build does not track third-party headers: an existing build directory must be rebuilt from scratch after the third-party update, or objects compiled against the old headers crash when linked with the new library. - **Rollback:** Revert this PR, rebuild the third-party libraries and do a clean BE build. No data is affected. ### Testing - `simdjson-5.2.0.patch` applies cleanly to the 5.2.0 tarball, and the patched library was built with the flags of `build_simdjson` into a copy of the `build-env-ldb-toolchain-latest` image. - RELEASE unit-test build (`BUILD_TYPE_UT=RELEASE ./run-be-ut.sh`), filter `*Json*:*JSON*:*Variant*:*VARIANT*:*Serde*:*SerDe*:*Jsonb*`: 1,486 tests, 1,471 pass with 3.11.6 and the same 1,471 with 5.2.0. The same 11 fail with both versions (`ColumnVariantV2Test` invariants, two `DataTypeSerDeDecodedValuesDeathTest` cases and `QuantileStateSerdeTest.writeColumnToPb`), when the saved binaries are run outside `run-be-ut.sh`. - ASAN unit-test build from scratch with 5.2.0 (`./run-be-ut.sh --clean --run`, same filter): 1,483 of 1,488 tests pass, 5 are skipped, none fail, and no sanitizer errors are reported. - The prebuilt third-party packages and the build-env image still contain 3.11.6 and need to be regenerated. ### Notes for Reviewers The speedups in Problem Summary come from a local performance test, not part of this PR. It was built with each simdjson version (RELEASE unit-test binaries, same toolchain) and the two binaries were run alternately on an Intel Xeon Gold 6548N, pinned to one core. Each number is the median of 3 runs, each run the fastest of 7 or 9 repetitions; runs differed by about 1 to 2%. Rows are log-like JSON objects of about 160 B, 270 B, 1.3 KB and 4.6 KB. The JSON reader cases read a 200,000-row file (40,000 rows for 1.3 KB) into 7 columns, one of them JSONB, or into 2 columns. | Workload | 3.11.6 | 5.2.0 | Speedup, `haswell` | Speedup, `icelake` | |---|---:|---:|---:|---:| | JSON reader, lines, 270 B, 7 columns | 1172 ns/row | 1130 ns/row | 1.04x | 1.06x | | JSON reader, lines, 270 B, 2 columns | 513 ns/row | 507 ns/row | 1.01x | 1.06x | | JSON reader, lines, 1.3 KB, 7 columns | 5364 ns/row | 4792 ns/row | 1.12x | 1.15x | | JSON reader, outer array, 270 B, 7 columns | 1134 ns/row | 1059 ns/row | 1.07x | 1.09x | | `JsonBinaryValue::from_json_string`, 160 B | 308 ns | 291 ns | 1.06x | 1.16x | | `JsonBinaryValue::from_json_string`, 270 B | 748 ns | 676 ns | 1.11x | 1.17x | | `JsonBinaryValue::from_json_string`, 4.6 KB | 16.8 us | 14.3 us | 1.18x | 1.20x | | `JsonFunctions::extract_from_object`, `$.msg`, 270 B | 85 ns | 97 ns | 0.88x | 1.03x | | `JsonFunctions::extract_from_object`, missing key, 270 B | 272 ns | 276 ns | 0.99x | 1.01x | | `JsonStringToVariantEncoder`, 160 B | 585 ns | 562 ns | 1.04x | 1.05x | | `JsonStringToVariantEncoder`, 270 B | 1500 ns | 1451 ns | 1.03x | 1.04x | | `JsonStringToVariantEncoder`, 1.3 KB | 9722 ns | 9305 ns | 1.04x | 1.07x | The time columns are for `haswell`, the kernel the BE selects. The `$.msg` lookup is 12% slower with `haswell` in the BE binary. The same steps in a standalone program (`padded_string`, `iterate`, `find_field_unordered("msg")`, same compiler and flags) are faster with 5.2.0, 118 ns to 105 ns with `haswell` and 95 ns to 79 ns with `icelake`, so the slowdown is probably code placement in the BE binary rather than more work in simdjson. ### Release note Upgrade simdjson to 5.2.0, which speeds up JSON loading, JSONB conversion and VARIANT parsing. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: built and tested on Linux x86_64 with clang only; ARM64 and GCC builds not run, and the prebuilt third-party packages are not regenerated. ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: code, the patch, and this description - Human review: all code reviewed by Daniel - Prompts: The author asked to work on a pull request that updates Doris to the latest simdjson and to quantify the performance benefits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
