nsivabalan commented on PR #18941: URL: https://github.com/apache/hudi/pull/18941#issuecomment-4936976092
**Perf validation — comparator microbenchmark (`String.compareTo` vs `StringUtils.compareUtf8Bytes`)** Ran a single-thread microbenchmark on Java 11 comparing the two comparators across the workloads used by the MDT bulk-insert partitioners, HFile log-block sort, and RI lookup key sort. Measured at 1K, 64K, and **2M keys** (a single RLI shard holds 2M–5M entries in practice, so 2M is the shard-scale sanity check). **Headline: this is a correctness fix that also happens to be a large perf win on realistic MDT record keys.** For ASCII-UUID keys (the realistic MDT record-key shape — high entropy in the first byte), the UTF-8 comparator is **7.8× faster** on a 2M-key sort: | n | String.compareTo | compareUtf8Bytes | speedup | |---:|---:|---:|---:| | 1K sort | 1.45 ms | 0.16 ms | **9×** | | 64K sort | 167 ms | 19 ms | **8.7×** | | 2M sort | 10.2 s | 1.3 s | **7.8×** | Why UTF-8 wins on UUIDs: `compareUtf8Bytes` bottoms out in `Arrays.compareUnsigned(byte[], byte[])`, a JIT intrinsic that uses SIMD (SSE2/AVX) to compare 16–32 bytes per instruction. `String.compareTo` on Java 11 walks UTF-16 chars serially with no SIMD. High-entropy first bytes mean most compares finish in the first vector step. **Full sweep at 2M keys / RLI shard scale:** | key shape | String sort | UTF-8 sort | delta | |---|---:|---:|---:| | ASCII-UUID (realistic MDT key) | 10.2 s | **1.3 s** | **−8.9 s (−87%)** | | BMP CJK | 4.5 s | 2.3 s | −49% | | Supplementary (fix's correctness target) | 4.3 s | 2.6 s | −40% | | ASCII-numeric (long shared-prefix — only regression) | 0.9 s | 1.4 s | +0.5 s (+60%) | **Only regression shape:** long shared-prefix ASCII (e.g. `user_0000000123` / `user_0000000456`). Two `getUTF8Bytes` allocations per compare dominate when JIT'd `String.compareTo` is already very fast on ASCII prefixes. Cost is +0.5–0.6 s per 2M-key shard, negligible vs HFile-write / shuffle costs at that scale, and not a realistic RI record-key pattern. **No regression at seconds granularity** for any shape this fix targets — even the pathological common-prefix numeric adds under a second at 2M keys. Full table (PAIRWISE + SORT + TREESET × 4 shapes × 3 sizes) is in a local dev-only bench file (`BenchStringComparators`, `@Disabled` by default); happy to land it separately if useful, but it's not part of this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
