xylaaaaa commented on issue #69884: URL: https://github.com/apache/doris/issues/69884#issuecomment-6529364425
Draft implementation PR opened: #69904 (`xylaaaaa:codex/delta-m1-reader` → `master`). The PR contains 19 commits and is intentionally still a draft; the local tasklist records 60/64 completed items. Implemented: Kernel 4.4.1 FE connector, native FileScannerV2 reader, HMS/filesystem catalogs and Hive routing, mapped/evolved schemas, portable deletion vectors, checkpoint sidecars, pruning/count, fixed-TTL snapshot caching, streaming planning and EXPLAIN/profile diagnostics. Thrift field 16 was rechecked against current master and branch-3.1/4.0/4.1/4.2 before opening the PR. Verified locally: full FE/ASAN BE builds; 165 BE, 75 Delta connector, 2 metastore, 25 FE core and 274 Hive-related unit tests; eight p0 suites twice on the optional Spark/HMS/MinIO environment; million-file planning in a 512 MiB JVM; MinIO and ordinary HDFS storage regression. Test data is retained for reuse, and restarts preserved all 1,227 formal MinIO objects by path, size and SHA-256. Performance measurement: all 125 TPC-H/TPC-DS queries completed cross-reader comparison and three paired repetitions across official regression runs. Q7/Q85 use the same fixed join order on both readers after default Hive memory failures; Q30 adapts the generator's column, and floating values use the framework comparator. These controls and the original failures are recorded in the PR. On local ASAN -O0 SF1, summed per-query medians are 28.4% lower for Delta on TPC-H and 26.1% higher on TPC-DS. Nine profile pairs show join/filter/parallelism differences; performance parity is not yet accepted. Remaining: - [ ] Resolve the performance gaps within M1, including Q22 variability and small-table scheduling. - [ ] OSS/COS/OBS/Azure/GCS and Kerberos HDFS validation, pending environments as agreed. - [ ] EN/ZH website documentation PR and website validation (patches prepared). - [ ] Final M1 acceptance and checklist reconciliation. The tracking issue stays open while these items are completed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
