Rich-T-kid commented on issue #24113: URL: https://github.com/apache/datafusion/issues/24113#issuecomment-6938489270
Reopening this because my earlier conclusion was too broad. The mixed `RLE_DICTIONARY`/plain-page failure mode is real, but it can be guarded independently from NDV. The new `dictionary_read_max_distinct_values` option fails closed unless every row group has NDV statistics and page-encoding statistics prove that every data page is dictionary encoded. It then compares the configured cap against the sum of row-group NDVs, a conservative per-file upper bound, and rechecks each file when it is opened. That keeps the user-controlled threshold requested here while excluding the encoding transition that caused the earlier regressions. I have also rerun the full TPC-H and TPC-DS SF1 matrices plus focused confirmation runs; no regression above 5% reproduced in both independent pairs. The PR will include the full benchmark method and results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
