zhuqi-lucas commented on issue #627: URL: https://github.com/apache/parquet-format/issues/627#issuecomment-5885403906
cc @etseidl @alamb — this came out of the dictionary-deferral work in apache/arrow-rs#11168, which you both looked at. That PR removes the dictionary decompression for column chunks that decode nothing. The limitation is that it can only ever help those chunks: anything that decodes a value still pays, and deferral cannot change that. An uncompressed dictionary page would be the same knob for both cases, and a dictionary is the part of a chunk where compression pays least anyway, since duplicates are already gone before the codec sees it. @etseidl the question I would most value your read on is whether the omission from `DictionaryPageHeader` is deliberate. My guess in the issue is that it is substantive rather than an oversight — `DataPageHeaderV2.is_compressed` is safe only because V2 is already gated behind a writer version, and a dictionary page has no such gate — but you would know the history better than I can reconstruct it. @alamb mostly for whether there is appetite on the arrow-rs side, since the format change would need an implementation to go with it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
