Rich-T-kid commented on code in PR #27500:
URL: https://github.com/apache/datafusion/pull/27500#discussion_r4620151407
##########
datafusion/datasource-parquet/src/metadata.rs:
##########
@@ -447,27 +502,18 @@ impl<'a> DFParquetMetadata<'a> {
})
.unwrap_or(schema);
- let schema = if self.enable_rle_to_dictionary {
+ let schema = if self.enable_rle_to_dictionary
+ || self.dictionary_read_max_distinct_values.is_some()
+ {
let schema_descr = file_metadata.schema_descr();
- // Top-level columns that have a dictionary page in at least one
row group.
- let dict_cols: HashSet<String> = metadata
- .row_groups()
- .iter()
- .flat_map(|rg| {
- rg.columns()
- .iter()
- .enumerate()
- .filter_map(|(col_idx, col)| {
- col.dictionary_page_offset()?;
- let col_desc = schema_descr.column(col_idx);
- let parts = col_desc.path().parts();
- // Skip nested columns: their leaf name doesn't
match the
- // Arrow top-level field name.
- (parts.len() == 1).then(|| parts[0].clone())
- })
- })
- .collect();
- if dict_cols.is_empty() {
+ let dict_cols = dictionary_promotion_candidates(
+ &metadata,
+ schema_descr,
+ self.dictionary_read_max_distinct_values,
+ );
+ if self.dictionary_read_max_distinct_values.is_some() {
+ set_dictionary_columns(&schema, &dict_cols).unwrap_or(schema)
Review Comment:
addressed in f323f0265. Per-file inference now only promotes eligible plain
fields, so dictionaries declared by the embedded Arrow schema are preserved.
The cross file pass separately demotes fields that are dictionary in one file
and plain in another. I also added the suggested regression case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]