parthchandra commented on code in PR #1756:
URL: https://github.com/apache/datafusion-comet/pull/1756#discussion_r2101451928
##########
native/core/src/parquet/parquet_exec.rs:
##########
@@ -61,12 +63,14 @@ pub(crate) fn init_datasource_exec(
file_groups: Vec<Vec<PartitionedFile>>,
projection_vector: Option<Vec<usize>>,
data_filters: Option<Vec<Arc<dyn PhysicalExpr>>>,
+ default_values: Option<HashMap<usize, ScalarValue>>,
session_timezone: &str,
) -> Result<Arc<DataSourceExec>, ExecutionError> {
let (table_parquet_options, spark_parquet_options) =
get_options(session_timezone);
- let mut parquet_source =
ParquetSource::new(table_parquet_options).with_schema_adapter_factory(
- Arc::new(SparkSchemaAdapterFactory::new(spark_parquet_options)),
- );
+ let mut parquet_source =
+
ParquetSource::new(table_parquet_options).with_schema_adapter_factory(Arc::new(
+ SparkSchemaAdapterFactory::new(spark_parquet_options,
default_values),
Review Comment:
I don't think it makes sense to do that even though it might make the code.
a little bit simpler. `default_values` are not exactly options. But I'm not
going to argue if you choose to do it that way.
##########
native/core/src/parquet/mod.rs:
##########
@@ -715,6 +715,7 @@ pub unsafe extern "system" fn
Java_org_apache_comet_parquet_Native_initRecordBat
file_groups,
None,
data_filters,
+ None,
Review Comment:
It's handled in the `ConstantColumnReader` which is shared between
native_comet and native_iceberg_compat.
Also see `ResolveDefaultColumns.getExistenceDefaultValues`. Not quite sure
what the difference between ExistenceDefaultValues and simply default values is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
