corwinjoy commented on code in PR #16351:
URL: https://github.com/apache/datafusion/pull/16351#discussion_r2136718671
##########
datafusion/common/src/config.rs:
##########
@@ -591,6 +930,12 @@ config_namespace! {
/// writing out already in-memory data, such as from a cached
/// data frame.
pub maximum_buffered_record_batches_per_stream: usize, default = 2
+
+ /// Optional file decryption properties
+ pub file_decryption_properties:
Option<ConfigFileDecryptionProperties>, default = None
+
+ /// Optional file encryption properties
+ pub file_encryption_properties:
Option<ConfigFileEncryptionProperties>, default = None
Review Comment:
@adamreeve and I are not completely sure where these settings should go. On
the session context there's only a way to set the "global" `ParquetOptions` but
not `TableParquetOptions`, which contains extra table-specific settings.
It does feel a bit wrong to put file-specific decryption properties in the
execution context (see later examples). Eg. if users were reading two different
encrypted Parquet files in one query they might need to set different
decryption properties for each file, so setting them in the execution context
wouldn't work. At the moment I think this scenario would require creating
separate listing tables and specifying `TableParquetOptions`. That's an edge
case so maybe I'm overthinking this, but maybe being able to set file
decryption properties in `ParquetReadOptions` would be a good idea?
This doesn't really fit all that well with the reader options that Parquet
has, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
