adriangb commented on code in PR #18627:
URL: https://github.com/apache/datafusion/pull/18627#discussion_r2520052544
##########
datafusion/datasource-json/src/source.rs:
##########
@@ -108,15 +112,33 @@ impl FileSource for JsonSource {
object_store: Arc<dyn ObjectStore>,
base_config: &FileScanConfig,
_partition: usize,
- ) -> Arc<dyn FileOpener> {
- Arc::new(JsonOpener {
+ ) -> Result<Arc<dyn FileOpener>> {
+ // Get the projected file schema for JsonOpener
+ let projected_schema = if let Some(split_projection) =
&self.projection {
+ // Project the file schema to only the needed columns
+ let file_schema = self.table_schema.file_schema();
+ Arc::new(file_schema.project(&split_projection.file_indices)?)
+ } else {
+ // No projection - use the full file schema
+ Arc::clone(self.table_schema.file_schema())
Review Comment:
TODO: do we need to dynamically generate a "full" projection that includes
table partition columns? Or should that be the default that each FileSource
initializes itself with (then we don't even need to check here!)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
