mitchsw opened a new issue, #23463:
URL: https://github.com/apache/datafusion/issues/23463
### Is your feature request related to a problem or challenge?
I'd like to build a custom `DataSource` that reuses an existing `FileSource`
(e.g. `ParquetSource`) instead of reimplementing all its internals. Today this
is not possible without dragging along a `FileScanConfig` that my custom
`DataSource` doesn't otherwise need.
The architecture diagram in `datafusion/datasource/src/source.rs` shows a
clean top-down layering:
```text
DataSourceExec ──► DataSource (trait) ──► FileScanConfig ──► FileSource
(trait) ──► ParquetSource / ...
```
Roughly speaking, the`DataSource` handles execution and partitioning and the
`FileSource` handles file-type specifics. But `FileSource` is currently coupled
to `FileScanConfig` (the concrete `DataSource`). Its open-time methods take a
`&FileScanConfig` and use it as a "bag of args" (e.g. `ParquetSource` pulls
`expr_adapter_factory`, `limit`, `preserve_order`, ...).
```rust
fn create_file_opener(&self, object_store: Arc<dyn ObjectStore>,
base_config: &FileScanConfig, partition: usize) -> ...;
fn create_morselizer(&self, object_store: Arc<dyn ObjectStore>, base_config:
&FileScanConfig, partition: usize) -> ...;
```
Decoupling the two would let a custom `DataSource` reuse any `FileSource`
directly.
### Describe the solution you'd like
Pass those inputs explicitly via a small, freely-constructible args struct,
replacing retrieval direct from the `&FileScanConfig`:
```rust
#[derive(Debug, Clone)]
#[non_exhaustive]
pub struct FileSourceArgs {
pub object_store: Arc<dyn ObjectStore>,
pub batch_size: usize,
pub limit: Option<usize>,
pub preserve_order: bool,
pub file_compression_type: FileCompressionType,
pub expr_adapter_factory: Option<Arc<dyn PhysicalExprAdapterFactory>>,
}
fn create_file_opener(&self, args: &FileSourceArgs, partition: usize) -> ...;
fn create_morselizer(&self, args: &FileSourceArgs, partition: usize) -> ...;
// with_batch_size removed — batch size now flows through FileSourceArgs
```
`FileScanConfig` builds `FileSourceArgs` inline (behavior unchanged).
However, now a custom `DataSource` can build the same struct and reuse any
`FileSource` too.
This is a **breaking change** to the public `FileSource` trait, but the
migration path is simple.
I have a draft PR ready for this issue.
### Describe alternatives you've considered
1. Do nothing. I can construct a dummy `FileScanConfig` within my custom
`DataSource` - simply to satisfy the interface - but the construction is quite
messy.
2. Extend the `DataSource` trait with all getter fns that a `FileSource` may
- might(?) avoid a breaking change, but adds coupling that may not make sense
for other custom DataSources.
3. Fork ParquetSource, add a custom `fn create_morselizer()` that doesn't
depend on `FileSourceArgs` - I'd prefer not to diverge from upstream
### Additional context
The goal of my custom DataSource is to increase I/O parallelism on an I/O
bound workload with plenty of CPU to spare. To achieve this, I am diverging
from the `FileStream` currently tightly coupled in a `FileScanConfig`.
P.S. this is my first datafusion issue. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
