gabotechs commented on code in PR #127:
URL:
https://github.com/apache/datafusion-iceberg/pull/127#discussion_r4640538855
##########
crates/datafusion/src/read/file_scan_task_provider/lazy.rs:
##########
@@ -36,19 +38,26 @@ use crate::to_datafusion_error;
/// Snapshot summary field from the spec's "Optional Snapshot Summary Fields".
const TOTAL_RECORDS: &str = "total-records";
-/// Discovers files while executing, in a single partition: the files of a
-/// snapshot that may match a predicate.
+/// Plans file tasks during execution. Each output partition reads from the
+/// same task stream; unopened partitions do not reserve files. Dropping the
+/// last reader drops the stream, and late readers see EOF.
+///
+/// Opening another reader does not restart the scan. Use DataFusion's
+/// `reset_plan_states` before executing the same parallel plan again.
#[derive(Debug, Clone)]
pub struct LazyFileScanTaskProvider {
table: Table,
schema: SchemaRef,
snapshot_id: Option<i64>,
projection: Vec<String>,
predicate: Option<Predicate>,
+ target_partitions: usize,
+ // Only used when callers open the provider directly, without
DataSourceExec.
+ direct_tasks: Arc<OnceLock<SharedFileTasks>>,
Review Comment:
Need to read a bit more, but this does not look right. Having fields that
are only used when this is used in a specific way... it seems to be leaking
details about how it's used, but this struct should not matter.
##########
crates/datafusion/src/read/file_scan_task_provider/provider.rs:
##########
@@ -53,6 +53,14 @@ pub trait FileScanTaskProvider: Any + Debug + Send + Sync {
context: FileScanTaskProviderContext,
) -> Result<iceberg::scan::FileScanTaskStream>;
+ /// Optionally supplies a single task stream for all partitions of one
execution.
+ /// The source lets each partition pull the next unclaimed task from that
stream.
+ /// Return a fresh stream on each call; defer I/O until it is polled.
+ /// The default uses [`Self::open`] for each partition instead.
+ fn open_shared(&self) -> Option<iceberg::scan::FileScanTaskStream> {
+ None
+ }
Review Comment:
This does not look right. We should avoid having "optional" methods that are
only relevant for 1 specific case in a trait that claims to be generic.
Ideally, we can keep this trait as "proxy" as possible to the DataSource,
without inventing new methods that do not exist there.
##########
crates/datafusion/src/read/scan.rs:
##########
@@ -327,6 +348,14 @@ impl DataSource for IcebergDataSource {
self.fetch
}
+ fn create_sibling_state(
+ &self,
+ _config: &ConfigOptions,
+ ) -> Option<Arc<dyn Any + Send + Sync>> {
+ let tasks = self.file_scan_task_provider.open_shared()?;
+ Some(Arc::new(SharedFileTasks::new(tasks)))
+ }
Review Comment:
This is weirdly threaded. The idea behind the sibling state introduced in
https://github.com/apache/datafusion/pull/21351 is that the returned `Any` type
is what owns the shared state, however, even if we do are consuming the sibling
state API, we are anyways storing state in the structs owned by the plan:
```rust
direct_tasks: Arc<OnceLock<SharedFileTasks>>,
```
I'd either:
- Not use the sibling state API at all
- Fully delegate statefulness to the sibling state
But not something in between.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]