andygrove commented on code in PR #114:
URL: 
https://github.com/apache/datafusion-iceberg/pull/114#discussion_r4615836268


##########
crates/datafusion/src/read/scan.rs:
##########
@@ -93,6 +98,39 @@ impl IcebergDataSource {
     pub fn fetch(&self) -> Option<usize> {
         self.fetch
     }
+
+    /// Estimates the rows the scan returns before its predicate and fetch,
+    /// from the snapshot summary.
+    ///
+    /// `total-records` counts rows before deletes. A position delete removes
+    /// at most one row, so position deletes are subtracted. An equality delete
+    /// can remove any number of rows, so equality deletes are ignored. Summary
+    /// fields are optional. A missing or invalid `total-records` leaves the
+    /// estimate unknown, and a missing or invalid `total-position-deletes`
+    /// counts as zero, rather than failing the query.
+    fn estimated_num_rows(&self) -> Precision<usize> {
+        let metadata = self.table.metadata();
+        let snapshot = match self.snapshot_id {
+            Some(snapshot_id) => metadata.snapshot_by_id(snapshot_id),
+            None => match metadata.current_snapshot() {
+                Some(snapshot) => Some(snapshot),
+                // The scan of a table with no current snapshot reads nothing.
+                None => return Precision::Exact(0),
+            },
+        };
+        // The scan fails on a snapshot id that doesn't exist.
+        let Some(snapshot) = snapshot else {
+            return Precision::Absent;
+        };
+        let summary = &snapshot.summary().additional_properties;
+        let count = |key| summary.get(key)?.parse::<usize>().ok();
+        match count(TOTAL_RECORDS) {
+            Some(records) => Precision::Inexact(
+                
records.saturating_sub(count(TOTAL_POSITION_DELETES).unwrap_or(0)),

Review Comment:
   Thanks for this. It's a nice, contained way to give join selection something 
to work with, and the unit tests cover the summary edge cases well.
   
   I'm a bit worried about subtracting `total-position-deletes`. After 
`rewrite_data_files` without `remove-dangling-deletes` (the default in Spark), 
`total-records` reflects the compacted files while `total-position-deletes` can 
still count deletes against data files that are gone. The estimate can then 
land far below the real row count, down to `Inexact(0)` with the 
`saturating_sub`. Because `join_selection` uses `num_rows` against 
`hash_join_single_partition_threshold_rows`, a large table could get picked as 
the `CollectLeft` build side, which is worse than reporting unknown stats.
   
   Would it be safer to report `total-records` as-is, as @JanKaul's 
`datafusion_iceberg` does? Overestimating seems like the safer failure mode 
here. If you'd like to keep the subtraction, maybe fall back to `total-records` 
(or `Absent`) when the deletes make up a large share of the records? A unit 
test for the "deletes exceed records" case could then assert the fallback 
instead of `Inexact(0)`.



##########
crates/datafusion/src/read/scan.rs:
##########
@@ -93,6 +98,39 @@ impl IcebergDataSource {
     pub fn fetch(&self) -> Option<usize> {
         self.fetch
     }
+
+    /// Estimates the rows the scan returns before its predicate and fetch,
+    /// from the snapshot summary.
+    ///
+    /// `total-records` counts rows before deletes. A position delete removes
+    /// at most one row, so position deletes are subtracted. An equality delete
+    /// can remove any number of rows, so equality deletes are ignored. Summary
+    /// fields are optional. A missing or invalid `total-records` leaves the
+    /// estimate unknown, and a missing or invalid `total-position-deletes`
+    /// counts as zero, rather than failing the query.
+    fn estimated_num_rows(&self) -> Precision<usize> {
+        let metadata = self.table.metadata();
+        let snapshot = match self.snapshot_id {
+            Some(snapshot_id) => metadata.snapshot_by_id(snapshot_id),

Review Comment:
   +1 to resolving the snapshot once at construction. Could we file an issue 
for it so it doesn't get lost, and link it here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to