cetra3 commented on code in PR #26630:
URL: https://github.com/apache/datafusion/pull/26630#discussion_r4384287330
##########
datafusion/common/src/dfschema.rs:
##########
@@ -1159,6 +1210,116 @@ impl Hash for DFSchema {
}
}
+// `name_index` is derived state, so it is left out of equality and `Debug`.
+impl PartialEq for DFSchema {
+ fn eq(&self, other: &Self) -> bool {
+ self.inner == other.inner
+ && self.field_qualifiers == other.field_qualifiers
+ && self.functional_dependencies == other.functional_dependencies
+ }
+}
+
+impl Eq for DFSchema {}
+
+impl std::fmt::Debug for DFSchema {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("DFSchema")
+ .field("inner", &self.inner)
+ .field("field_qualifiers", &self.field_qualifiers)
+ .field("functional_dependencies", &self.functional_dependencies)
+ .finish()
+ }
+}
+
+/// Schemas with up to this many fields resolve names by comparing against
every
+/// field; wider ones use a lazily built [`NameIndex`]. Below it, the scan is
+/// cheaper than building the index (the same trade-off as
+/// `LINEAR_FIELD_SCAN_MAX` in `datafusion-datasource-parquet`).
+const LINEAR_NAME_SCAN_MAX: usize = 8;
Review Comment:
I've done a sweep, and there isn't much difference between them locally:
```
┌────────────────────────────────────────────┬─────────────┬──────────────────┬──────────────────┬──────────────────┐
│ Benchmark │ main │ 8 (the PR)
│ 64 │ 256 │
├────────────────────────────────────────────┼─────────────┼──────────────────┼──────────────────┼──────────────────┤
│ logical_union_8_branches │ 339.1 µs │ +2.1% / +1.8%
│ +2.2% / +2.1% │ −0.2% / −0.3% │
├────────────────────────────────────────────┼─────────────┼──────────────────┼──────────────────┼──────────────────┤
│ logical_correlated_subquery_in │ 109.9 µs │ +1.7% / +1.4%
│ +1.4% / +1.2% │ +0.2% / −0.3% │
├────────────────────────────────────────────┼─────────────┼──────────────────┼──────────────────┼──────────────────┤
│ with_param_values_many_columns │ 228.0 µs │ −36.8% / −36.8%
│ −35.4% / −35.7% │ −16.4% / −14.8% │
├────────────────────────────────────────────┼─────────────┼──────────────────┼──────────────────┼──────────────────┤
│ logical_trivial_join_high_numbered_columns │ 113.2 µs │ −6.7% / −6.3%
│ −6.9% / −6.5% │ −6.7% / −6.6% │
├────────────────────────────────────────────┼─────────────┼──────────────────┼──────────────────┼──────────────────┤
│ physical_many_self_joins │ 4.36 ms │ −5.6% / −5.9%
│ −6.1% / −5.8% │ −5.8% / −5.0% │
├────────────────────────────────────────────┼─────────────┼──────────────────┼──────────────────┼──────────────────┤
│ *_select_all_from_1000 (3 benchmarks) │ 4.2–10.6 ms │ −67%
│ −67% │ −67% │
├────────────────────────────────────────────┼─────────────┼──────────────────┼──────────────────┼──────────────────┤
│ optimizer_tpcds_all │ 155.5 ms │ −2.8% / −2.7%
│ −2.7% / −2.3% │ −1.8% / −2.4% │
├────────────────────────────────────────────┼─────────────┼──────────────────┼──────────────────┼──────────────────┤
│ Other 6 benchmarks │ │ within about
±2% │ within about ±2% │ within about ±2% │
└────────────────────────────────────────────┴─────────────┴──────────────────┴──────────────────┴──────────────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
