Hemanth Boyina created SPARK-59377:
--------------------------------------
Summary: DSv2 column statistics are silently dropped when the
column name needs quoting or differs only in case
Key: SPARK-59377
URL: https://issues.apache.org/jira/browse/SPARK-59377
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Hemanth Boyina
DataSourceV2Relation.transformV2Stats matches connector-reported column
statistics to
output attributes with:
if (attribute.name.equals(key.describe())) \{ ... }
`key` is a NamedReference and `describe()` defaults to toString(), which
back-tick-quotes
any name part that is not a plain identifier (via
QuotingUtils.quoteIfNeeded). Two problems
follow:
1. Any top-level column whose name needs quoting -- e.g. "col-1", "1col", or
names with
dots/spaces -- produces key.describe() == "`col-1`", which never equals
attribute.name
("col-1"), so its ColumnStat is silently discarded.
2. The comparison uses String.equals, which is case-sensitive regardless of
spark.sql.caseSensitive. A stat reported as "ID" is dropped for an output
attribute
"id" even under Spark's default case-insensitive resolution.
There is no error or warning. The lost column statistics (distinctCount,
min/max, nullCount,
histogram) mean the cost-based optimizer produces worse
cardinality/selectivity estimates,
and therefore worse join-order, broadcast, and aggregate decisions, whenever
CBO is enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
