David Mollitor created SPARK-59539:
--------------------------------------
Summary: Avoid per-row autoboxing in InSet code generation for
integral and date/time types
Key: SPARK-59539
URL: https://issues.apache.org/jira/browse/SPARK-59539
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Background
InSet backs IN predicates with many elements (and x = a OR x = b OR ... chains
after OptimizeIn folds them). In whole-stage-generated code,
InSet.genCodeWithSet evaluates membership as set.contains(value) over a Scala
Set[Any]. Because contains takes Object, a primitive subject is autoboxed on
every row (a Long.valueOf/Integer.valueOf allocation per row), and the set
stores boxed elements.
Only Byte/Short/Int/Date sets up to spark.sql.optimizer.inSetSwitchThreshold
(default 400) avoid this, via a JVM switch. The switch bytecode requires int
labels, so Long/Timestamp/TimestampNTZ can never use it and always take the
boxed path at every size; large Int/Date sets (> 400) also fall back to it.
h2. Proposal
For integral and date/time types whose Catalyst internal representation is a
primitive int or long (Long, Timestamp, TimestampNTZ, Int, Date, Short, Byte),
build a sorted primitive long[]/int[] on the driver and probe it in generated
code with java.util.Arrays.binarySearch — a primitive overload with no
autoboxing and no allocation. Byte/Short are widened to int.
The change is gated by a new internal flag
spark.sql.optimizer.useSpecializedSetForInSet.enabled (default true) as an
operational fallback. It affects only whole-stage code generation: the
interpreted eval path (which receives an already-boxed value, so there is no
boxing to remove) and the switch path are unchanged, and the InSet node itself
is unchanged, so there is no plan-shape or golden-file change.
Out of scope, kept on the existing boxed path:
- Float/Double — their SQL equality (NaN = NaN, -0.0 = +0.0) requires the
special handling the current path provides.
- String/Decimal/binary/complex — not primitive; UTF8String is already an
object, so there is no boxing to remove.
h2. Benefit
Membership over a Long column, 10M rows, non-matching values (worst case —
every row evaluates the full predicate). Current boxed InSet vs the
sorted-array binary search (local microbenchmark, indicative):
||set size||boxed InSet (ms)||sorted-array binarySearch (ms)||
|20|107|40|
|100|95|54|
|500|151|82|
~1.7–2.7x faster with zero per-row allocation.
h2. Correctness
Semantics are unchanged. The generated three-valued-logic structure is
identical (match → true; non-match with a null in the list → null via the
existing hasNull handling; null subject handled by nullSafeCodeGen); only the
membership test changes from a boxed Set.contains to a primitive
Arrays.binarySearch. For the in-scope types, equality is exact primitive
comparison, so no special-value handling is required. Covered by existing and
new PredicateSuite tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
