[
https://issues.apache.org/jira/browse/SPARK-59539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59539:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid per-row autoboxing in InSet code generation for integral and date/time
> types
> ----------------------------------------------------------------------------------
>
> Key: SPARK-59539
> URL: https://issues.apache.org/jira/browse/SPARK-59539
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> h2. Background
> InSet backs IN predicates with many elements (and x = a OR x = b OR ...
> chains after OptimizeIn folds them). In whole-stage-generated code,
> InSet.genCodeWithSet evaluates membership as set.contains(value) over a Scala
> Set[Any]. Because contains takes Object, a primitive subject is autoboxed on
> every row (a Long.valueOf/Integer.valueOf allocation per row), and the set
> stores boxed elements.
> Only Byte/Short/Int/Date sets up to spark.sql.optimizer.inSetSwitchThreshold
> (default 400) avoid this, via a JVM switch. The switch bytecode requires int
> labels, so Long/Timestamp/TimestampNTZ can never use it and always take the
> boxed path at every size; large Int/Date sets (> 400) also fall back to it.
> h2. Proposal
> For integral and date/time types whose Catalyst internal representation is a
> primitive int or long (Long, Timestamp, TimestampNTZ, Int, Date, Short,
> Byte), build a sorted primitive long[]/int[] on the driver and probe it in
> generated code with java.util.Arrays.binarySearch — a primitive overload with
> no autoboxing and no allocation. Byte/Short are widened to int.
> The change is gated by a new internal flag
> spark.sql.optimizer.useSpecializedSetForInSet.enabled (default true) as an
> operational fallback. It affects only whole-stage code generation: the
> interpreted eval path (which receives an already-boxed value, so there is no
> boxing to remove) and the switch path are unchanged, and the InSet node
> itself is unchanged, so there is no plan-shape or golden-file change.
> Out of scope, kept on the existing boxed path:
> - Float/Double — their SQL equality (NaN = NaN, -0.0 = +0.0) requires the
> special handling the current path provides.
> - String/Decimal/binary/complex — not primitive; UTF8String is already an
> object, so there is no boxing to remove.
> h2. Benefit
> Membership over a Long column, 10M rows, non-matching values (worst case —
> every row evaluates the full predicate). Current boxed InSet vs the
> sorted-array binary search (local microbenchmark, indicative):
> ||set size||boxed InSet (ms)||sorted-array binarySearch (ms)||
> |20|107|40|
> |100|95|54|
> |500|151|82|
> ~1.7–2.7x faster with zero per-row allocation.
> h2. Correctness
> Semantics are unchanged. The generated three-valued-logic structure is
> identical (match → true; non-match with a null in the list → null via the
> existing hasNull handling; null subject handled by nullSafeCodeGen); only the
> membership test changes from a boxed Set.contains to a primitive
> Arrays.binarySearch. For the in-scope types, equality is exact primitive
> comparison, so no special-value handling is required. Covered by existing and
> new PredicateSuite tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
