[
https://issues.apache.org/jira/browse/SPARK-59765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59765:
-----------------------------------
Labels: pull-request-available (was: )
> Codegen size checks over empty TPC-DS tables measure stubs for broadcast joins
> ------------------------------------------------------------------------------
>
> Key: SPARK-59765
> URL: https://issues.apache.org/jira/browse/SPARK-59765
> Project: Spark
> Issue Type: Test
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Max Gekk
> Priority: Major
> Labels: pull-request-available
>
> Two instruments measure the code whole-stage codegen generates for the TPC-DS
> queries without running them: {{BenchmarkQueryTest.checkGeneratedCode}}, used
> by {{TPCDSQuerySuite}} and its variants (see SPARK-59764), and
> {{WholeStageCodegenSizeBenchmark}} (SPARK-57915), the instrument for the
> SPARK-56908 umbrella. Both plan each query over empty TPC-DS tables with AQE
> disabled, and call {{doCodeGen()}} on every {{WholeStageCodegenExec}}.
> With AQE disabled, {{BroadcastHashJoinExec.prepareBroadcast}} executes the
> broadcast while generating code. Over an empty table the relation is
> {{EmptyHashedRelation}}, and {{HashJoin}} then generates a one-line comment
> in place of the join: "If HashedRelation is empty, hash inner join simply
> returns nothing" (and the semi join equivalent), in which case the code of
> every operator after the join in the stage is not generated either. For an
> anti join it generates only the pass-through. {{BroadcastNestedLoopJoinExec}}
> has a similar shortcut for an empty build side. So a stage with a broadcast
> join is measured as a stub, and its method sizes and constant pools are
> understated.
> How much, on master (4cfbd411966): with {{WholeStageCodegenSizeBenchmark}}'s
> setup (injected SF100 statistics with CBO, AQE off, empty tables), the
> generated code of 324 of the 2201 whole-stage stages of the 129 TPC-DS v1.4
> and v2.7 queries contains the empty-relation stub, in 127 of the 129 queries.
> Measured by planning each query as {{TPCDSQueryWithStatsSuite}} does, taking
> {{codegenStringSeq(executedPlan)}}, and counting the stages whose source
> contains "If HashedRelation is empty".
> In a fork close to master, generating those stages in full made 374 stages
> larger, the largest by about 3.5KB of bytecode for one method. So the
> benchmark's totals and the 8000-byte check both see less code than the
> queries compile over real data.
> Proposed fix, for both: generate each stage's code from a copy of the stage
> in which every {{BroadcastExchangeExec}} (including one under a
> {{ReusedExchangeExec}}) reads a one-row {{LocalTableScanExec}} of non-null
> default values. The joins then generate the code they generate for data, and
> nothing else about the plan changes. Assert that no generated code still
> contains the empty-relation comment, so the stub cannot come back unnoticed.
> One approximation remains: a one-row build side makes every key unique, so a
> join generates its unique-key form; a join whose real build side repeats keys
> also has the loop over matches.
> Populating the tables with data would also work, but a fixed row is filtered
> out by any literal predicate on the build side (for example {{d_year =
> 2000}}), which leaves the relation empty again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
