Eric Yang created SPARK-57313:
---------------------------------
Summary: SampleExec does not update numOutputRows metric when
whole-stage codegen is disabled
Key: SPARK-57313
URL: https://issues.apache.org/jira/browse/SPARK-57313
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Eric Yang
1SampleExec1 declares a 1numOutputRows1 SQL metric. Its whole-stage-codegen
path (`doConsume`) increments it per emitted row, but the interpreted doExecute
returns the sampled RDD without ever updating the metric.
As a result, when whole-stage codegen is off
(spark.sql.codegen.wholeStage=false), the Sample / TABLESAMPLE operator reports
number of output rows: 0 in the SQL UI, while the codegen path reports the
correct count.
Repro:
{code:java}
spark.conf.set("spark.sql.codegen.wholeStage", "false")
val df = spark.range(0, 1000, 1, 1).sample(withReplacement = false, fraction
= 0.5, seed = 1)
df.collect()
// Sample node in the SQL tab shows "number of output rows: 0" (expected
~518) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
