[
https://issues.apache.org/jira/browse/SPARK-60072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60072:
-----------------------------------
Labels: pull-request-available (was: )
> DataFrame pivot rejects count('*') with INVALID_USAGE_OF_STAR_OR_REGEX
> ----------------------------------------------------------------------
>
> Key: SPARK-60072
> URL: https://issues.apache.org/jira/browse/SPARK-60072
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 4.2.0, 4.1.1
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: pull-request-available
>
> Since Spark 4.0, a DataFrame pivot whose aggregate is {{count("*")}} fails
> analysis. The same query works in 3.5.
> {code:java}
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.getOrCreate()
> from pyspark.sql import functions as F
> df = spark.createDataFrame([(1, 'a'), (2, None)], "x int, s string")
> df.groupBy('s').pivot('x', [1, 2]).agg(F.count('*')).show() {code}
> Expected (3.5.8): {{[Row(s=None, 1=None, 2=1), Row(s='a', 1=1, 2=None)]}}
> Actual (4.0.0, 4.1.1, 4.2.0):
> {code:java}
> AnalysisException: [INVALID_USAGE_OF_STAR_OR_REGEX] Invalid usage of '*' in
> Pivot. SQLSTATE: 42000;
> 'Pivot List(s#1), x#0: int, [1, 2], ['count(*)]
> +- LogicalRDD [x#0, s#1], false {code}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
