[
https://issues.apache.org/jira/browse/SPARK-61081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61081:
-----------------------------------
Labels: pull-request-available (was: )
> GROUP BY, DISTINCT and hash functions on GEOMETRY or GEOGRAPHY fail with
> scala.MatchError
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-61081
> URL: https://issues.apache.org/jira/browse/SPARK-61081
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Joel Robin
> Priority: Major
> Labels: pull-request-available
>
> h3. Summary
> A query fails with scala.MatchError or INTERNAL_ERROR when it groups,
> deduplicates or hashes a GEOMETRY or GEOGRAPHY value. The analyzer accepts
> these queries. The failure occurs at execution or during optimization.
> h3. Environment
> * Repository: https://github.com/apache/spark
> * Branch: master
> * Commit: 352ac9bf228 (2026-11-06), version 5.0.0-SNAPSHOT
> * Configuration: default. spark.sql.geospatial.enabled is true by default.
> * Run mode: local[4], Java 21
> h3. Steps to reproduce
> Run these statements in a Spark SQL session:
> {code:sql}
> -- 1. GROUP BY
> SELECT g, COUNT(*) AS n
> FROM (SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040',
> 4326) AS g FROM range(10))
> GROUP BY g;
> -- 2. COUNT(DISTINCT)
> SELECT COUNT(DISTINCT g) AS cd
> FROM (SELECT ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040',
> 4326) AS g FROM range(10));
> -- 3. hash()
> SELECT hash(ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040',
> 4326)) AS h;
> {code}
> h3. Expected result
> Each statement returns a correct result, or the analyzer stops it with a
> clear error.
> For comparison, ORDER BY g and g = g fail with a clear error:
> DATATYPE_MISMATCH.INVALID_ORDERING_TYPE.
> h3. Actual result
> Statements 1 and 2 abort the job:
> {noformat}
> org.apache.spark.SparkException: Job aborted due to stage failure: ...
> scala.MatchError: org.apache.spark.unsafe.types.BinaryView@15dc1861 (of class
> org.apache.spark.unsafe.types.BinaryView)
> at
> org.apache.spark.sql.catalyst.expressions.InterpretedHashFunction.hash(hash.scala:713)
> at
> org.apache.spark.sql.catalyst.expressions.CollationAwareMurmur3Hash.computeHash(hash.scala:899)
> at
> org.apache.spark.sql.catalyst.expressions.HashExpression.eval(hash.scala:405)
> at
> org.apache.spark.sql.catalyst.expressions.Pmod.eval(arithmetic.scala:1200)
> at
> org.apache.spark.sql.catalyst.expressions.InterpretedUnsafeProjection.apply(InterpretedUnsafeProjection.scala:84)
> at
> org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$8(ShuffleExchangeExec.scala:445)
> {noformat}
> Statement 3 fails during optimization:
> {noformat}
> [INTERNAL_ERROR] The Spark SQL phase optimization failed with an internal
> error. ...
> Caused by: scala.MatchError:
> org.apache.spark.unsafe.types.BinaryView@15dc1861 (of class
> org.apache.spark.unsafe.types.BinaryView)
> at
> org.apache.spark.sql.catalyst.expressions.InterpretedHashFunction.hash(hash.scala:713)
> at
> org.apache.spark.sql.catalyst.expressions.Murmur3Hash.computeHash(hash.scala:862)
> at
> org.apache.spark.sql.catalyst.expressions.HashExpression.eval(hash.scala:405)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.tryFold(expressions.scala:73)
> {noformat}
> These queries fail in the same way:
> * approx_count_distinct(g)
> * SELECT DISTINCT g
> * DISTRIBUTE BY g, when g is not a constant. (With a constant,
> OptimizeRepartition changes the query to one partition, and the query does
> not fail.)
> * xxhash64(g)
> * GROUP BY and COUNT(DISTINCT) on GEOGRAPHY, for example ST_GeogFromWKB(...)
> h3. Cause
> # InterpretedHashFunction.hash (hash.scala:713) has no case for BinaryView.
> BinaryView is the physical value of GEOMETRY and GEOGRAPHY.
> # HashExpression.computeHashWithTailRec (hash.scala:638) has no case for
> GeometryType or GeographyType. Code generation fails first, then Spark falls
> back to the interpreted path, which also fails. With
> spark.sql.codegen.factoryMode=CODEGEN_ONLY, the error is: scala.MatchError:
> GeometryType(4326).
> # The analyzer does not stop these queries. ExprUtils.checkValidGroupingExprs
> (ExprUtils.scala:228-235) rejects only VariantType.
> HashExpression.checkInputDataTypes rejects only MapType and VariantType.
> h3. Related problem: spill to sort-based aggregation
> A GROUP BY on a geometry key also fails when the hash aggregate spills, also
> with one partition. GenerateOrdering generates c1.compare(c2) for the
> geometry key, but BinaryView has only compareTo. To force the spill, set
> spark.sql.TungstenAggregate.testFallbackStartsAt=1,2 and run:
> {code:sql}
> SELECT g, COUNT(*) AS n
> FROM (SELECT /*+ REPARTITION(1) */
> ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326) AS g FROM
> range(10))
> GROUP BY g;
> {code}
> Result:
> {noformat}
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 30, Column 20: A method named "compare" is not declared in any enclosing
> class nor any supertype, nor through a static import
> at
> org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:192)
> at
> org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:84)
> at
> org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:277)
> at
> org.apache.spark.sql.execution.aggregate.HashAggregateExec$.spillHashMapToSorter(HashAggregateExec.scala:1452)
> {noformat}
> h3. Note on existing tests
> CollationAggregationSuite (SPARK-58847) expects GROUP BY key, geometry to
> return a correct result. The test calls repartition(1) first. With one
> partition, the plan has no hash shuffle, so the test does not find this
> problem. The same query without repartition(1) fails with the MatchError
> above.
> h3. Workaround
> Group by the binary value: GROUP BY ST_AsBinary(g).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
