Hi devs,
I'm working on getting Apache Sedona upgraded to work with Spark 3.2, and
ran into a weird issue I wanted to get some feedback on. The PR and current
discussion can be found here:
https://github.com/apache/incubator-sedona/pull/557
To try to sum up in a quick way, this library defines custom expressions
and registers the expressions using
sparkSession.sessionState.functionRegistry.registerFunction. One of the
unit tests is now failing because the function can't be found when a
temporary view using that function is created in pure SQL.
Examples:
This fails with Undefined function: 'ST_PolygonFromEnvelope'. This function
is neither a registered temporary function nor a permanent function
registered in the database 'default'.:
spark.sql(
"""
|CREATE OR REPLACE TEMP VIEW pixels AS
|SELECT pixel, shape FROM pointtable
|LATERAL VIEW EXPLODE(ST_Pixelize(shape, 1000, 1000,
ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS
pixel
""".stripMargin)
// Test visualization partitioner
val zoomLevel = 2
val newDf = VizPartitioner(spark.table("pixels"), zoomLevel,
"pixel", new Envelope(0, 1000, 0, 1000))
But both of these work fine:
val table = spark.sql(
"""
|SELECT pixel, shape FROM pointtable
|LATERAL VIEW EXPLODE(ST_Pixelize(shape, 1000, 1000,
ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS
pixel
""".stripMargin)
// Test visualization partitioner
val zoomLevel = 2
val newDf = VizPartitioner(table, zoomLevel, "pixel", new
Envelope(0, 1000, 0, 1000))
val table = spark.sql(
"""
|SELECT pixel, shape FROM pointtable
|LATERAL VIEW EXPLODE(ST_Pixelize(shape, 1000, 1000,
ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS
pixel
""".stripMargin)
table.createOrReplaceTempView("pixels")
// Test visualization partitioner
val zoomLevel = 2
val newDf = VizPartitioner(spark.table("pixels"), zoomLevel,
"pixel", new Envelope(0, 1000, 0, 1000))
So the main question is, is this a feature or a bug?
--
Adam Binford
