Kontinuation commented on code in PR #1706:
URL: https://github.com/apache/sedona/pull/1706#discussion_r1864059919
##########
spark/common/src/main/scala/org/apache/sedona/spark/SedonaContext.scala:
##########
@@ -65,9 +65,6 @@ object SedonaContext {
RasterRegistrator.registerAll(sparkSession)
UdtRegistrator.registerAll()
UdfRegistrator.registerAll(sparkSession)
- if (sparkSession.conf.get("spark.sedona.enableParserExtension",
"true").toBoolean) {
- ParserRegistrator.register(sparkSession)
- }
Review Comment:
NOTE: This introduces a behavior change. Now setting up
`spark.sql.extensions` is required for enabling the parser extension. I think
this is a reasonable change and I'm OK with it.
##########
spark/common/src/main/scala/org/apache/sedona/sql/ParserRegistrator.scala:
##########
@@ -38,21 +38,6 @@ object ParserRegistrator {
val field =
sparkSession.sessionState.getClass.getDeclaredField("sqlParser")
field.setAccessible(true)
field.set(sparkSession.sessionState, parser)
- return // return if the new constructor is available
- } catch {
- case _: Exception =>
- }
-
- // try to register the parser with the legacy constructor for spark 3.0
- try {
- val parserClassName = "org.apache.sedona.sql.parser.SedonaSqlParser"
- val delegate: ParserInterface = sparkSession.sessionState.sqlParser
-
- val parser =
- ParserFactory.getParser(parserClassName,
sparkSession.sessionState.conf, delegate)
- val field =
sparkSession.sessionState.getClass.getDeclaredField("sqlParser")
- field.setAccessible(true)
- field.set(sparkSession.sessionState, parser)
} catch {
case _: Exception =>
}
Review Comment:
ParserRegistrator.scala is not needed anymore, as for now we register
parsers using the extension interface instead of using the reflection trick. We
can delete this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
