Hey, It seems like the GeoSpark repo is not publicly accessible?
But from the filepath it seems like the Spark codebase itself was forked or modified. The examples that I've seen seem to suggest that you need to register custom Spark-Native functions inside Spark's private namespace like you said (FunctionRegistry.scala I believe). I was wondering if it was possible to add the more efficient Spark-Native functions in my user application without having to fork or modify Spark itself. Thanks, Arwin From: Georg Heiler Sent: Wednesday, August 21, 11:18 PM Subject: Re: Creating custom Spark-Native catalyst/codegen functions To: Arwin Tio Cc: user@spark.apache.org Look at https://github.com/DataSystemsLab/GeoSpark/tree/master/sql/src/main/scala/org/apache/spark/sql/geospark sql for an example. Using custom function registration and functions residing inside sparks private namespace should work. But I am not aware of a public user facing API. Is there any I am missing? Arwin Tio < arwin....@hotmail.com<mailto:arwin....@hotmail.com>> schrieb am Do. 22. Aug. 2019 04:28: Hi friends, I am looking into converting some UDFs/UDAFs to Spark-Native functions to leverage Catalyst and codegen. Looking through some examples (for example: https://github.com/apache/spark/pull/7214/files for Levenshtein) it seems like we need to add these functions to the Spark framework itself. Is there a way to add custom Spark-Native functions in "userspace"? Thank you! Arwin
