mjohns-databricks commented on issue #2472: URL: https://github.com/apache/sedona/issues/2472#issuecomment-3528467401
Due to the issue @Kontinuation is addressing with #2499 ... - I was unable to run Sedona 1.8.0 on DBR 17.3 (Spark 4) - However, I was able to run Sedona 1.8.0 on DBR 16.4 (Spark 3.5) using https://mvnrepository.com/artifact/org.apache.sedona/sedona-spark-shaded-3.5_2.13/1.8.0 - Once 1.8.1 has the fix for DBR17.3, look forward to running there. - There is still the very real issue (especially for DBR17.1+) of Sedona function registrations conflicting / taking over for Databricks built-in functions. While that is being sorted out further, see #2397, recommend you verify the provided of a given function, e.g. see screenshot for testing which provider is offering ST_Area (as both Sedona and Databricks provide, but if Sedona is registered on a classic compute, it wins so to speak) <img width="624" height="421" alt="Image" src="https://github.com/user-attachments/assets/f9b5bef8-7578-48ce-984e-3a7e0714cfcd"; /> However, on the same cluster, a given user may not know which functions are provided by Sedona and which are provided by Databricks, so they might end up interrogating quite a few functions to very which is providing which. For cluster owners who know exactly what they are doing this is manageable, but often we have cluster users (e.g. on a shared cluster) who do not know what is what and so upon hitting any error executing may check Databricks docs, e.g. for [ST_Area](https://docs.databricks.com/aws/en/sql/language-manual/functions/st_area), and then be very confused when the documented params are not supported. Regardless ... Here is a quick screen shot of adding the JARs to the cluster (no init script in this case), just in case it helps. <img width="478" height="291" alt="Image" src="https://github.com/user-attachments/assets/2a3a773b-26f8-4104-9bad-37d09b16cb43"; /> And here is the verification <img width="624" height="656" alt="Image" src="https://github.com/user-attachments/assets/7e211edd-bcde-414e-ae6c-a9908e354636"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
