This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sedona.git
The following commit(s) were added to refs/heads/master by this push:
new 84146249db [DOCS] Update the Databricks doc to reflect issues on DBR
16 (#1877)
84146249db is described below
commit 84146249dbb1927b79529bc8138dca3c1ef3a5ac
Author: Jia Yu <[email protected]>
AuthorDate: Tue Mar 25 19:06:57 2025 -0700
[DOCS] Update the Databricks doc to reflect issues on DBR 16 (#1877)
---
docs/setup/databricks.md | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/docs/setup/databricks.md b/docs/setup/databricks.md
index d0a8d332d9..ff51a9eb70 100644
--- a/docs/setup/databricks.md
+++ b/docs/setup/databricks.md
@@ -19,8 +19,10 @@
In Databricks advanced editions, you need to install Sedona via [cluster
init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as
described below. Sedona is not guaranteed to be 100% compatible with
`Databricks photon acceleration`. Sedona requires Spark internal APIs to inject
many optimization strategies, which sometimes is not accessible in `Photon`.
-!!!note
- The following steps use DBR including Apache Spark 3.4.x as an example.
Please change the Spark version according to your DBR version. Please pay
attention to the Spark version postfix and Scala version postfix on our [Maven
Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's
compatibility can be found
[here](https://docs.databricks.com/en/release-notes/runtime/index.html).
+The following steps use DBR including Apache Spark 3.4.x as an example. Please
change the Spark version according to your DBR version. Please pay attention to
the Spark version postfix and Scala version postfix on our [Maven Coordinate
page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility
can be found
[here](https://docs.databricks.com/en/release-notes/runtime/index.html).
+
+!!! bug
+ Databricks Runtime 16.2 (non-LTS) introduces a change in the json4s
dependency, which may lead to compatibility issues with Apache Sedona. We
recommend using a currently supported LTS version, such as Databricks Runtime
15.4 LTS or 14.3 LTS, to ensure stability. A patch will be provided once an
official Databricks Runtime 16 LTS version is released.
### Download Sedona jars
@@ -75,6 +77,7 @@ From your cluster configuration (`Cluster` -> `Edit` ->
`Configuration` -> `Adva
spark.sql.extensions
org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator org.apache.sedona.core.serde.SedonaKryoRegistrator
+spark.sedona.enableParserExtension false
```
From your cluster configuration (`Cluster` -> `Edit` -> `Configuration` ->
`Advanced options` -> `Init Scripts`) add the newly created `Workspace` init
script
