This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch branch-1.7.1
in repository https://gitbox.apache.org/repos/asf/sedona.git
The following commit(s) were added to refs/heads/branch-1.7.1 by this push:
new 8a111df33e [DOCS] Update the Databricks doc to reflect issues on DBR
16 (#1877)
8a111df33e is described below
commit 8a111df33e58be305e8a92673e59fea5894bd9b0
Author: Jia Yu <[email protected]>
AuthorDate: Tue Mar 25 19:06:57 2025 -0700
[DOCS] Update the Databricks doc to reflect issues on DBR 16 (#1877)
---
docs/setup/databricks.md | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/docs/setup/databricks.md b/docs/setup/databricks.md
index d0a8d332d9..ff51a9eb70 100644
--- a/docs/setup/databricks.md
+++ b/docs/setup/databricks.md
@@ -19,8 +19,10 @@
In Databricks advanced editions, you need to install Sedona via [cluster
init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as
described below. Sedona is not guaranteed to be 100% compatible with
`Databricks photon acceleration`. Sedona requires Spark internal APIs to inject
many optimization strategies, which sometimes is not accessible in `Photon`.
-!!!note
- The following steps use DBR including Apache Spark 3.4.x as an example.
Please change the Spark version according to your DBR version. Please pay
attention to the Spark version postfix and Scala version postfix on our [Maven
Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's
compatibility can be found
[here](https://docs.databricks.com/en/release-notes/runtime/index.html).
+The following steps use DBR including Apache Spark 3.4.x as an example. Please
change the Spark version according to your DBR version. Please pay attention to
the Spark version postfix and Scala version postfix on our [Maven Coordinate
page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility
can be found
[here](https://docs.databricks.com/en/release-notes/runtime/index.html).
+
+!!! bug
+ Databricks Runtime 16.2 (non-LTS) introduces a change in the json4s
dependency, which may lead to compatibility issues with Apache Sedona. We
recommend using a currently supported LTS version, such as Databricks Runtime
15.4 LTS or 14.3 LTS, to ensure stability. A patch will be provided once an
official Databricks Runtime 16 LTS version is released.
### Download Sedona jars
@@ -75,6 +77,7 @@ From your cluster configuration (`Cluster` -> `Edit` ->
`Configuration` -> `Adva
spark.sql.extensions
org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator org.apache.sedona.core.serde.SedonaKryoRegistrator
+spark.sedona.enableParserExtension false
```
From your cluster configuration (`Cluster` -> `Edit` -> `Configuration` ->
`Advanced options` -> `Init Scripts`) add the newly created `Workspace` init
script
