This is an automated email from the ASF dual-hosted git repository. jiayu pushed a commit to branch branch-1.7.2 in repository https://gitbox.apache.org/repos/asf/sedona.git
commit 51f17d256aeef54f424013006cd04d4b4fa2533b Author: Matthew Powers <[email protected]> AuthorDate: Wed Jun 11 05:52:40 2025 -0400 [DOCS] sedona distance (#1986) * [DOCS] add page on distance computations with spark * lint * make concepts pages specific to spark --- docs/image/tutorial/concepts/distance1.png | Bin 0 -> 67423 bytes docs/image/tutorial/concepts/distance2.png | Bin 0 -> 33315 bytes docs/image/tutorial/concepts/distance3.png | Bin 0 -> 36178 bytes docs/image/tutorial/concepts/distance4.png | Bin 0 -> 42298 bytes docs/image/tutorial/concepts/distance5.png | Bin 0 -> 31026 bytes docs/tutorial/concepts/clustering-algorithms.md | 4 +- docs/tutorial/concepts/distance-spark.md | 280 ++++++++++++++++++++++++ docs/tutorial/concepts/spatial-joins.md | 18 +- mkdocs.yml | 1 + 9 files changed, 292 insertions(+), 11 deletions(-) diff --git a/docs/image/tutorial/concepts/distance1.png b/docs/image/tutorial/concepts/distance1.png new file mode 100644 index 0000000000..4dd92363c0 Binary files /dev/null and b/docs/image/tutorial/concepts/distance1.png differ diff --git a/docs/image/tutorial/concepts/distance2.png b/docs/image/tutorial/concepts/distance2.png new file mode 100644 index 0000000000..e9760ed217 Binary files /dev/null and b/docs/image/tutorial/concepts/distance2.png differ diff --git a/docs/image/tutorial/concepts/distance3.png b/docs/image/tutorial/concepts/distance3.png new file mode 100644 index 0000000000..e3b292e06e Binary files /dev/null and b/docs/image/tutorial/concepts/distance3.png differ diff --git a/docs/image/tutorial/concepts/distance4.png b/docs/image/tutorial/concepts/distance4.png new file mode 100644 index 0000000000..660a6433db Binary files /dev/null and b/docs/image/tutorial/concepts/distance4.png differ diff --git a/docs/image/tutorial/concepts/distance5.png b/docs/image/tutorial/concepts/distance5.png new file mode 100644 index 0000000000..1499aa0f2d Binary files /dev/null and b/docs/image/tutorial/concepts/distance5.png differ diff --git a/docs/tutorial/concepts/clustering-algorithms.md b/docs/tutorial/concepts/clustering-algorithms.md index 79cb72cd8f..6d1a027d73 100644 --- a/docs/tutorial/concepts/clustering-algorithms.md +++ b/docs/tutorial/concepts/clustering-algorithms.md @@ -17,7 +17,7 @@ under the License. --> -# Apache Sedona Clustering Algorithms +# Apache Sedona Clustering Algorithms with Apache Spark Clustering algorithms group similar data points into “clusters.” Apache Sedona can run clustering algorithms on large geometric datasets. @@ -28,7 +28,7 @@ Note that the term cluster is overloaded here: This page uses “cluster” to refer to the output of a clustering algorithm. -## Clustering with DBSCAN +## Clustering with DBSCAN and Spark This page explains how to use Apache Sedona to perform density-based spatial clustering of applications with noise (“DBSCAN”). diff --git a/docs/tutorial/concepts/distance-spark.md b/docs/tutorial/concepts/distance-spark.md new file mode 100644 index 0000000000..52c52e144c --- /dev/null +++ b/docs/tutorial/concepts/distance-spark.md @@ -0,0 +1,280 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# Compute distance with Sedona and Apache Spark + +This post explains how to compute the distance between two points or geometric objects using Apache Sedona and Apache Spark. + +You will learn how to compute the distance on a two-dimensional Cartesian plane and how to calculate distance for geospatial data, taking into account the curvature of the Earth. + +Let’s start with an example on how to compute the distance between two points in a two-dimensional Cartesian plane. + +## Distance between two points with Spark and Sedona + +Suppose you have four points and would like to compute the distance between `point_a` and `point_b` and the distance between `point_c` and `point_d`. + + + +Let’s create a DataFrame with these points. + +```python +df = sedona.createDataFrame([ + (Point(2, 3), Point(6, 4)), + (Point(6, 2), Point(9, 2)), +], ["start", "end"]) +``` + +The `start` and `end` columns both have the `geometry` type. + +Now use the `ST_Distance` function to compute the distance between the points. + +```python +df.withColumn( + "distance", + ST_Distance(col("start"), col("end")) +).show() +``` + +Here are the results: + +``` ++-----------+-----------+-----------------+ +| start| end| distance| ++-----------+-----------+-----------------+ +|POINT (2 3)|POINT (6 4)|4.123105625617661| +|POINT (6 2)|POINT (9 2)| 3.0| ++-----------+-----------+-----------------+ +``` + +The `ST_Distance` function makes it relatively straightforward to compute the distance between points on a two-dimensional plane. + +## Distance between two longitude/latitude points with Spark and Sedona + +Let’s create two longitude/latitude points and compute the distance between them. Start by creating a DataFrame with the longitude and latitude values. + +```python +seattle = Point(-122.335167, 47.608013) +new_york = Point(-73.935242, 40.730610) +sydney = Point(151.2, -33.9) +df = sedona.createDataFrame( + [ + (seattle, new_york), + (seattle, sydney), + ], + ["place1", "place2"], +) +``` + +Let’s compute the distance between these points now: + +```python +df.withColumn( + "st_distance_sphere", + ST_DistanceSphere(col("place1"), col("place2")) +).show() +``` + +Here are the results: + +``` ++--------------------+--------------------+--------------------+ +| place1| place2| st_distance_sphere| ++--------------------+--------------------+--------------------+ +|POINT (-122.33516...|POINT (-73.935242...| 3870075.7867602874| +|POINT (-122.33516...| POINT (151.2 -33.9)|1.2473172370818963E7| ++--------------------+--------------------+--------------------+ +``` + +We use the `ST_DistanceSphere` function to calculate the distance, taking into account the Earth's curvature. The function returns the distance in meters. + +Let’s see how to compute the distance between two points with a spheroid model of the Earth. + +## Compute distance between points with a spheroid with Spark and Sedona + +Let’s use the same DataFrame from the previous section, but compute the distance using a spheroid model of the world. + +```python +res = df.withColumn( + "st_distance_spheroid", + ST_DistanceSpheroid(col("place1"), col("place2")) +) +res.select("place1_name", "place2_name", "st_distance_spheroid").show() +``` + +Here are the results: + +``` ++-----------+-----------+--------------------+ +|place1_name|place2_name|st_distance_spheroid| ++-----------+-----------+--------------------+ +| seattle| new_york| 3880173.4858397646| +| seattle| sydney|1.2456531875384018E7| ++-----------+-----------+--------------------+ +``` + +The `ST_DistanceSpheroid` function returns the meters between the two locations. The spheroid distance computation yields similar results to those obtained when you model the Earth as a sphere. Expect the spheroid function to return results that are slightly more accurate. + +## Distance between two geometric objects with Spark and Sedona + +Let’s take a look at how to compute the distance between a linestring and a polygon. Suppose you have the following objects: + + + +The distance between two polygons is the minimum Euclidean distance between any two points. + +Let’s compute the distance: + +```python +res = df.withColumn( + "distance", + ST_Distance(col("geom1"), col("geom2")) +) +``` + +Now, take a look at the results: + +``` ++---+---+--------+ +|id1|id2|distance| ++---+---+--------+ +|a |b |2.0 | ++---+---+--------+ +``` + +You can readily see the minimum distance between the two polygons in the graph. + +## Three-dimensional minimum Cartesian distance + +Let’s take a look at how to compute the distance between two points, factoring in the elevation of the points. + +We will examine the distance between someone standing on top of the Empire State Building and someone at sea level. + +Let’s create the DataFrame: + +```python +empire_state_ground = Point(-73.9857, 40.7484, 0) +empire_state_top = Point(-73.9857, 40.7484, 380) +df = sedona.createDataFrame([ + (empire_state_ground, empire_state_top), +], ["point_a", "point_b"]) +``` + +Now compute the distance and the 3D distance between the points: + +```python +res = df.withColumn( + "distance", + ST_Distance(col("point_a"), col("point_b")) +).withColumn( + "3d_distance", + ST_3DDistance(col("point_a"), col("point_b")) +) +``` + +Take a look at the results: + +``` ++--------------------+--------------------+--------+-----------+ +| point_a| point_b|distance|3d_distance| ++--------------------+--------------------+--------+-----------+ +|POINT (-73.9857 4...|POINT (-73.9857 4...| 0.0| 380.0| ++--------------------+--------------------+--------+-----------+ +``` + +`ST_Distance` does not factor in the elevation of the point. `ST_3DDistance` factors in the elevation when measuring the distance. + +## Compute Frechet distance with Spark and Sedona + +Let’s create a Sedona DataFrame with the following linestrings: + + + +Here’s how to create the Sedona DataFrame: + +```python +a = LineString([(1, 1), (1, 3), (2, 4)]) +b = LineString([(1.1, 1), (1.1, 3), (3, 4)]) +c = LineString([(7, 1), (7, 3), (6, 4)]) +df = sedona.createDataFrame([ + (a, "a", b, "b"), + (a, "a", c, "c"), +], ["geometry1", "geometry1_id", "geometry2", "geometry2_id"]) +``` + +Compute the Frechet distance: + +```python +res = df.withColumn( + "frechet_distance", + ST_FrechetDistance(col("geometry1"), col("geometry2")) +) +``` + +Now view the results: + +``` +res.select("geometry1_id", "geometry2_id", "frechet_distance").show() + ++------------+------------+----------------+ +|geometry1_id|geometry2_id|frechet_distance| ++------------+------------+----------------+ +| a| b| 1.0| +| a| c| 6.0| ++------------+------------+----------------+ +``` + +This image visualizes the distances so you have a better intuition for the algorithm: + + + +## Compute the max distance between geometries with Spark and Sedona + +Suppose you have the following geometric objects: + + + +Here’s how to compute the max distance between some of these geometries. Run the computations: + +```python +res = df.withColumn( + "max_distance", + ST_MaxDistance(col("geom1"), col("geom2")) +) +``` + +Now view the results: + +``` +res.select("id1", "id2", "max_distance").show(truncate=False) + ++---+---+-----------------+ +|id1|id2|max_distance | ++---+---+-----------------+ +|a |b |8.246211251235321| +|a |c |7.615773105863909| ++---+---+-----------------+ +``` + +You can easily find the maximum distance between two geometric objects. + +## Conclusion + +Sedona enables you to perform various types of distance computations. It also allows you to compute distance based on different models of the Earth and more complex distance computations, like distance factoring in elevation. + +Ensure you use the distance function that best suits your analysis. diff --git a/docs/tutorial/concepts/spatial-joins.md b/docs/tutorial/concepts/spatial-joins.md index e58f30ee3e..68caec5ce3 100644 --- a/docs/tutorial/concepts/spatial-joins.md +++ b/docs/tutorial/concepts/spatial-joins.md @@ -17,13 +17,13 @@ under the License. --> -# Apache Sedona Spatial Joins +# Apache Sedona Spatial Joins with Spark This post explains how to perform spatial joins with Apache Sedona. You will learn about the different types of spatial joins and how to run them efficiently. This page provides basic examples that clearly illustrate the key conceptual points of spatial joins. It also elaborates on spatial join concepts for real-world-sized datasets and highlights key performance enhancements. -## Spatial join within +## Spatial join within using Spark Look at the following graph containing three points and two polygons. `point_b` is within `polygon_y`, `point_c` is within `polygon_x`, and `point_a` isn’t within any polygon. @@ -102,7 +102,7 @@ Here’s the same result: +--------+----------+ ``` -## Spatial join crosses +## Spatial join crosses with Spark Look at the following graph containing one polygon and two lines. `line_a` and `line_b` cross `polygon_x`. `line_c` does not cross `polygon_x`. @@ -132,7 +132,7 @@ Here is the result: A spatial join with `ST_Crosses` lets us identify the lines that cross the polygon. -## Spatial join with touches +## Spatial join with touches using Spark Suppose you have a polygon and two lines. `line_a` does not touch the polygon, and `line_b` does touch the polygon. See the following diagram: @@ -188,7 +188,7 @@ Here’s the result of the join: Now, let’s look at running a join to see if points are within a polygon. -## Spatial join overlaps +## Spatial join overlaps with Spark The following diagram shows two polygons and a few shapes. `polygon_a` overlaps `polygon_x`. Neither `line_b`, `line_c`, or `point_d` overlap with `polygon_y` or `polygon_x`. @@ -217,7 +217,7 @@ Here is the result: +--------+----------+ ``` -## Spatial join K-nearest neighbors (KNN spatial join) +## Spatial join K-nearest neighbors (KNN spatial join) with Spark Suppose you have tables with addresses and coffee shop locations. You’d like to find the two nearest coffee shops to each address. @@ -278,7 +278,7 @@ Here’s a visualization of the results: You can easily see the coffee shops that are closest to each address. -## Spatial distance join +## Spatial distance join with Spark Look at the following graph, which shows a point and different transit stations. Let’s perform a spatial join to find all the transit stations within 2.5 units of the point. @@ -335,7 +335,7 @@ Sedona is an excellent tool for finding locations within a certain distance from Sedona uses the Euclidean distance between two objects so the distance unit has the same CRS of the original coordinates. To directly operate on WGS84 coordinates with meter distance, you should use `ST_DistanceSphere`, `ST_DistanceSpheroid`, or `ST_DWithnin(useSpheroid = true)`. -## Spatial range join +## Spatial range join with Spark All joins triggered by `ST_Intersects`, `ST_Contains`, `ST_Within`, `ST_DWithin`, `ST_Touches`, and `ST_Crosses` are considered a range join. This section illustrates another range join, but we've already covered several range joins on this page. @@ -393,7 +393,7 @@ Here are the results: Range joins are helpful in many practical applications. -## Spatial join optimizations +## Spatial join optimizations for Sedona and Apache Spark You can optimize spatial joins by using better file formats, indexing your data, or optimizing your queries. diff --git a/mkdocs.yml b/mkdocs.yml index 4531553e21..eee89788d2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -70,6 +70,7 @@ nav: - Concepts: - Spatial Joins: tutorial/concepts/spatial-joins.md - Clustering Algorithms: tutorial/concepts/clustering-algorithms.md + - Distance: tutorial/concepts/distance-spark.md - Map visualization SQL app: - Scala/Java: tutorial/viz.md - Use Apache Zeppelin: tutorial/zeppelin.md
