Re: [DISCUSS] SPIP: Add geospatial types to Spark

Sat, 29 Mar 2025 12:07:06 -0700

Thank you Menelaos, will do!

To give a little background, Jia and Sedona community, also GeoParquet community, and others really put much effort contributing to defining the Parquet and Iceberg geo types, which couldn't be done without their experience and help! 

But I do agree with Wenchen , now that the types are in most common data sources in ecosystem , I think Apache Spark as a common platform needs to have this type definition for inter-op, otherwise users of vanilla Spark cannot work with those data sources with stored geospatial data.  (Imo a similar rationale in adding timestamp nano in the other ongoing SPIP.).  

And like Wenchen said, the SPIP’s goal doesnt seem to be to fragment the ecosystem by implementing Sedona’s advanced geospatial analytic tech in Spark itself, which you may be right belongs in pluggable frameworks.  Menelaus may explain more about the SPIP goal.

I do hope there can be more collaboration across communities (like in Iceberg/Parquet collaboration) in getting Sedona community’s experience in making sure these type definitions are optimal , and compatible for Sedona.

Thanks!
Szehon


On Mar 29, 2025, at 8:04 AM, Menelaos Karavelas <menelaos.karave...@gmail.com> wrote:


Hello Szehon,

I just created a Google doc and also linked it in the JIRA:


Menelaos


On Mar 28, 2025, at 2:19 PM, Szehon Ho <szehon.apa...@gmail.com> wrote:

Thanks Menelaos, this is exciting !  Is there a google doc we can comment, or just on the JIRA?

Thanks
Szehon

On Fri, Mar 28, 2025 at 1:41 PM Ángel Álvarez Pascua <angel.alvarez.pas...@gmail.com> wrote:
Sorry, I only had a quick look at the proposal, looked for WKT and didn't find anything.

It's been years since I worked on geospatial projects and I'm not an expert (at all). Maybe starting with something simple but useful like conversion WKT<=>WKB?  


El vie, 28 mar 2025, 21:27, Menelaos Karavelas <menelaos.karave...@gmail.com> escribió:
In the SPIP Jira the proposal is to add the expressions ST_AsBinary, ST_GeomFromWKB, and ST_GeogFromWKB.
Is there anything else that you think should be added?

Regarding WKT, what do you think should be added?

- Menelaos


On Mar 28, 2025, at 1:02 PM, Ángel Álvarez Pascua <angel.alvarez.pas...@gmail.com> wrote:

What about adding support for WKT/WKB?

El vie, 28 mar 2025 a las 20:50, Ángel Álvarez Pascua (<angel.alvarez.pas...@gmail.com>) escribió:
+1 (non-binding)

El vie, 28 mar 2025, 18:48, Menelaos Karavelas <menelaos.karave...@gmail.com> escribió:
Dear Spark community,

I would like to propose the addition of new geospatial data types (GEOMETRY and GEOGRAPHY) which represent geospatial values as recently added as new logical types in the Parquet specification.

The new types should improve Spark’s ability to read the new Parquet logical types and perform some minimal meaningful operations on them.


Looking forward to your comments and feedback.


Best regards,

Menelaos Karavelas



Reply via email to