+1 for this idea, it seems worth doing. Regarding Nick's comment, the precedent here probably isn't spark-packages (never ASF governed) but probably Apache Bahir (extensions to multiple distributed analytic platforms) which was the ASF's home for out-of-core Spark/Flink connectors, and eventually went to Attic in 2024. Two things sank it: scope sprawl across too many backends, and no real removal path which led to dead connectors piling up enlessly.
In any case, if we keep it Python DS only (+1 with Allison's proposal here as well) and it should be sustainable. Also, AI does lower the cost of writing connectors, but not owning them (especially over time). Review, security, and testing against live backends only grow with contribution volume. It would probably be useful to point the tooling at CI and verification, not maintenance. All in all, great proposal and thank you Allison for brining this up! On 2026/08/18 23:45:47 Allison Wang wrote: > Hi all, > > I would like to discuss whether Apache Spark should provide an experimental, > community-maintained home for the ecosystem around the PySpark Data Source > API > <https://spark.apache.org/docs/latest/api/python/tutorial/sql/python_data_source.html> > : > > > Proposed repository: apache/spark-python-datasources > > Existing implementation: allisonwang-db/pyspark-data-sources > <https://github.com/allisonwang-db/pyspark-data-sources> > > The intent would not be to make these data sources part of Spark core. > Instead, the repository would provide an Apache-governed place where the > community can collaborate on reusable implementations of the public Python > Data Source API, share practical examples, and grow the ecosystem around > the API without expanding Spark core itself. > > The existing project can serve as the initial contribution. It contains > batch and streaming readers and writers built with the public PySpark Data > Source API, covering a range of external systems and use cases. > > I propose starting with a deliberately lightweight model: > > - > > The repository would be experimental and community-supported. > - > > It would focus specifically on implementations built on the public PySpark > Data Source API. > > <https://spark.apache.org/docs/latest/api/python/tutorial/sql/python_data_source.html> > - > > Code in this repository would remain separate from Spark core and would > not carry the same compatibility or support guarantees as Spark itself. > - > > New contributions would go through community review, with > maintainability, dependencies, licensing, testing, and security taken into > consideration. > - > > Implementations that become unmaintained or no longer meet the > repository's requirements could be deprecated or removed through the normal > community process. > - > > The repository would be governed by the Apache Spark community and > follow ASF policies. > > The existing project is currently published on PyPI as pyspark-data-sources > using the pyspark_datasources import namespace. For continuity, I would > prefer to retain those names if they are compatible with ASF release and > branding requirements, but the package naming is not essential to this > proposal. I am willing to help maintain the repository, review > contributions, and support the release process. > > The main question I would like feedback on is whether the Spark community > thinks it is useful to provide this kind of lightweight, experimental home > for extensions built on a public Spark API, while keeping those > integrations explicitly outside Spark core. > > If the community supports this proposal, I will work with the Spark PMC on > the required JIRA and ASF IP-clearance steps, move the approved code to the > Apache repository, update the package metadata, and transfer PyPI > publishing to an ASF-controlled release process. Existing PyPI releases and > installation commands would remain unchanged. > > I would appreciate any feedback on this proposal. > > Thanks, > > Allison > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
