Thanks everyone for the feedback! Since there are no objections, I will work with a PMC member to create the repo.
On Sun, Aug 23, 2026 at 8:31 AM Uroš Bojanić <[email protected]> wrote: > +1 for this idea, it seems worth doing. > > Regarding Nick's comment, the precedent here probably isn't spark-packages > (never ASF governed) but probably Apache Bahir (extensions to multiple > distributed analytic platforms) which was the ASF's home for out-of-core > Spark/Flink connectors, and eventually went to Attic in 2024. Two things > sank it: scope sprawl across too many backends, and no real removal path > which led to dead connectors piling up enlessly. > > In any case, if we keep it Python DS only (+1 with Allison's proposal here > as well) and it should be sustainable. Also, AI does lower the cost of > writing connectors, but not owning them (especially over time). Review, > security, and testing against live backends only grow with contribution > volume. It would probably be useful to point the tooling at CI and > verification, not maintenance. > > All in all, great proposal and thank you Allison for brining this up! > > On 2026/08/18 23:45:47 Allison Wang wrote: > > Hi all, > > > > I would like to discuss whether Apache Spark should provide an > experimental, > > community-maintained home for the ecosystem around the PySpark Data > Source > > API > > < > https://spark.apache.org/docs/latest/api/python/tutorial/sql/python_data_source.html > > > > : > > > > > > Proposed repository: apache/spark-python-datasources > > > > Existing implementation: allisonwang-db/pyspark-data-sources > > <https://github.com/allisonwang-db/pyspark-data-sources> > > > > The intent would not be to make these data sources part of Spark core. > > Instead, the repository would provide an Apache-governed place where the > > community can collaborate on reusable implementations of the public > Python > > Data Source API, share practical examples, and grow the ecosystem around > > the API without expanding Spark core itself. > > > > The existing project can serve as the initial contribution. It contains > > batch and streaming readers and writers built with the public PySpark > Data > > Source API, covering a range of external systems and use cases. > > > > I propose starting with a deliberately lightweight model: > > > > - > > > > The repository would be experimental and community-supported. > > - > > > > It would focus specifically on implementations built on the public > PySpark > > Data Source API. > > < > https://spark.apache.org/docs/latest/api/python/tutorial/sql/python_data_source.html > > > > - > > > > Code in this repository would remain separate from Spark core and > would > > not carry the same compatibility or support guarantees as Spark > itself. > > - > > > > New contributions would go through community review, with > > maintainability, dependencies, licensing, testing, and security taken > into > > consideration. > > - > > > > Implementations that become unmaintained or no longer meet the > > repository's requirements could be deprecated or removed through the > normal > > community process. > > - > > > > The repository would be governed by the Apache Spark community and > > follow ASF policies. > > > > The existing project is currently published on PyPI as > pyspark-data-sources > > using the pyspark_datasources import namespace. For continuity, I would > > prefer to retain those names if they are compatible with ASF release and > > branding requirements, but the package naming is not essential to this > > proposal. I am willing to help maintain the repository, review > > contributions, and support the release process. > > > > The main question I would like feedback on is whether the Spark community > > thinks it is useful to provide this kind of lightweight, experimental > home > > for extensions built on a public Spark API, while keeping those > > integrations explicitly outside Spark core. > > > > If the community supports this proposal, I will work with the Spark PMC > on > > the required JIRA and ASF IP-clearance steps, move the approved code to > the > > Apache repository, update the package metadata, and transfer PyPI > > publishing to an ASF-controlled release process. Existing PyPI releases > and > > installation commands would remain unchanged. > > > > I would appreciate any feedback on this proposal. > > > > Thanks, > > > > Allison > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
