Hi Sem, Nick, and all,

It is natural for users to expect a built-in DataFrame API for graph
processing, just as Spark provides DataFrames alongside RDDs and spark.ml
alongside the RDD-based spark.mllib. GraphFrames already provides that
graph abstraction, but it remains a separate library.

There is also a gap for Spark Connect users: GraphX’s RDD-based API is not
supported through Connect. Bringing GraphFrames into Spark, together with
contributions and expertise from the GraphFrames community, would address
both needs: a built-in DataFrame-oriented graph API and graph algorithms
available through Spark Connect.

Thanks,
Ruifeng

On Mon, Nov 9, 2026 at 4:21 PM Sem <[email protected]> wrote:

> Hello!
>
> It's difficult to say with any certainty, as it was 10 years ago.
> However, from some old code comments in the GraphFrames codebase, I can
> infer that the intention was to stabilize the API before merging it
> with the upstream.
>
> Best regards,
> Sem
>
> On Sun, 2026-11-08 at 18:16 -0500, Nicholas Chammas wrote:
> > Great to see this proposal. GraphFrames has long been a very useful
> > library, and I have used it many times.
> >
> > One historical curiosity: Why was this not originally a part of the
> > project? Was it just a historical accident, or was there some
> > original blocker which has since been resolved?
> >
> > NIck
> >
> >
> > > On Nov 8, 2026, at 2:39 PM, Sem via dev <[email protected]>
> > > wrote:
> > >
> > > Hello Apache Spark community,
> > >
> > > I would like to start a discussion about merging the GraphFrames
> > > project into the Apache Spark project.
> > >
> > > SPIP:
> > >
> https://docs.google.com/document/d/1v8OEV5Qqk5LE1XID4ZiPF5rLkAk-Zz6fp0tns9Z_TA8/edit?usp=sharing
> > >
> > > Shepherd: Ruifeng Zheng
> > >
> > > Following the deprecation of GraphX, there is no public, non-
> > > deprecated
> > > API for working with graph-like data. At the same time, the
> > > GraphFrames
> > > project (https://github.com/graphframes/graphframes) provides a
> > > solid
> > > foundation for representing and processing connected data as
> > > Spark's
> > > DataFrames, as well as a rich standard library of ready-to-use
> > > graph
> > > algorithms such as PageRank, Connected Components and Node
> > > Embeddings.
> > >
> > > The proposal suggests moving the GraphFrames code (Apache 2.0
> > > licence)
> > > to the Apache Spark project as an in-tree module.
> > >
> > > Please reply to this thread. A [VOTE] will follow once the
> > > discussion
> > > has concluded.
> > >
> > > Best regards,
> > > Sem
> > >
> > > -------------------------------------------------------------------
> > > --
> > > To unsubscribe e-mail: [email protected]
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: [email protected]
>

Reply via email to