So it’s deprecated but I will review some basic graph X PRs as I would like us to bring graph X back to life — but under our current release structure we need to deprecate now if we want to be able to remove it in the next few years.
Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her On Tue, Nov 12, 2024 at 6:47 PM Ángel <angel.alvarez.pas...@gmail.com> wrote: > I thought that too ... until I read the message from Matei Zaharia: > > "Votes to deprecate both SparkR and GraphX have passed. These components > will officially be deprecated in Spark 4." > > Didn't know in open source you could deprecate things that have been there > years so lightly without carrying out any impact analysis and in the middle > of an active (and interesting, btw) discussion. > > El mar, 12 nov 2024, 21:59, Russell Jurney <russell.jur...@gmail.com> > escribió: > >> That is unfortunate. I saw someone volunteer to review my PRs. I thought >> there was a holdout? >> >> On Tue, Nov 12, 2024 at 12:56 PM Ángel <angel.alvarez.pas...@gmail.com> >> wrote: >> >>> Nope. didn't miss that, in fact, I mentioned that graphframes used >>> GraphX under the hood. >>> >>> The thing is ... even though we were trying to get maintainers the >>> deprecation of GraphX passed suddenly in the middle of that discussion. >>> >>> El mar, 12 nov 2024, 21:47, Russell Jurney <russell.jur...@gmail.com> >>> escribió: >>> >>>> I guess you missed where Reynold Xin suggested we instead bring >>>> GraphFrames into Spark and others agreed? >>>> >>>> On Tue, Nov 12, 2024 at 12:08 PM Ángel <angel.alvarez.pas...@gmail.com> >>>> wrote: >>>> >>>>> You only have to look at the subject of this thread of mails. It says >>>>> nothing about graphframes. I thought we were "fighting" against >>>>> deprecating >>>>> GraphX because it seemed not have any maintainers in quite a few time. >>>>> Maybe I got it wrong. >>>>> >>>>> El mar, 12 nov 2024, 19:12, Russell Jurney <russell.jur...@gmail.com> >>>>> escribió: >>>>> >>>>>> Not sure what you mean? GraphX is the core Apache Spark technology >>>>>> underneath GraphFrames - parts of GraphFrames use it. `git grep -i >>>>>> graphx | >>>>>> wc -l` shows 147 hits for `graphx` in GraphFrames master branch as of >>>>>> now. >>>>>> >>>>>> I started out getting familiar with the GraphFrames codebase with >>>>>> some low hanging fixes for housekeeping - GraphFrames hasn't been >>>>>> maintained, the docs are outdated and most examples didn't work. The >>>>>> friends graph didn't even load in all of the examples, as the Python >>>>>> graphframes.examples.Graphs interface changed to be object oriented. I am >>>>>> now a maintainer of the Github repo thanks to Weichen Xu, who is >>>>>> reviewing >>>>>> my PRs along with Reynold Xin (very nice, thanks!). I am getting familiar >>>>>> with GraphFrames internals before looking at what it takes to get it into >>>>>> Apache Spark proper. I was just letting people know that I am actively >>>>>> taking on the work that we are discussing. >>>>>> >>>>>> Next, I am also going to get to know GraphX in the next couple of >>>>>> weeks to better understand what moving GraphFrames into Spark would look >>>>>> like. >>>>>> >>>>>> In a week or so, I will announce the hackathon. Need to figure out a >>>>>> time that works for the people who've already expressed interest. >>>>>> >>>>>> Thanks, >>>>>> Russell Jurney >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Nov 12, 2024 at 6:48 AM Ángel <angel.alvarez.pas...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> But the goal wasn't to fix bugs in GraphX? What has that to do with >>>>>>> graphframes? >>>>>>> >>>>>>> El mar, 12 nov 2024, 12:58, Russell Jurney <russell.jur...@gmail.com> >>>>>>> escribió: >>>>>>> >>>>>>>> I started working on GraphFrames this weekend, got it building and >>>>>>>> started with some docs PRs. A lot of the example code no longer >>>>>>>> worked, so >>>>>>>> I fixed it. I'm updating the docs to indicate our plan to integrate it >>>>>>>> with >>>>>>>> Apache Spark. >>>>>>>> >>>>>>>> I'll announce a hackathon in the next week or so :) >>>>>>>> >>>>>>>> Russell >>>>>>>> >>>>>>>> On Wed, Oct 16, 2024 at 6:53 PM Russell Jurney < >>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>> >>>>>>>>> For starters I created a ticket. I'm going to work on the project >>>>>>>>> a bit and then name a date and time. >>>>>>>>> >>>>>>>>> https://github.com/graphframes/graphframes/issues/460 >>>>>>>>> >>>>>>>>> On Tue, Oct 15, 2024 at 7:48 PM Ángel < >>>>>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> We could create a prioritized list of the most important bugs to >>>>>>>>>> fix first and distribute the tasks among us. We can also share the >>>>>>>>>> knowledge we gain from resolving them. >>>>>>>>>> btw, what happened to the (great) hackathon idea? any date/s in >>>>>>>>>> mind? >>>>>>>>>> >>>>>>>>>> El mié, 16 oct 2024 a las 3:53, Russell Jurney (< >>>>>>>>>> russell.jur...@gmail.com>) escribió: >>>>>>>>>> >>>>>>>>>>> I've never used Visual Studio Code for Java / Scala, but I find >>>>>>>>>>> IntelliJ works really well. >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 15, 2024 at 4:58 PM Mich Talebzadeh < >>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> VSC is also another good candidate heavily used in the banks. >>>>>>>>>>>> Anyway you parted with money so all the best. >>>>>>>>>>>> >>>>>>>>>>>> cheers >>>>>>>>>>>> >>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>> >>>>>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>>>>>> College London >>>>>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>>>>>> London, United Kingdom >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>> essential to >>>>>>>>>>>> note that, as with any advice, quote "one test result is worth >>>>>>>>>>>> one-thousand expert opinions (Werner >>>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>
