Hm. Since it sounds like a plan why Russell you go ahead and create a SPIP for it, then, this discussion takes a formal approach and is documented. Otherwise we are just flogging a dead horse so to speak.
HTH Mich Talebzadeh, Architect | Data Engineer | Data Science | Financial Crime PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College London <https://en.wikipedia.org/wiki/Imperial_College_London> London, United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Wed, 13 Nov 2024 at 20:10, Russell Jurney <russell.jur...@gmail.com> wrote: > It might be, but graph processing is a desirable, very useful feature of > Spark. GraphX doesn't see more popularity because it never got a DataFrame > interface. If someone is willing to add one and maintain it, that seems > best of all. > > Russ > > On Wed, Nov 13, 2024 at 7:12 AM Ángel <angel.alvarez.pas...@gmail.com> > wrote: > >> Seems to me.... it would be easier to move GraphX to graphframes than the >> opposite. >> >> El mar, 8 oct 2024 a las 21:52, Reynold Xin (<r...@databricks.com.invalid>) >> escribió: >> >>> We can also consider the following: move GraphFrame into Spark, and make >>> GraphX an internal impl detail of GraphFrame. Then we can over time change >>> the implementation, simplify it (not sure if it is possible, but somebody >>> can look into it).... >>> >>> On Mon, Oct 7, 2024 at 7:04 PM Russell Jurney <russell.jur...@gmail.com> >>> wrote: >>> >>>> Took a look at recent activity. Spark 3.5 support >>>> <https://github.com/graphframes/graphframes/commit/e54f249605dde60787f9b41b88ed7d5872b7dfab> >>>> was >>>> added a year ago. I'm sure we'll add Spark 4 support as soon as it is out. >>>> >>>> There is a new issue to organize a GraphFrames Hackathon >>>> <https://github.com/graphframes/graphframes/issues/460>. Please sign >>>> up to help! https://github.com/graphframes/graphframes/issues/460 >>>> >>>> I seriously need GraphX and GraphFrames to make it... I have no other >>>> way of doing property graph motif matching on large graphs. It's kind of >>>> important to me. >>>> >>>> Some slides on my work with GraphFrames: >>>> >>>> [image: image.png] >>>> >>>> [image: image.png] >>>> >>>> [image: image.png] >>>> >>>> [image: image.png] >>>> >>>> [image: image.png] >>>> >>>> Russell >>>> >>>> >>>> On Mon, Oct 7, 2024 at 6:06 PM Holden Karau <holden.ka...@gmail.com> >>>> wrote: >>>> >>>>> That’s awesome! >>>>> >>>>> Twitter: https://twitter.com/holdenkarau >>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>> Pronouns: she/her >>>>> >>>>> >>>>> On Mon, Oct 7, 2024 at 5:42 PM Russell Jurney < >>>>> russell.jur...@gmail.com> wrote: >>>>> >>>>>> I’ll organize a hackathon. A friend wants to finish the >>>>>> implementation of Lucian modularity for GraphFrames. I’ll fix some GraphX >>>>>> bugs at it. >>>>>> >>>>>> I did just blog all about the motif matching in GraphFrames: >>>>>> >>>>>> https://blog.graphlet.ai/financial-crime-and-corruption-network-motifs-4cf2e8e10eb5 >>>>>> >>>>>> Russ >>>>>> >>>>>> On Mon, Oct 7, 2024 at 5:38 PM Holden Karau <holden.ka...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> So this discuss thread and the vote thread to deprecate to leave the >>>>>>> option of removing it during 4.X are probably the highest profile it’s >>>>>>> been >>>>>>> in years. >>>>>>> >>>>>>> In the past for parts of Spark I’ve cared about I’ve organized >>>>>>> virtual meetings to co-ordinate work — if your connected with some of >>>>>>> the >>>>>>> Spark+Graph community reaching out to find others and organizing a >>>>>>> meeting >>>>>>> could be a way to raise the profile a bit? Maybe organize a virtual >>>>>>> hackathon (I’m meaning to try this for some other things so happy to >>>>>>> share >>>>>>> what I learn from doing that)? >>>>>>> >>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>> Pronouns: she/her >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 7, 2024 at 5:02 PM Russell Jurney < >>>>>>> russell.jur...@gmail.com> wrote: >>>>>>> >>>>>>>> I’ll look for a bug to fix. If GraphX is outside of Spark, Spark >>>>>>>> would tend to break GraphFrames and it will be burdensome on an >>>>>>>> external >>>>>>>> project to keep up. Graph computing on Spark is implrtant to a lot of >>>>>>>> people, is there a way to raise visibility here? >>>>>>>> >>>>>>>> On Mon, Oct 7, 2024 at 4:24 PM Holden Karau <holden.ka...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> There are no specific tickets associated with the lack of >>>>>>>>> maintaince or this as the component has not been maintained for a >>>>>>>>> sufficiently long time. If your interested in taking it on that’s >>>>>>>>> wonderful, probably starting with fixing some bugs could be a great >>>>>>>>> place >>>>>>>>> to start and figure out if it’s something you want to do long term. >>>>>>>>> >>>>>>>>> I would recommend making a first bug fix in a actively maintained >>>>>>>>> area of Spark to get to >>>>>>>>> Know some reviewers since there is not anyone tracking the GraphX >>>>>>>>> PRs. >>>>>>>>> >>>>>>>>> As a note I don’t think GraphX is required for Graph Frames long >>>>>>>>> term, so another option would be to talk to the GraphFrames folks and >>>>>>>>> move >>>>>>>>> the GraphX code over to it. >>>>>>>>> >>>>>>>>> Ideally we’d have someone willing to act as a mentor or guide but >>>>>>>>> so far we have no volunteers (especially no one familiar with the >>>>>>>>> graph X >>>>>>>>> code). >>>>>>>>> >>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>> Pronouns: she/her >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Oct 7, 2024 at 3:25 PM Russell Jurney < >>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> I volunteer to maintain GraphX to keep GraphFrames a viable >>>>>>>>>> project. I don’t have a clear view on whether it works with Spark 4 >>>>>>>>>> or if >>>>>>>>>> it needs updates? I don’t have Spark commits but I’m a committer on >>>>>>>>>> Apache >>>>>>>>>> DataFu and mentored the Spark feature for it. >>>>>>>>>> >>>>>>>>>> Can someone tell me what is involved? Point me at a ticket? >>>>>>>>>> >>>>>>>>>> Russell >>>>>>>>>> >>>>>>>>>> On Mon, Oct 7, 2024 at 12:11 AM Erik Eklund < >>>>>>>>>> eekl...@definitivehc.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> We rely on GraphX for an important component of our product. And >>>>>>>>>>> we really want it to stay a typed interface. Please keep GraphX. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Erik >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *From: *Holden Karau <holden.ka...@gmail.com> >>>>>>>>>>> *Date: *Sunday, October 6, 2024 at 06:22 >>>>>>>>>>> *To: *Ángel <angel.alvarez.pas...@gmail.com> >>>>>>>>>>> *Cc: *Russell Jurney <russell.jur...@gmail.com>, Mich >>>>>>>>>>> Talebzadeh <mich.talebza...@gmail.com>, Spark dev list < >>>>>>>>>>> dev@spark.apache.org>, user @spark <u...@spark.apache.org> >>>>>>>>>>> *Subject: *Re: [DISCUSS] Deprecate GraphX OR Find new >>>>>>>>>>> maintainers interested in GraphX OR leave it as is? >>>>>>>>>>> >>>>>>>>>>> So are there companies using it? And are they willing to >>>>>>>>>>> contribute to maintaining it? >>>>>>>>>>> >>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>> >>>>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f%3fq%3dhk_email&c=E,1,OT9ylxCx5xRNCToPSzu0VEvefs4uts16fTBydH2NiLHMGEwLjrEXgkhU8W-Ai6xD8VDMyWea44GBMOEecMNdapaZKZbBTrZpquOBKi6YRlqu-FVAzji6-w,,&typo=1> >>>>>>>>>>> >>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>> https://amzn.to/2MaRAG9 >>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,h0ccgHctUPRY4zAN_qZ-qdBgLDpQLtm7KaOL4u12U4PR7PeJ4MUBOS8bbD7CNssUIMqRMvY_pOqbh7PfLY0lRpQh9mfqBC0KnSHBZzxxSJJr-55r5kv6YjYwrA,,&typo=1> >>>>>>>>>>> >>>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>>>> >>>>>>>>>>> Pronouns: she/her >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Oct 5, 2024 at 9:17 PM Ángel < >>>>>>>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> That would definitely affect companies using GraphX, but at >>>>>>>>>>> least they’d have the choice to migrate their code. >>>>>>>>>>> >>>>>>>>>>> I think that’s probably the way to go. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> El dom, 6 oct 2024 a las 6:09, Holden Karau (< >>>>>>>>>>> holden.ka...@gmail.com>) escribió: >>>>>>>>>>> >>>>>>>>>>> So removing GraphX from Spark would not prevent GraphFrames from >>>>>>>>>>> continuing, they could pick up the GraphX source and incorporate it >>>>>>>>>>> into >>>>>>>>>>> their project. >>>>>>>>>>> >>>>>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>>>>> >>>>>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f%3fq%3dhk_email&c=E,1,9xMMQlY7gtmkqxT0NTmS8KMg4wOUjw0PWKM-oepAYAkE-SiM5pyXCb80AuRZYJ4zMIedVlwVMAKi_eh52Hof0LsteXx2eIslnsDBdmVeuocpILpneg,,&typo=1> >>>>>>>>>>> >>>>>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>> https://amzn.to/2MaRAG9 >>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,kbGbMBRMidAYi0aqUmj949vRahpEjVzSgJv_YYtO5EteSXZy4RrMYXJU48mN2CyS5sdovsgiFAAiBLnyQ29gCCn8xbTrEJmfIhjtH7tD4N31VUoLtQ,,&typo=1> >>>>>>>>>>> >>>>>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>>>>> >>>>>>>>>>> Pronouns: she/her >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Oct 5, 2024 at 5:22 PM Russell Jurney < >>>>>>>>>>> russell.jur...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> A lot of people like me use GraphFrames for its connected >>>>>>>>>>> components implementation and its motif matching feature. I am >>>>>>>>>>> willing to >>>>>>>>>>> work on it to keep it alive. They did a 0.8.3 release not too long >>>>>>>>>>> ago. >>>>>>>>>>> Please keep GraphX alive. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Oct 5, 2024 at 3:44 PM Mich Talebzadeh < >>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> I added the user list as they may have vested interest here and >>>>>>>>>>> and hopefully can contribute >>>>>>>>>>> >>>>>>>>>>> Few suggestions: >>>>>>>>>>> >>>>>>>>>>> 1. Data-Driven Decision Making: Return to the core >>>>>>>>>>> metrics—analyze usage trends, performance benchmarks, and the >>>>>>>>>>> actual impact >>>>>>>>>>> on businesses that rely on GraphX. Objectivity can be restored >>>>>>>>>>> by letting >>>>>>>>>>> data speak louder than opinions so to speak. >>>>>>>>>>> 2. Broaden the Discussion: Engage more stakeholders from >>>>>>>>>>> diverse backgrounds (especially spark users) to bring in new >>>>>>>>>>> perspectives >>>>>>>>>>> and counterbalance the more vocal but potentially narrow >>>>>>>>>>> interests of core >>>>>>>>>>> maintainers or open-source contributors. >>>>>>>>>>> 3. Define Clear Criteria for Decision Making: Agree on a set >>>>>>>>>>> of objective criteria by which the project’s future will be >>>>>>>>>>> judged. These >>>>>>>>>>> could include market demand, contribution levels, maintenance >>>>>>>>>>> costs, >>>>>>>>>>> alternative solutions, and alignment with the overall Spark >>>>>>>>>>> ecosystem >>>>>>>>>>> goals. Some have already been covered. >>>>>>>>>>> 4. Timely Conclusion of Discussions: Set a timeline for >>>>>>>>>>> making a decision. Long, open-ended discussions tend to lose >>>>>>>>>>> focus. Putting >>>>>>>>>>> deadlines forces participants to focus on key issues and >>>>>>>>>>> prevents endless >>>>>>>>>>> debates. >>>>>>>>>>> 5. Borrowing from commercial settings, it is often necessary >>>>>>>>>>> for a strong leadership team to step in and make the final >>>>>>>>>>> decision after >>>>>>>>>>> considering the input. When the objectivity of discussions >>>>>>>>>>> starts to wane, >>>>>>>>>>> leadership needs to cut through the round discussions and steer >>>>>>>>>>> towards >>>>>>>>>>> action based on business and technical realities. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> HTH >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>>>>> >>>>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>>>>> College London >>>>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>>>>> >>>>>>>>>>> London, United Kingdom >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [image: Image removed by sender.] view my Linkedin profile >>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fen.everybodywiki.com%2fMich_Talebzadeh&c=E,1,U1JaGVMkko53HkJO5fwmkIXfziTOWL3K1CkAeHwFG55TbZQUd5xVNLGpLt2o0ytujE6zaLpqU2GWCZqHSbo3SU4Wh9Rl8NG4bWPbFWUwyw,,&typo=1> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>> essential to >>>>>>>>>>> note that, as with any advice, quote "one test result is worth >>>>>>>>>>> one-thousand expert opinions (Werner >>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, 5 Oct 2024 at 06:26, Ángel < >>>>>>>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> I completely agree with everyone here. I don’t think the issue >>>>>>>>>>> is deprecating it; to me, the problem lies in not providing a new >>>>>>>>>>> and >>>>>>>>>>> better solution for handling graphs in Spark. In the past, I used >>>>>>>>>>> GraphX >>>>>>>>>>> via GraphFrames for record linkage, and I found it both useful and >>>>>>>>>>> effective. Is there any discussion about a potential replacement? >>>>>>>>>>> >>>>>>>>>>> I’d be willing to help maintain GraphX, though I don’t have >>>>>>>>>>> previous experience with maintaining open-source projects. All I can >>>>>>>>>>> promise is good intentions, willingness to learn and lots of energy >>>>>>>>>>> and >>>>>>>>>>> passion. Is that enough? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Btw, what's your take on this? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> · *GraphX* will be deprecated in favor of a new >>>>>>>>>>> graphing component, SparkGraph, based on Cypher >>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fneo4j.com%2fdeveloper%2fcypher-query-language%2f&c=E,1,5sP_K0oxQDLYIfWhFPwgNEmTuXMR7tvCjLLcf_ZBAv7oIBySxARy9TyrqNkmZKfXwrIDrhe6TVBCUun2luRV_mAbSD4rooD9YRt5GYYgbHbBUYerg1mpA4Oe6eo,&typo=1>, >>>>>>>>>>> a much richer graph language than previously offered by GraphX. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://cloud.google.com/blog/products/data-analytics/introducing-spark-3-and-hadoop-3-on-dataproc-image-version-2-0 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> El sáb, 5 oct 2024 a las 2:17, Mark Hamstra (< >>>>>>>>>>> markhams...@gmail.com>) escribió: >>>>>>>>>>> >>>>>>>>>>> As I wrote to Holden privately, I might well change my vote to >>>>>>>>>>> be in >>>>>>>>>>> favor of a deprecation label combined with some effective means >>>>>>>>>>> of >>>>>>>>>>> communicating that this doesn't mean the end for GraphX if >>>>>>>>>>> interested >>>>>>>>>>> contributors come forward to rescue it. I don't like either the >>>>>>>>>>> idea >>>>>>>>>>> of keeping unmaintained code and public APIs around (especially >>>>>>>>>>> if >>>>>>>>>>> there are problems with them) or the idea of removing Spark >>>>>>>>>>> functionality just because no one has contributed to it for a >>>>>>>>>>> while. A >>>>>>>>>>> naked deprecation label feels somewhat drastic and pre-emptive >>>>>>>>>>> to me. >>>>>>>>>>> I don't expect that GraphX will be the last part of Spark to run >>>>>>>>>>> the >>>>>>>>>>> risk of death through neglect, and I think we need an effective >>>>>>>>>>> means >>>>>>>>>>> of encouraging resuscitation that a deprecation label on its own >>>>>>>>>>> does >>>>>>>>>>> not provide. On the other hand, if no one really is willing to >>>>>>>>>>> come to >>>>>>>>>>> the aid of GraphX or other neglected functionality given adequate >>>>>>>>>>> warning of possible removal, I'm not then opposed to the usual >>>>>>>>>>> deprecation and removal process. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Oct 4, 2024 at 4:10 PM Sean Owen <sro...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> > >>>>>>>>>>> > This is a reasonable discussion, but maybe the more practical >>>>>>>>>>> point is: are you sure you want to block this unilaterally? This >>>>>>>>>>> effectively makes a decision that GraphX cannot be removed for a >>>>>>>>>>> long >>>>>>>>>>> while. I'd understand it more if we had an active maintainer and/or >>>>>>>>>>> active >>>>>>>>>>> user proposing to veto, but my understanding is this is just a >>>>>>>>>>> proposal to >>>>>>>>>>> block this on behalf of some users, someone else who might do some >>>>>>>>>>> work and >>>>>>>>>>> hasn't to date for some reason. Add to that the fact that the 'pro' >>>>>>>>>>> arguments all seem to be arguments for working on GraphFrames, and >>>>>>>>>>> I find >>>>>>>>>>> this somewhat drastic. >>>>>>>>>>> > >>>>>>>>>>> > On Fri, Oct 4, 2024 at 5:23 PM Mark Hamstra < >>>>>>>>>>> markhams...@gmail.com> wrote: >>>>>>>>>>> >> >>>>>>>>>>> >> "You can't say nothing is removable until there are no users." >>>>>>>>>>> >> >>>>>>>>>>> >> That is not what I am saying. Rather, I am countering what >>>>>>>>>>> others seem >>>>>>>>>>> >> to be suggesting: There are no users and no interest, >>>>>>>>>>> therefore we can >>>>>>>>>>> >> and should deprecate. >>>>>>>>>>> >> >>>>>>>>>>> >> On Fri, Oct 4, 2024 at 3:10 PM Sean Owen <sro...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >> > >>>>>>>>>>> >> > I could flip this argument around. More strongly, not being >>>>>>>>>>> deprecated means "won't be removed" and likewise implies support and >>>>>>>>>>> development. I don't think either of the latter have been true for >>>>>>>>>>> years. >>>>>>>>>>> What suggests this will change? A todo list is not going to do >>>>>>>>>>> anything, >>>>>>>>>>> IMHO. >>>>>>>>>>> >> > >>>>>>>>>>> >> > I'm also concerned about the cost of that, which I have >>>>>>>>>>> observed. GraphX PRs are almost certainly not going to be reviewed >>>>>>>>>>> because >>>>>>>>>>> of its state. Deprecation both communicates that reality, and >>>>>>>>>>> leaves an >>>>>>>>>>> option open, whereas not deprecating forecloses that option for a >>>>>>>>>>> while. >>>>>>>>>>> >> > >>>>>>>>>>> >> > I don't think the question is, does anyone use it? because >>>>>>>>>>> anyone can continue to use it -- in Spark 3.x for sure, and in 4.x >>>>>>>>>>> if not >>>>>>>>>>> removed. >>>>>>>>>>> >> > You can't say nothing is removable until there are no users. >>>>>>>>>>> >> > >>>>>>>>>>> >> > Also, why would GraphFrames not be the logical home of this >>>>>>>>>>> going forward anyway? which I think is the subtext. >>>>>>>>>>> >> > >>>>>>>>>>> >> > On Fri, Oct 4, 2024 at 4:56 PM Mark Hamstra < >>>>>>>>>>> markhams...@gmail.com> wrote: >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> I'm -1(*) because, while it technically means "might be >>>>>>>>>>> removed in the >>>>>>>>>>> >> >> future", I think developers and users are more prone to >>>>>>>>>>> interpret >>>>>>>>>>> >> >> something being marked as deprecated as "very likely will >>>>>>>>>>> be removed >>>>>>>>>>> >> >> in the future, so don't depend on this or waste your time >>>>>>>>>>> contributing >>>>>>>>>>> >> >> to its further development." I don't think the latter is >>>>>>>>>>> what we want >>>>>>>>>>> >> >> just because something hasn't been updated meaningfully in >>>>>>>>>>> a while. >>>>>>>>>>> >> >> There have been How To articles for GraphX and Graph >>>>>>>>>>> Frames posted in >>>>>>>>>>> >> >> the not too distant past, and the Google Search trend >>>>>>>>>>> shows a pretty >>>>>>>>>>> >> >> steady level of interest, not a decline to zero, so I >>>>>>>>>>> don't think that >>>>>>>>>>> >> >> it is accurate to declare that there is no use or interest >>>>>>>>>>> in GraphX. >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> Unless retaining GraphX is imposing significant costs on >>>>>>>>>>> continuing >>>>>>>>>>> >> >> Spark development, I can't support deprecating GraphX. I >>>>>>>>>>> can support >>>>>>>>>>> >> >> encouraging GraphX and Graph Frames development through >>>>>>>>>>> something like >>>>>>>>>>> >> >> a To Do list or document of "What we'd like to see in the >>>>>>>>>>> way of >>>>>>>>>>> >> >> further development of Spark's graph processing >>>>>>>>>>> capabilities" -- i.e., >>>>>>>>>>> >> >> things that encourage and support new contributions to >>>>>>>>>>> address any >>>>>>>>>>> >> >> shortcomings in Spark's graph processing, not things that >>>>>>>>>>> discourage >>>>>>>>>>> >> >> contributions and use in the way that I believe simply >>>>>>>>>>> declaring >>>>>>>>>>> >> >> GraphX to be deprecated would. >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> On Sun, Sep 29, 2024 at 11:04 AM Holden Karau < >>>>>>>>>>> holden.ka...@gmail.com> wrote: >>>>>>>>>>> >> >> > >>>>>>>>>>> >> >> > Since we're getting close to cutting a 4.0 branch I'd >>>>>>>>>>> like to float the idea of officially deprecating Graph X. What that >>>>>>>>>>> would >>>>>>>>>>> mean (to me) is we would update the docs to indicate that Graph X is >>>>>>>>>>> deprecated and it's APIs may be removed at anytime in the future. >>>>>>>>>>> >> >> > >>>>>>>>>>> >> >> > Alternatively, we could mark it as "unmaintained and in >>>>>>>>>>> search of maintainers" with a note that if no maintainers are >>>>>>>>>>> found, we may >>>>>>>>>>> remove it in a future minor version. >>>>>>>>>>> >> >> > >>>>>>>>>>> >> >> > Looking at the source graph X, I don't see any >>>>>>>>>>> meaningful active development going back over three years*. There >>>>>>>>>>> is even a >>>>>>>>>>> thread on user@ from 2017 asking if graph X is maintained >>>>>>>>>>> anymore, with no response from the developers. >>>>>>>>>>> >> >> > >>>>>>>>>>> >> >> > Now I'm open to the idea that GraphX is stable and >>>>>>>>>>> "works as is" and simply doesn't require modifications but given >>>>>>>>>>> the user >>>>>>>>>>> thread I'm a little concerned here about bringing this API with us >>>>>>>>>>> into >>>>>>>>>>> Spark 4 if we don't have anyone signed up to maintain it. >>>>>>>>>>> >> >> > >>>>>>>>>>> >> >> > * Excluding globally applied changes >>>>>>>>>>> >> >> > -- >>>>>>>>>>> >> >> > Twitter: https://twitter.com/holdenkarau >>>>>>>>>>> >> >> > Fight Health Insurance: >>>>>>>>>>> https://www.fighthealthinsurance.com/ >>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.fighthealthinsurance.com%2f&c=E,1,9CeJ-bKUShnxOFZMc15zJG1qgfAB9rnSDzrmLzNiXb8qE0NXedNCoZy4HobcS7laOMqtvJzYjvDzjBld1FaCPZpOBW6cf1l_xaG4bEbjYoDpNG0zuQ9_K5TW&typo=1> >>>>>>>>>>> >> >> > Books (Learning Spark, High Performance Spark, etc.): >>>>>>>>>>> https://amzn.to/2MaRAG9 >>>>>>>>>>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2famzn.to%2f2MaRAG9&c=E,1,HJPBNbN3nfUZcb0-2OgveqIE5I5lvPSv-bOfRXIprFdSsGMlNq15o6rueLf2ZQRfytMu0-t3IxSjYou2uuPzUrSAqJ0LV42n2hG8rnkkpN4AA5w4mQZFTs4,&typo=1> >>>>>>>>>>> >> >> > YouTube Live Streams: >>>>>>>>>>> https://www.youtube.com/user/holdenkarau >>>>>>>>>>> >> >> > Pronouns: she/her >>>>>>>>>>> >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> >> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>>>> >> >> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>>>> >>>>>>>>>>>
