Merged, thanks guys! Russ
On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <bjornjorgen...@gmail.com> wrote: > I tested stuff in this PR > https://github.com/graphframes/graphframes/pull/477 and then I made this > PR https://github.com/graphframes/graphframes/pull/478 > > søn. 12. jan. 2025 kl. 23:10 skrev Ángel <angel.alvarez.pas...@gmail.com>: > >> Hi Russell, >> >> I've just got the OOM error during Test 13. I'm running it from IntelliJ >> on Windows with Java 11. >> >> [image: image.png] >> I'll look into it over the course of the next week. >> >> Regards, >> Ángel >> >> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<russell.jur...@gmail.com>) >> escribió: >> >>> Friends of GraphFrames (github.com/graphframes/graphframes), I have a >>> question for you... >>> >>> I can't get the unit test 'two components and two dangling vertices' in >>> the org.graphframes.lib.ConnectedComponentsSuite >>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148> >>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a >>> little stuck on completing a docs release with a motif finding tutorial >>> <https://github.com/graphframes/graphframes/pull/473> due to this issue. >>> >>> The problem is outlined in this gist: >>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf >>> >>> Can someone else please try this and see if it passes on the master >>> branch? >>> >>> > build/sbt clean compile package test >>> >>> I've tried giving it lots of RAM just to see if it would help, as much >>> as 32g driver and 16g for executors and... it has no effect. The test graph >>> is 8 nodes and 6 edges >>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>, >>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of >>> my CPU get used, it spikes as indicated in the image in the gist. >>> >>> I am running the following setup: >>> >>> * Ubuntu 20.04 (22.04 in the Docker image) >>> * OpenJDK 11 (I also tried 8, same problem) >>> * Scala 2.12.20 (I also tried 2.13, same problem) >>> * Python 3.11 (I also tried 3.9, same problem) >>> >>> Or I am running the Dockerfile in the gist >>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile> >>> . >>> >>> Any help much appreciated! Thanks >>> >>> ----------------------------------------------------------------- >>> Oh, some new community stuff for GraphFrames. Hackathon announced next >>> week :) >>> >>> >>> - GraphFrames Mailing List <https://groups.google.com/g/graphframes/>: >>> ask questions about GraphFrames on our Google Group >>> - #graphframes Discord Channel on GraphGeeks >>> <https://discord.com/channels/1162999022819225631/1326257052368113674> >>> >>> Thanks! >>> Russell Jurney @rjurney <http://twitter.com/rjurney> >>> russell.jur...@gmail.com LI <http://linkedin.com/in/russelljurney> FB >>> <http://facebook.com/jurney> datasyndrome.com >>> >> > > -- > Bjørn Jørgensen > Vestre Aspehaug 4, 6010 Ålesund > <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g> > Norge > <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g> > > +47 480 94 297 >
