Are you sure that temporarily disabling a global setting like AQE is the best approach to fix this issue? I increased the number of shuffle partitions in the Spark session configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the "checkpoint interval" test ran perfectly without throwing an OOM error. Why? No idea, but it worked.
El lun, 13 ene 2025 a las 16:45, Russell Jurney (<russell.jur...@gmail.com>) escribió: > Merged, thanks guys! > > Russ > > On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <bjornjorgen...@gmail.com> > wrote: > >> I tested stuff in this PR >> https://github.com/graphframes/graphframes/pull/477 and then I made this >> PR https://github.com/graphframes/graphframes/pull/478 >> >> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <angel.alvarez.pas...@gmail.com >> >: >> >>> Hi Russell, >>> >>> I've just got the OOM error during Test 13. I'm running it from IntelliJ >>> on Windows with Java 11. >>> >>> [image: image.png] >>> I'll look into it over the course of the next week. >>> >>> Regards, >>> Ángel >>> >>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (< >>> russell.jur...@gmail.com>) escribió: >>> >>>> Friends of GraphFrames (github.com/graphframes/graphframes), I have a >>>> question for you... >>>> >>>> I can't get the unit test 'two components and two dangling vertices' >>>> in the org.graphframes.lib.ConnectedComponentsSuite >>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148> >>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a >>>> little stuck on completing a docs release with a motif finding tutorial >>>> <https://github.com/graphframes/graphframes/pull/473> due to this >>>> issue. >>>> >>>> The problem is outlined in this gist: >>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf >>>> >>>> Can someone else please try this and see if it passes on the master >>>> branch? >>>> >>>> > build/sbt clean compile package test >>>> >>>> I've tried giving it lots of RAM just to see if it would help, as much >>>> as 32g driver and 16g for executors and... it has no effect. The test graph >>>> is 8 nodes and 6 edges >>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>, >>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of >>>> my CPU get used, it spikes as indicated in the image in the gist. >>>> >>>> I am running the following setup: >>>> >>>> * Ubuntu 20.04 (22.04 in the Docker image) >>>> * OpenJDK 11 (I also tried 8, same problem) >>>> * Scala 2.12.20 (I also tried 2.13, same problem) >>>> * Python 3.11 (I also tried 3.9, same problem) >>>> >>>> Or I am running the Dockerfile in the gist >>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile> >>>> . >>>> >>>> Any help much appreciated! Thanks >>>> >>>> ----------------------------------------------------------------- >>>> Oh, some new community stuff for GraphFrames. Hackathon announced next >>>> week :) >>>> >>>> >>>> - GraphFrames Mailing List >>>> <https://groups.google.com/g/graphframes/>: ask questions about >>>> GraphFrames on our Google Group >>>> - #graphframes Discord Channel on GraphGeeks >>>> <https://discord.com/channels/1162999022819225631/1326257052368113674> >>>> >>>> Thanks! >>>> Russell Jurney @rjurney <http://twitter.com/rjurney> >>>> russell.jur...@gmail.com LI <http://linkedin.com/in/russelljurney> FB >>>> <http://facebook.com/jurney> datasyndrome.com >>>> >>> >> >> -- >> Bjørn Jørgensen >> Vestre Aspehaug 4, 6010 Ålesund >> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g> >> Norge >> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g> >> >> +47 480 94 297 >> >
