Finally revisiting this issue. I can now build Spark through IntelliJ (I had to delete `.idea/` and reload the project, making sure to import the Maven config), but am struggling to get the breakpoint/debugging to work.
I setup IntelliJ per the instructions under Debug Spark Remotely <http://spark.apache.org/developer-tools.html>, but the debugger never seems to connect to the JVM. If I switch the debugger mode from "Listen to remote JVM" (which is recommended by the guide) to "Attach to remote JVM", I can get the debugger to talk to the JVM, but my breakpoint still doesn't fire. So to elaborate, I have a) a breakpoint set inside CatalogSuite.scala, and b) the IntelliJ debugger running in "Listen" mode. Then I run this: ``` $ ./build/sbt > set javaOptions in Test += "-agentlib:jdwp=transport=dt_socket,server=n,address=192.168.1.146:5005,suspend=y,onthrow=<FQ exception class name>,onuncaught=<y/n>" > testOnly org.apache.spark.sql.internal.CatalogSuite ``` (I copied those Java options from the IntelliJ debug config window.) But the breakpoint doesn't fire. Any ideas as to what I'm missing? On Tue, Dec 17, 2019 at 12:36 AM Sean Owen <sro...@gmail.com> wrote: > I just make a new test suite or something, set breakpoints, and > execute it in IJ. That generally works fine. You may need to set the > run configuration to have the right working dir (Spark project root), > and set the right system property to say 'this is running in a test' > in some cases. What are you having trouble with, does it build? > > On Mon, Dec 16, 2019 at 11:27 PM Nicholas Chammas > <nicholas.cham...@gmail.com> wrote: > > > > I normally stick to the Python parts of Spark, but I am interested in > walking through the DSv2 code and understanding how it works. I tried > following the "IDE Setup" section of the developer tools page, but quickly > hit several problems loading the project. > > > > Can anyone point to a tutorial or guide on running Spark through a > debugger, for folks like me who are not regular IntelliJ or Scala users? > > > > My goal is just to be able to run something like > `spark.read.text(...).show()` and follow it step by step through the > codebase. > > > > Nick > > >
