Use spark.executor.extraJavaOptions
https://spark.apache.org/docs/latest/configuration.html#runtime-environment
On Thu, Oct 15, 2020 at 1:22 PM Akshat Bordia
wrote:
> Hi,
>
> We are trying to debug an issue with Spark and need to connect a remote
> debugger to the executors thread.
Hi,
We are trying to debug an issue with Spark and need to connect a remote
debugger to the executors thread. The general options with spark-submit (=-
agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005) seems to
allow only debugging driver thread. Does anyone have any insight on
ttp://spark.apache.org/developer-tools.html>, but the debugger never
seems to connect to the JVM. If I switch the debugger mode from "Listen to
remote JVM" (which is recommended by the guide) to "Attach to remote JVM",
I can get the debugger to talk to the JVM, but my breakpoint
p" section of the developer tools page, but quickly hit several
> problems loading the project.
>
> Can anyone point to a tutorial or guide on running Spark through a debugger,
> for folks like me who are not regular IntelliJ or Scala users?
>
> My goal is just to be able t
lems loading the project.
Can anyone point to a tutorial or guide on running Spark through a
debugger, for folks like me who are not regular IntelliJ or Scala users?
My goal is just to be able to run something like
`spark.read.text(...).show()` and follow it step by step through the
codebase.
Nick
Hi Josh,
I came across this post when looking for a debugger or RDD visualization
tool for Spark. I am using Spark 0.9.1 and upgrading soon to Spark 1.0. The
links you posted are dead. Can you please direct me to how I can debug my
existing Spark job.
Will I need to edit my existing job's
