Dear Ching-Mallete ,
There are machines master01, master02 and master03 for the cluster, I could see
the stderr log and stdout log in the directories /.../yarn/logs under those
machines.In the stderr there are some notes, I just use log to follow the track
of the bug, however log in mapPartitions could not be found, and any other log
is there.Will only Spark UI will see them? Or some places in the machine will
also see ...
Thank you in advance!Zhiliang
On Tuesday, November 10, 2015 11:42 AM, Deng Ching-Mallete
<[email protected]> wrote:
Hi Zhiliang,
You should be able to see them in the executor logs, which you can view via the
Spark UI, in the Executors page (stderr log).
HTH,Deng
On Tue, Nov 10, 2015 at 11:33 AM, Zhiliang Zhu <[email protected]>
wrote:
Hi All,
I need debug spark job, my general way is to print out the log, however, some
bug is in spark functions as mapPartitions etc, and not any log printed from
those functionscould be found...Would you help point what is way to the log in
the spark own function as mapPartitions? Or, what is general way to debug spark
job.
Thank you in advance!
Zhiliang
