Re: Using accumulator to push custom logs to driver

2016-02-01 Thread Holden Karau
Ah if its manual ad-hoc logging of the 100 to 200 lines then thats probably OK. On Mon, Feb 1, 2016 at 3:48 PM, Utkarsh Sengar wrote: > Not alot of string messages, I need it mostly for debugging purposed which > I will use on an ahdoc basis - manually add debug statements which returns > info a

Re: Using accumulator to push custom logs to driver

2016-02-01 Thread Utkarsh Sengar
Not alot of string messages, I need it mostly for debugging purposed which I will use on an ahdoc basis - manually add debug statements which returns info about the dataset etc. I would assume the strings will vary from 100-200lines max, that would be about 50-100KB if they are really long lines.

Re: Using accumulator to push custom logs to driver

2016-02-01 Thread Holden Karau
I wouldn't use accumulators for things which could get large, they can become kind of a bottle neck. Do you have a lot of string messages you want to bring back or only a few? On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar wrote: > I am trying to debug code executed in executors by logging. Even