Not alot of string messages, I need it mostly for debugging purposed which I will use on an ahdoc basis - manually add debug statements which returns info about the dataset etc. I would assume the strings will vary from 100-200lines max, that would be about 50-100KB if they are really long lines.
-Utkarsh On Mon, Feb 1, 2016 at 3:40 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > I wouldn't use accumulators for things which could get large, they can > become kind of a bottle neck. Do you have a lot of string messages you want > to bring back or only a few? > > On Mon, Feb 1, 2016 at 3:24 PM, Utkarsh Sengar <utkarsh2...@gmail.com> > wrote: > >> I am trying to debug code executed in executors by logging. Even when I >> add log4j's LOG.info(..) inside .map() I don't see it in mesos task logs in >> the corresponding slaves. >> Its anyway inefficient to keep checking multiple slaves for logs. >> >> One way to deal with this is to push logs to a central location. >> >> Another way (for debugging purposes) is to use accumulators . Is it >> advisable to use accumulators to push string from executors to driver? >> It will simplify things when I am debugging datasets, bugs which is hard >> to reproduce locally etc. >> >> Suggestions/comments? >> >> -Utkarsh >> > > > > -- > Cell : 425-233-8271 > Twitter: https://twitter.com/holdenkarau > -- Thanks, -Utkarsh
