On Mon, Apr 21, 2014 at 7:59 PM, Jim Carroll <jimfcarr...@gmail.com> wrote:
> > I'm experimenting with a few things trying to understand how it's working. > I > took the JavaSparkPi example as a starting point and added a few System.out > lines. > > I added a system.out to the main body of the driver program (not inside of > any Functions). > I added another to the mapper. > I added another to the reducer. > > I set up a simple "standalone" distribution with a 1 master (no zookeeper) > and a 1 worker. > > The main println and the reducer println print out from the driver program. > > The mapper one doesn't print in the shell I started the worker in, nor does > it appear in any logs (in case stdout was redirected). > > However, the program executes (and the mapper executes on the worker) since > I can see the job run in the logs and I get an answer. > > Where should I be looking? > It is in the executor logs. The executor is a process started by the worker for your application, which runs your custom code. So the logs are on the worker machines. The default location is within the Spark directory under the "work" subdirectory. For example ~/spark-0.9.0-incubating/work/app-20140418164503-0025/0/stdout. It is often much easier to access this log from the UI. Click on the application ID on the master, or on a worker, and then you see the stdout/stderr links. Thanks. Apologies if this is a dumb question - I searched for answer but > only found a reference to another list posting where the user thought his > driver print statement should have printed on the worker. > I think it's a perfectly good question, and you might want to make a suggestion for adding it to the documentation somewhere. Jim > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/stdout-in-workers-tp4537.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
