Hi Patrick
How do I know which part of the code is in the driver and which in task?
The structure of my code is as below-
Š
Static boolean done=false;
Š
Public static void main(..
..
JavaRDD<String> lines = ..
..
While (!done) {
..
While (..) {
JavaPairRDD<Integer, List<Integer>> labs1 = labs.map (new PairFunction<Š );
!! Here I have System.out.println (A)
} // inner while
!! Here I have System.out.println (B)
If (Š) {
Done = true;
!! Also here some System.out.println (C)
Break;
}
Else {
If (Š) {
!! More System.out.println (D)
labs = labs.map(Š) ;
}
}
} // outer while
!! Even more System.out.println (E)
} // main
} //class
I get the console outputs on the master for (B) and (E). I do not see any
stdout in the worker node. I find the stdout and stderr in the
<spark>/work/<appid>/0/. I see output
in stderr but not in stdout.
I do get all the outputs on the console when I run it in local mode.
Sorry I am new and may be asking some naïve question but it is really
confusing to me. Thanks for your help.
Ranjan
On 3/9/14, 10:50 PM, "Patrick Wendell" <[email protected]> wrote:
>Hey Sen,
>
>Is your code in the driver code or inside one of the tasks?
>
>If it's in the tasks, the place you would expect these to be is in
>stdout file under <spark>/<appid>/work/[stdout/stderr]. Are you seeing
>at least stderr logs in that folder? If not then the tasks might not
>be running on the workers machines. If you see stderr but not stdout
>that's a bit of a puzzler since they both go through the same
>mechanism.
>
>- Patrick
>
>On Sun, Mar 9, 2014 at 2:32 PM, Sen, Ranjan [USA] <[email protected]>
>wrote:
>> Hi
>> I have some System.out.println in my Java code that is working ok in a
>>local
>> environment. But when I run the same code on a standalone mode in a EC2
>> cluster I do not see them at the worker stdout (in the worker node under
>> <spark location>/work ) or at the driver console. Could you help me
>> understand how do I troubleshoot?
>>
>> Thanks
>> Ranjan
