Hi
I am wrote a little java job to try and figure out how RDD pipe works.
Bellow is my test shell script. If in the script I turn on debugging I get
output. In my console. If debugging is turned off in the shell script, I do
not see anything in my console. Is this a bug or feature?
I am running the job locally on a Mac
Thanks
Andy
Here is my Java
rdd.pipe(pwd + "/src/main/bin/RDDPipe.sh").collect();
#!/bin/sh
#
# Use this shell script to figure out how spark RDD pipe() works
#
set -x # turns shell debugging on
#set +x # turns shell debugging off
while read x ;
do
echo RDDPipe.sh $x ;
Done
Here is the output if debugging is turned on
$ !grep
grep RDDPipe run.sh.out
+ echo RDDPipe.sh 0
+ echo RDDPipe.sh 0
+ echo RDDPipe.sh 2
+ echo RDDPipe.sh 0
+ echo RDDPipe.sh 3
+ echo RDDPipe.sh 0
+ echo RDDPipe.sh 0
$
