Update:
I use shell script to execute the spark-shell, inside the my-script.sh:
$SPARK_HOME/bin/spark-shell < $HOME/test.scala > $HOME/test.log 2>&1 &
Although it correctly finish the println("hallo world"), but the strange thing
is that my-script.sh finished before spark-shell even finish executing the
script.
Best regards,
Henry Hung
From: MA33 YTHung1
Sent: Thursday, August 28, 2014 10:01 AM
To: [email protected]
Subject: how to correctly run scala script using spark-shell through stdin
(spark v1.0.0)
HI All,
Right now I'm trying to execute a script using this command:
nohup $SPARK_HOME/bin/spark-shell < $HOME/my-script.scala > $HOME/my-script.log
2>&1 &
my-script.scala just have 1 line of code:
println("hallo world")
But after waiting for a minute, I still don't receive the result from
spark-shell.
And the output log only contains:
Spark assembly has been built with Hive, including Datanucleus jars on classpath
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/data1/hadoop/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/data1/hadoop/hbase-0.96.0-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/08/28 09:38:27 INFO spark.SecurityManager: Changing view acls to: hadoop
14/08/28 09:38:27 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(hadoop)
14/08/28 09:38:27 INFO spark.HttpServer: Starting HTTP Server
14/08/28 09:38:27 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/08/28 09:38:27 INFO server.AbstractConnector: Started
[email protected]:49382<mailto:[email protected]:49382>
My question is:
What is the right way to execute spark script?
I really want to use spark-shell as a way to run cron job in the future, just
like when I run R script.
Best regards,
Henry
________________________________
The privileged confidential information contained in this email is intended for
use only by the addressees as indicated by the original sender of this email.
If you are not the addressee indicated in this email or are not responsible for
delivery of the email to such a person, please kindly reply to the sender
indicating this fact and delete all copies of it from your computer and network
server immediately. Your cooperation is highly appreciated. It is advised that
any unauthorized use of confidential information of Winbond is strictly
prohibited; and any information in this email irrelevant to the official
business of Winbond shall be deemed as neither given nor endorsed by Winbond.
________________________________
The privileged confidential information contained in this email is intended for
use only by the addressees as indicated by the original sender of this email.
If you are not the addressee indicated in this email or are not responsible for
delivery of the email to such a person, please kindly reply to the sender
indicating this fact and delete all copies of it from your computer and network
server immediately. Your cooperation is highly appreciated. It is advised that
any unauthorized use of confidential information of Winbond is strictly
prohibited; and any information in this email irrelevant to the official
business of Winbond shall be deemed as neither given nor endorsed by Winbond.
