Hi All
I have one master and one worker on AWS (amazon web service) and am running
spark map reduce code provided on the link
https://spark.apache.org/examples.html
We are using Spark version 1.0.2
Word Count
val file = spark.textFile("hdfs://...")
val counts = file.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")
To get this working I have compiled a jar file.
spark-submit --master spark://ip-172-31-24-183.ec2.internal:7077 --class Sample
/home/ec2-user/scalatest/target/scala-2.11/test_big_2.11-0.1.jar
when I run the spark job it gives me error: Connection refused -
1. There is not result shown on the master
2. Error on worker is as below
14/10/07 10:16:16 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkWorker@ip-172-31-27-51.ec2.internal:39142] ->
[akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]: Error
[Association failed with
[akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkExecutor@ip-172-31-27-51.ec2.internal:38752]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: ip-172-31-27-51.ec2.internal/172.31.27.51:38752
]
Could some one help clarify if this is a
* spark issue
* issue with the jar file
* or if slave cannot connect to master or itself or some port ?
Please help
Thanks
Jacob
