share how you submit your job what cluster(yarn, standalone) On 7 January 2016 at 23:24, Michael Pisula <michael.pis...@tngtech.com> wrote:
> Hi there, > > I ran a simple Batch Application on a Spark Cluster on EC2. Despite having > 3 > Worker Nodes, I could not get the application processed on more than one > node, regardless if I submitted the Application in Cluster or Client mode. > I also tried manually increasing the number of partitions in the code, no > effect. I also pass the master into the application. > I verified on the nodes themselves that only one node was active while the > job was running. > I pass enough data to make the job take 6 minutes to process. > The job is simple enough, reading data from two S3 files, joining records > on > a shared field, filtering out some records and writing the result back to > S3. > > Tried all kinds of stuff, but could not make it work. I did find similar > questions, but had already tried the solutions that worked in those cases. > Would be really happy about any pointers. > > Cheers, > Michael > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-job-uses-only-one-Worker-tp25909.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
