do you see in master ui that workers connected to master & before you are running your app there are 2 available cores in master ui per each worker? I understand that there are 2 cores on each worker - the question is do they got registered under master
regarding port it's very strange, please post what is problem connecting to 7077 use *--total-executor-cores 4 in your submit* if you can post master ui screen after you submitted your app On 8 January 2016 at 00:02, Michael Pisula <michael.pis...@tngtech.com> wrote: > I had tried several parameters, including --total-executor-cores, no > effect. > As for the port, I tried 7077, but if I remember correctly I got some kind > of error that suggested to try 6066, with which it worked just fine (apart > from this issue here). > > Each worker has two cores. I also tried increasing cores, again no effect. > I was able to increase the number of cores the job was using on one worker, > but it would not use any other worker (and it would not start if the number > of cores the job wanted was higher than the number available on one worker). > > > On 07.01.2016 22:51, Igor Berman wrote: > > read about *--total-executor-cores* > not sure why you specify port 6066 in master...usually it's 7077 > verify in master ui(usually port 8080) how many cores are there(depends on > other configs, but usually workers connect to master with all their cores) > > On 7 January 2016 at 23:46, Michael Pisula <michael.pis...@tngtech.com> > wrote: > >> Hi, >> >> I start the cluster using the spark-ec2 scripts, so the cluster is in >> stand-alone mode. >> Here is how I submit my job: >> spark/bin/spark-submit --class demo.spark.StaticDataAnalysis --master >> spark://<host>:6066 --deploy-mode cluster demo/Demo-1.0-SNAPSHOT-all.jar >> >> Cheers, >> Michael >> >> >> On 07.01.2016 22:41, Igor Berman wrote: >> >> share how you submit your job >> what cluster(yarn, standalone) >> >> On 7 January 2016 at 23:24, Michael Pisula < <michael.pis...@tngtech.com> >> michael.pis...@tngtech.com> wrote: >> >>> Hi there, >>> >>> I ran a simple Batch Application on a Spark Cluster on EC2. Despite >>> having 3 >>> Worker Nodes, I could not get the application processed on more than one >>> node, regardless if I submitted the Application in Cluster or Client >>> mode. >>> I also tried manually increasing the number of partitions in the code, no >>> effect. I also pass the master into the application. >>> I verified on the nodes themselves that only one node was active while >>> the >>> job was running. >>> I pass enough data to make the job take 6 minutes to process. >>> The job is simple enough, reading data from two S3 files, joining >>> records on >>> a shared field, filtering out some records and writing the result back to >>> S3. >>> >>> Tried all kinds of stuff, but could not make it work. I did find similar >>> questions, but had already tried the solutions that worked in those >>> cases. >>> Would be really happy about any pointers. >>> >>> Cheers, >>> Michael >>> >>> >>> >>> -- >>> View this message in context: >>> <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-job-uses-only-one-Worker-tp25909.html> >>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-job-uses-only-one-Worker-tp25909.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: <user-unsubscr...@spark.apache.org> >>> user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: <user-h...@spark.apache.org> >>> user-h...@spark.apache.org >>> >>> >> >> -- >> Michael Pisula * michael.pis...@tngtech.com * +49-174-3180084 >> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring >> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke >> Sitz: Unterföhring * Amtsgericht München * HRB 135082 >> >> > > -- > Michael Pisula * michael.pis...@tngtech.com * +49-174-3180084 > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > >
