Any update on this guys ? On Wed, Dec 28, 2016 at 10:19 AM, Nishant Kumar <nishant.ku...@applift.com> wrote:
> I have updated my question: > > http://stackoverflow.com/questions/41345552/spark- > streaming-with-yarn-executors-not-fully-utilized > > On Wed, Dec 28, 2016 at 9:49 AM, Nishant Kumar <nishant.ku...@applift.com> > wrote: > >> Hi, >> >> I am running spark streaming with Yarn with - >> >> *spark-submit --master yarn --deploy-mode cluster --num-executors 2 >> --executor-memory 8g --driver-memory 2g --executor-cores 8 ..* >> >> I am consuming Kafka through DireactStream approach (No receiver). I >> have 2 topics (each with 3 partitions). >> >> I reparation RDD (i have one DStream) into 16 parts (assuming no of >> executor * num of cores = 2 * 8 = 16 *Is it correct ?*) and then i do >> foreachPartition and writes each partition to local file and then send it >> to other server (not spark) through http (Using apache http client post >> with multi-part). >> >> *When i checked details of this step (or JOB is it correct naming?) >> through Spark UI, it showed that total 16 task executed on single executor >> with 8 task at a time.* >> >> This is Spark UI details - >> >> *Details for Stage 717 (Attempt 0)* >> >> Index ID Attempt Status Locality Level Executor ID / Host Launch Time >> Duration GC Time Shuffle Read Size / Records Errors >> 0 5080 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:46 2 s 11 ms 313.3 KB / 6137 >> 1 5081 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:46 2 s 11 ms 328.5 KB / 6452 >> 2 5082 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:46 2 s 11 ms 324.3 KB / 6364 >> 3 5083 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:46 2 s 11 ms 321.5 KB / 6306 >> 4 5084 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:46 2 s 11 ms 324.8 KB / 6364 >> 5 5085 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:46 2 s 11 ms 320.8 KB / 6307 >> 6 5086 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:46 2 s 11 ms 323.4 KB / 6356 >> 7 5087 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:46 3 s 11 ms 316.8 KB / 6207 >> 8 5088 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:48 2 s 317.7 KB / 6245 >> 9 5089 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:48 2 s 320.4 KB / 6280 >> 10 5090 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:48 2 s 323.0 KB / 6334 >> 11 5091 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:48 2 s 323.7 KB / 6371 >> 12 5092 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:48 2 s 316.7 KB / 6218 >> 13 5093 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:48 2 s 321.0 KB / 6301 >> 14 5094 0 SUCCESS NODE_LOCAL 1 / executor1_machine_host_name 2016/12/27 >> 12:11:48 2 s 321.4 KB / 6304 >> >> I was expecting it to execute 16 parallel task (2 executor * 8 core) on >> either one or more executor. I think i am missing something. Please help. >> >> >> -- >> *Nishant Kumar* >> Senior Software Engineer >> >> Phone: +91 80088 42030 >> Skype: nishant.kumar_applift >> >> >> *AppLift India* >> 107/3, 80 Feet Main Road, >> Koramangala 4th Block, >> Bangalore - 560034 >> www.AppLift.com <http://www.applift.com/> >> > > > > -- > *Nishant Kumar* > Senior Software Engineer > > Phone: +91 80088 42030 > Skype: nishant.kumar_applift > > > *AppLift India* > 107/3, 80 Feet Main Road, > Koramangala 4th Block, > Bangalore - 560034 > www.AppLift.com <http://www.applift.com/> > -- *Nishant Kumar* Senior Software Engineer Phone: +91 80088 42030 Skype: nishant.kumar_applift *AppLift India* 107/3, 80 Feet Main Road, Koramangala 4th Block, Bangalore - 560034 www.AppLift.com <http://www.applift.com/> -- Meet us at: @ *PG Connects*, London, Jan 16-17 @ *Mobile Games Forum*, London, Jan 17-18 @ *GMASA*, Jakarta, Jan 26 @* Casual Connect*, Berlin, Feb 7-9 @* Mobile World Congress*, Barcelona, Feb 27-Mar 2 @* GDC*, San Francisco, Feb 27-Mar 3 Click here <http://www.applift.com> to see all the events we will be attending.
