hi, may be I am not quite clear in my head on this one. But how do we know that 1 yarn container = 1 executor?
Regards, Gourav Sengupta On Tue, Oct 9, 2018 at 8:53 PM Dillon Dukek <dillon.du...@placed.com.invalid> wrote: > Can you send how you are launching your streaming process? Also what > environment is this cluster running in (EMR, GCP, self managed, etc)? > > On Tue, Oct 9, 2018 at 10:21 AM kant kodali <kanth...@gmail.com> wrote: > >> Hi All, >> >> I am using Spark 2.3.1 and using YARN as a cluster manager. >> >> I currently got >> >> 1) 6 YARN containers(executors=6) with 4 executor cores for each >> container. >> 2) 6 Kafka partitions from one topic. >> 3) You can assume every other configuration is set to whatever the >> default values are. >> >> Spawned a Simple Streaming Query and I see all the tasks get scheduled on >> one YARN container. am I missing any config? >> >> Thanks! >> >
