Hi All, Just floating this email again. Grateful for any suggestions.
Akshay Bhardwaj +91-97111-33849 On Mon, May 20, 2019 at 12:25 AM Akshay Bhardwaj < akshay.bhardwaj1...@gmail.com> wrote: > Hi All, > > I am running Spark 2.3 on YARN using HDP 2.6 > > I am running spark job using dynamic resource allocation on YARN with > minimum 2 executors and maximum 6. My job read data from parquet files > which are present on S3 buckets and store some enriched data to cassandra. > > My question is, how does YARN decide which nodes to launch containers? > I have around 12 YARN nodes running in the cluster, but still i see > repeated patterns of 3-4 containers launched on the same node for a > particular job. > > What is the best way to start debugging this reason? > > Akshay Bhardwaj > +91-97111-33849 >
