Re: Dynamic Resource allocation in Spark Streaming

2017-12-03 Thread Sourav Mazumder
can confirm > spark.dynamicAllocation.enabled is enough. > > Best Regards > Richard > > From: Sourav Mazumder > Date: Sunday, December 3, 2017 at 12:31 PM > To: user > Subject: Dynamic Resource allocation in Spark Streaming > > Hi, > > I see the following jira is reso

Re: Dynamic Resource allocation in Spark Streaming

2017-12-03 Thread Qiao, Richard
Sourav: I’m using spark streaming 2.1.0 and can confirm spark.dynamicAllocation.enabled is enough. Best Regards Richard From: Sourav Mazumder Date: Sunday, December 3, 2017 at 12:31 PM To: user Subject: Dynamic Resource allocation in Spark Streaming Hi, I see the following

Dynamic Resource allocation in Spark Streaming

2017-12-03 Thread Sourav Mazumder
Hi, I see the following jira is resolved in Spark 2.0 https://issues.apache.org/jira/browse/SPARK-12133 which is supposed to support Dynamic Resource Allocation in Spark Streaming. I also see the JiRA https://issues.apache.org/jira/browse/SPARK-22008 which is about fixing numer of executor