It's has been a while since last update on the thread. Now Spark 2.0 is
available, do you guys know if there's any progress on Dynamic Allocation &
Spark Streaming?
On Mon, Oct 19, 2015 at 1:13 PM, robert towne
wrote:
> I have watched a few videos from Databricks/Andrew Or around the Spark 1.2
>
://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.scheduler.BatchInfo
From: Kyle Lin
Date: Friday, November 6, 2015 at 11:48 AM
To: Tathagata Das
Cc: robert towne, user
Subject: Re: Dynamic Allocation & Spark Streaming
Hey there
I run Spark streaming 1.5.1 on YARN
Hey there
I run Spark streaming 1.5.1 on YARN with Dynamic allocation, and use direct
stream API to read data from Kafka.
Spark job can dynamically request a executor when
reaching spark.dynamicAllocation.schedulerBacklogTimeout.
However, it won't dynamically remove executor when there is no mor
Unfortunately the title on the JIRA is extremely confusing. I have fixed it.
The reason why dynamic allocation does not work well with streaming is that
the heuristic that is used to automatically scale up or down the number of
executors works for the pattern of task schedules in batch jobs, not f
