Hi Tim,
First of all, let m wish you a happy and fulfilling New Year.
Sorry for the delay in my response. I was out for the xmas break.
I've added my thoughts to the ticket from the perspective of a streaming
Job.
@TD: What do you think?
-kr, Gerard.
On Tue, Dec 23, 2014 at 8:02 PM, Timothy Che
Hi Gerard,
SPARK-4286 is the ticket I am working on, which besides supporting shuffle
service it also supports the executor scaling callbacks (kill/request
total) for coarse grain mode.
I created SPARK-4940 to discuss more about the distribution problem, and
let's bring our discussions there.
Ti
Hi Tim,
That would be awesome. We have seen some really disparate Mesos allocations
for our Spark Streaming jobs. (like (7,4,1) over 3 executors for 4 kafka
consumer instead of the ideal (3,3,3,3))
For network dependent consumers, achieving an even deployment would
provide a reliable and reproduc
Hi Gerard,
Really nice guide!
I'm particularly interested in the Mesos scheduling side to more evenly
distribute cores across cluster.
I wonder if you are using coarse grain mode or fine grain mode?
I'm making changes to the spark mesos scheduler and I think we can propose a
best way to achi
