Hi Nauroz,
If you’re using Flink 1.5 on Yarn, it supports dynamic task manager allocation
by default [1]. After skimming the code, it seems to me that in general if
requested parallelism is larger than available task slots, new task managers
will be requested via ResourceManager (please correct
Hi,
Yet Flink does not support auto-scaling. However, there is an umbrella JIRA
issue[1]
to cover the discussion about it. And I think the design doc(draft)
attached[2] could help.
Best,
tison.
[1] https://issues.apache.org/jira/browse/FLINK-10407
[2]
https://docs.google.com/document/d/1XKDXnrp8
Hi,
Flink does not support auto-scaling, yet. Rescaling operations currently
are always manual, i.e take a savepoint of the Flink job, and when
restoring from the savepoint, define a new parallelism for the job.
As for the metrics to be used for auto-scaling, I can imagine that it would
be possibl
Dear Flink Contributors and users,
I am a PhD student and I was interested to know, using which matrices, and when
does Flink performs scaling-in or scaling out of resources? I did search the
flink's website where I could only find information about how dynamic scaling
is performed in stateless
Hi,
this JIRA is a good starting point:
https://issues.apache.org/jira/browse/FLINK-3755
If you don't care about processing guarantees and you are using a stateless
streaming job, you can implement a simple Kafka consumer that uses Kafka's
consumer group mechanism. I recently implemented such a Ka
Thanks Stephan. I understand guaranteeing exactly once semantics with the
dynamic scaling is tough. If I were to let go of the exactly once
requirement, is it not possible in current version? It would be really
great if you can point me to the JIRA tracking this work.
On Mon, Aug 29, 2016 at 2:30
Hi!
There is a lot of work in progress on that feature, and it looks like you
can expect the next version to have some upscale/downscale feature that
maintains exactly-once semantics.
Stephan
On Mon, Aug 29, 2016 at 9:00 AM, Abhishek Agarwal
wrote:
> Is it possible to upscale or downscale a f
Is it possible to upscale or downscale a flink application without
re-deploying (similar to rebalancing in storm)?
--
Regards,
Abhishek Agarwal
