Hi,
I think we have to rethink a bit how the state interfaces work. Having an
interface where you snapshot a type T and restore a type T are not well
suited for dealing with job updates/state updates. For example, let's look
at the current Checkpointed interface. The value that you return is
serial
Hi,
Yes I think it makes sense. :)
Gyula
On Fri, Aug 12, 2016, 17:02 Ufuk Celebi wrote:
> I will update the design doc with more details for the Checkpointed
> variants and remove Option 2 (I think that's an orthogonal thing).
>
> The way I see it now, we should have base CheckpointedBase inter
I will update the design doc with more details for the Checkpointed
variants and remove Option 2 (I think that's an orthogonal thing).
The way I see it now, we should have base CheckpointedBase interface,
have the current Checkpointed interface be a subclass for not
repartitionable state. Then we
Hi Aljoscha,
Yes this is pretty much how I think about it as well.
Basically the state in this case would be computed from the side inputs
with the same state update logic on all operators. I think it is imprtant
that operators compute their own state or at least observe all state
changes otherwi
Hi Gyula,
I was thinking about this as well, in the context of side-inputs, which
would be a generalization of your use case. If I'm not mistaken. In my head
I was calling it global state. Essentially, this state would be the same on
all operators and when checkpointing you would only have to check
Hi,
Let me try to explain what I mean by broadcast states.
I think it is a very common pattern that people broadcast control messages
to operators that also receive normal input events.
some examples: broadcast a model for prediction, broadcast some information
that should be the same at all subt
Comments inline.
On Thu, Aug 11, 2016 at 8:06 PM, Gyula Fóra wrote:
> Option 1:
> I think the main problem here is sending all the state everywhere will not
> scale at all. I think this will even fail for some internal Flink operators
> (window timers I think are kept like this, maybe Im wrong he
Hi Ufuk,
Thanks for the great proposal I think this will be a very cool feature :)
Option 1:
I think the main problem here is sending all the state everywhere will not
scale at all. I think this will even fail for some internal Flink operators
(window timers I think are kept like this, maybe Im w
Hey all!
I've created a short FLIP for rescalable non-partitioned state:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-8%3A+Rescalable+Non-Partitioned+State
This is related to an effort led by Till, Aljoscha, Stephan, and
Stefan to allow rescaling of Flink jobs (see FLINK-3755, FLINK-43
