Dear Stefan, 

Thanks for the clarification. 

How is however the state recovered in the case of a task failure? Assuming 
there is a topology of 10 workers and a worker dies. The state in this case, 
after restarting the entire execution, will how exactly be distributed across 
the workers? 

Dominik

> On 8 Dec 2016, at 10:04, Stefan Richter <[email protected]> wrote:
> 
> Hi Dominik,
> 
> as Gordon’s response only covers keyed-state, I will briefly explain what 
> happens for non-keyed operator state. In contrast to Flink 1.1, Flink 1.2 
> checkpointing does not write a single blackbox object (e.g. ONE object that 
> is a set of all kafka offsets is emitted), but a list of blackbox objects 
> instead (e.g. think of all kafka offsets being emitted individually, as 
> MULTIPLE objects). While Flink 1.2 still has no knowledge about the emitted 
> objects in the list (thus they remain blackboxes), what the contract allows 
> is that those objects can be freely redistributed in case of scale-out or 
> scale-in. Scaling is merely splitting or merging of the checkpointed lists.
> 
> Best,
> Stefan 
> 
>> Am 08.12.2016 um 08:00 schrieb Tzu-Li (Gordon) Tai <[email protected] 
>> <mailto:[email protected]>>:
>> 
>> Hi Dominik,
>> 
>> Do you mean how Flink redistributes an operator’s state when the parallelism 
>> of the operator is changed?
>> If so, you can take a look at [1] and [2].
>> 
>> Cheers,
>> Gordon
>> 
>> [1] https://issues.apache.org/jira/browse/FLINK-3755 
>> <https://issues.apache.org/jira/browse/FLINK-3755>
>> [2] 
>> https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9WyssQ63w/edit#
>>  
>> <https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9WyssQ63w/edit#>
>> 
>> On December 8, 2016 at 4:40:18 AM, Dominik Safaric ([email protected] 
>> <mailto:[email protected]>) wrote:
>> 
>>> Hi everyone, 
>>> 
>>> In the case of scaling out a Flink cluster, how does Flink handle operator 
>>> state partitioning of a staged topology?  
>>> 
>>> Regards, 
>>> Dominik 
> 

Reply via email to