Hi,
Thanks for the explanation, I had not heard about the Raw State before, and it sounds like that is not surprising if it is rarely used, and doesn't seem to be included in the latest versions of the documentation. Is there perhaps some older version of the documentation you could point me to where I can see how Raw state can be used in Flink? Best Regards, Christopher Gustafson ________________________________ Från: Seth Wiesman <[email protected]> Skickat: den 11 mars 2022 17:57:21 Till: Christopher Gustafson Kopia: [email protected] Ämne: Re: Checkpointing in StateFun I assume you are talking about the checkpointing in the feedback package? StateFun only relies on Flink checkpointing for fault tolerance. All state is stored in standard checkpoint / savepoints and can be used to restore from failure, upgrade a job, rescale, etc. Just like any other snapshot. StateFun internally relies on a form of streaming iteration. The feedback package contains the code for that iteration which sends records back upstream. To ensure exactly-once semantics, in-flight records on the feedback edge need to be stored in the snapshot. It does so through Flinks raw keyed state abstraction. This is the same thing state backends use under the hood to snapshot out your keyed state in any rich function. Raw keyed state has been a stable part of Flinks API forever, but is rarely seen in user code. I often describe the statefun runtime to people as the world's most complex datastream application; you will find advanced usage patterns most people have not seen, but nothing that isn't well supported by Flink. Seth On Fri, Mar 11, 2022 at 2:24 AM Christopher Gustafson <[email protected]<mailto:[email protected]>> wrote: Hi, I am doing some experiments with StateFun and different Flink state backends. I was looking through the StateFun source code and saw that it is doing some own logic when it comes to checkpointing. My initial understanding was that checkpointing was implicitly leveraged with Flink. I did not find a lot of documentation about this, so I am asking if you could provide a description of how checkpointing works in StateFun, and how it is different from regular Flink checkpointing? Thanks in advance, Christopher Gustafson
