Re: Developing Beam applications using Flink checkpoints

2020-05-19 Thread Ivan San Jose
e Beam program to Flink. > > On Tue, May 19, 2020 at 11:38 AM Ivan San Jose < > isanj...@theworkshop.com> wrote: > > Actually I'm also thinking about how Beam coders are related with > > runner's serialization... I mean, on Beam you specify a coder per > >

Re: Developing Beam applications using Flink checkpoints

2020-05-19 Thread Ivan San Jose
result, so that is doing a double serialization, does it make sense? Or how does it work? On Tue, 2020-05-19 at 08:54 +0000, Ivan San Jose wrote: > Yep, sorry if I'm bothering you but I think I'm still not getting > this, > how could I tell Beam to tell Flink to use that serializ

Re: Developing Beam applications using Flink checkpoints

2020-05-19 Thread Ivan San Jose
k/blob/master/flink-core/src/main/java/org/apache/flink/api/common/typeutils/base/MapSerializerSnapshot.java > > On Tue, May 19, 2020 at 10:31 AM Ivan San Jose < > isanj...@theworkshop.com> wrote: > > Thanks for your complete answer Arvid, we will try to approach all > > thi

Re: Developing Beam applications using Flink checkpoints

2020-05-19 Thread Ivan San Jose
187 > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/custom_serialization.html#the-typeserializersnapshot-abstraction > [3] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html > > On Fri, May 1

Developing Beam applications using Flink checkpoints

2020-05-15 Thread Ivan San Jose
Hi, we are starting to use Beam with Flink as runner on our applications, and recently we would like to get advantages that Flink checkpoiting provides, but it seems we are not understanding it clearly. Simplifying, our application does the following: - Read meesages from a couple of Kafka topic