Re: Clarification around versioning and flink's state serialization

Thanks for the clarification, that is clear now. Look forward to seeing your slides, safe travels. On Sat, Dec 22, 2018 at 8:25 AM Tzu-Li (Gordon) Tai wrote: > 1. Correct. Under the hood, evolvability of schema relies on the type's > serializer implementation to support it. > In Flink 1.7, this

Re: Clarification around versioning and flink's state serialization

1. Correct. Under the hood, evolvability of schema relies on the type's serializer implementation to support it. In Flink 1.7, this had been done only for Avro's Flink built-in serializer (i.e. the AvroSerializer class) for now, so you don't need to provide a custom serializer for this. For any oth

[ANNOUNCE] Berlin Buzzwords 2019 - Call for Presentations open until Febr. 17th

Hi everyone, The Call for Presentations for the Berlin Buzzwords 2019 conference is open until Febr. 17th. Berlin Buzzwords [1] is an amazing conference on all things Scale, Search, and Stream(!) with a great and open minded community and a strong focus on open source. Next year's edition takes p

HA with HDFS question

First off, I am new to using HDFS to store things, so expect stupid questions. I am working on hardening our Flink cluster for production usage. This includes setting up an HA flink cluster, saving checkpoint and savepoints to a central location etc. I have a functioning HDFS setup inside an HA Ku

Re: Serious stability issues when running on YARN (Flink 1.7.0)

After some investigation it seems like there was a Thread leak in the job as the operator implementation shut down the threads in the close method which was never called in the failure scenario. (had to change to dispose) Sorry for the false alarm. Gyula On Fri, Dec 21, 2018 at 8:58 AM Gyula Fór

Re: Clarification around versioning and flink's state serialization

Yes that helps a lot! Just to clarify: - If using Avro types in 1.7, no explicit declaration of serializers needs to be done to have state evolution. But all other evolvable types (e.g Protobuf) still need to be registered and evolved manually? - If specifying `disableGenericTypes` on my execution

Re: Clarification around versioning and flink's state serialization

For the documents I would recommend reading through: https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/schema_evolution.html On Fri, Dec 21, 2018, 9:55 PM Tzu-Li (Gordon) Tai Hi, > > Yes, if Flink does not recognize your registered state type, it will by > default use

Re: Clarification around versioning and flink's state serialization

Hi, Yes, if Flink does not recognize your registered state type, it will by default use Kryo for the serialization. And generally speaking, Kryo does not have good support for evolvable schemas compared to other serialization frameworks such as Avro or Protobuf. The reason why Flink defaults to K

Re: Data loss when restoring from savepoint

Stefan, I managed to analyze savepoint with bravo. It seems that the data that's missing from output *is* found in savepoint. I simplified my test case to the following: - job 1 has bee running for ~10 days - savepoint X created & job 1 cancelled - job 2 started with restore from savepoint X The

Clarification around versioning and flink's state serialization

Hi all, I am trying to understand the situation with state serialization in flink. I'm looking at a number of sources, but slide 35 from here crystalizes my confusion: https://www.slideshare.net/FlinkForward/flink-forward-berlin-2017-tzuli-gordon-tai-managing-state-in-apache-flink So, I understa

Re: Flink 1.5 + resource elasticity resulting in overloaded workers

@flink comitters I get that you don't want to be aware of task managers but would it make sense to change SlotManager (I briefly looked over the code and i think that's the code that is responsible for this) to it randomly selects slots ? or add an option to make it do this if this is not somethi

Re: Flink 1.7 doesn't work with Kafka Table Source Descriptors

Add Dev Group On Fri, Dec 21, 2018 at 6:21 PM dhanuka ranasinghe < dhanuka.priyan...@gmail.com> wrote: > Hi All, > > I have tried to read data from Kafka from Flink using Table API. It's > working fine with Flink 1.4 but when upgrade to 1.7 given me below error. I > have attached the libraries ad

RE: 回复：buffer pool is destroyed

Thank you. Forgot to mention that the error occurred when I was testing my code in Intellij. From: zhijiang [mailto:wangzhijiang...@aliyun.com] Sent: Friday, December 21, 2018 4:40 PM To: user@flink.apache.org; Chan, Shuang (PWTB 291) Subject: 回复：buffer pool is destroyed Hi Shuang, Normally th

回复：buffer pool is destroyed

Hi Shuang, Normally this exception you mentioned is not the root cause of failover, and it is mainly caused by cancel process to make task exit. You can further check whether there are other failures in job master log to find the root cause. Best, Zhijiang -