Re: Alternatives of KafkaDeserializationSchema.isEndOfStream()

Hi Ayush, Your use-case should be supported. Sorry, we don't have a good way to support this in Flink 1.14. I am going to propose a FLIP to fix it in Flink 1.15. Thanks, Dong On Thu, Dec 9, 2021 at 7:11 PM Ayush Chauhan wrote: > My usecase is that as soon as the avro message version is chan

Re: Alternatives of KafkaDeserializationSchema.isEndOfStream()

change? > > On Fri, Dec 17, 2021 at 3:03 AM Dong Lin wrote: > >> Hi Ayush, >> >> Your use-case should be supported. Sorry, we don't have a good way to >> support this in Flink 1.14. >> >> I am going to propose a FLIP to fix it in Flink 1.15. >>

Re: Alternatives of KafkaDeserializationSchema.isEndOfStream()

to the discussion of dynamic EOF? > I hope I gave some pointers. > > [1] > https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/enumerator/cursor/StopCursor.java#L41-L41 > > On Fri, Dec 17, 20

Re: Alternatives of KafkaDeserializationSchema.isEndOfStream()

ds in > OffsetsInitializer) for Table/SQL. We could even provide a predefined > strategy for schema changes when the schema registry is used. > > If you already have use-cases that relies on the deserialized data, then > let's move the stopping logic to RecordEmitter. At this p

Re: Alternatives of KafkaDeserializationSchema.isEndOfStream()

(SourceCollector) if they want to do dynamic EOF. To make our discussion more efficient and possibly involve more people for comments, I will create a FLIP and open a discussion thread for the FLIP. Thanks, Dong On Tue, Dec 28, 2021 at 7:40 PM Dong Lin wrote: > Hi Arvid, > > Thanks a lo

Re: Alternatives of KafkaDeserializationSchema.isEndOfStream()

sed+on+de-serialized+records#FLIP208:UpdateKafkaSourcetodetectEOFbasedondeserializedrecords-RejectedAlternatives>) and have opened the discussion thread. Maybe we can continue the discussion there. Cheers, Dong On Wed, Dec 29, 2021 at 12:38 PM Dong Lin wrote: > Hi Arvid, > > After dis

Re: Examples / Documentation for Flink ML 2

Hi Bonino, Thanks for your interest! Flink ML is currently ready for experienced algorithm developers to try it out because we have setup the basic APIs and infrastructure to develop algorithms. Five algorithms (i.e. kmeans, naive bays, knn, logistic regression and one-hot encoder) has been imple

Re: Examples / Documentation for Flink ML 2

If you think it's worth, we may contribute some > documentation, e.g., tutorial based on what we learn while setting up our > test project with Flink ML. Is it something that might be of interest for > you? > > Best regards > > Dario > On 1/18/22 04:51, Dong Lin wrote: &g

Re: Examples / Documentation for Flink ML 2

mewhat > "separate" from Flink and the same I think holds for the documentation. How > do you think it is better to proceed? > > Best regards > > Dario Bonino > On 1/19/22 09:36, Dong Lin wrote: > > Hi Bonino, > > Definitely, it will be great to build up the

Re: [DISCUSS] Planning Flink 1.17

Thanks for kicking off the release plan. +1 for the proposed timeline. Best, Dong On Thu, Oct 20, 2022 at 3:46 PM Qingsheng Ren wrote: > Hi everyone, > > As we are approaching the official release of Flink 1.16, it’s a good time > to kick off some discussions and march toward 1.17. > > - Rele

FeatHub - A feature store that uses Apache Flink for real-time feature ETL

Hi everyone, In case some of you might be interested, we (developers at Alibaba Flink team) recently developed and open-sourced FeatHub , a feature store that uses Apache Flink for feature engineering. The github page https://github.com/alibaba/feathub provides

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

Congratulations! Dong On Mon, Mar 27, 2023 at 5:24 PM Yu Li wrote: > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project called Apache > Paimon(incubating) [1] [2] [3]. The new project still

ETL real-time features using Flink with application-level metrics

Hi all, I am writing this email to promote our open-source feature store project ( FeatHub ) that supports using Flink (production-ready) and Spark (not production-ready) to compute real-time / offline features with pythonic declarative feature specifications.

FeatHub : a feature store for ETL real-time features using Flink

Dong Lin 于2023年8月14日 周一09:02写道： > Hi all, > > I am writing this email to promote our open-source feature store project ( > FeatHub <https://github.com/alibaba/feathub>) that supports using Flink > (production-ready) and Spark (not production-ready) to compute real-time / &g