Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Thanks for discovering this problem, Qingsheng! I'm also +1 for reverting the breaking changes. IIUC, currently, the behavior of "numXXXOut" metrics of the new and old sink is inconsistent. We have to break one of them to have consistent behavior. Sink V2 is an evolving API which is just introduc

RE: Re:Question about Flink Broadcast State event ordering

Hi Qing again, Another point to consider: broadcast streams are subject to watermarking. i.e. * You can wait to process the broadcast records only after the watermark passed, then * order those records by time * keep all broadcast records where the watermark not yet passed in some e

RE: Re:Question about Flink Broadcast State event ordering

Thanks both for your advice, I will give them a try! From: Schwalbe Matthias Sent: 10 October 2022 08:35 To: 仙路尽头谁为峰 ; Qing Lim Cc: User Subject: RE: Re:Question about Flink Broadcast State event ordering Hi Qing again, Another point to consider: broadcast streams are subject to watermarking.

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Thanks Qingsheng for starting this thread. +1 on reverting sink metric name and releasing 1.15.3 to fix this inconsistent behavior. Best, Leonard > 2022年10月10日 下午3:06，Jark Wu 写道： > > Thanks for discovering this problem, Qingsheng! > > I'm also +1 for reverting the breaking changes. >

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Hi Qingsheng, Do you have any idea what has happened in the process here? Do we know why they were changed? I was under the impression that these metric names were newly introduced due to the new interfaces and because it still depends on each connector implementing these. Sidenote: metric names

Re: videos Flink Forward San Francisco 2022

Hi Günter, I've understood that only the keynotes were recorded and not the other sessions. Best regards, Martijn On Sun, Oct 9, 2022 at 4:10 PM guenterh.lists wrote: > Sorry if this question was already posted > > By now only a few videos of the conference were published (mainly the > keynot

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

+1 for reverting these changes in Flink 1.16. For 1.15.3, can we make these metrics available via both names (numXXXOut and numXXXSend)? In this way we don't break it for those who already migrated to 1.15 and numXXXSend. That means we still need to change SinkWriterOperator to use another metric

Re: videos Flink Forward San Francisco 2022

Hey, that's sad. Is it possible for future Flink Forwards to record again and publish all sessions? Best regards Martin Am 10. Oktober 2022 11:26:26 MESZ schrieb Martijn Visser : >Hi Günter, > >I've understood that only the keynotes were recorded and not the other >sessions. > >Best regards,

Re: videos Flink Forward San Francisco 2022

I think everyone would be happier with the videos published on Youtube but it's unfortunately at the discretion of the organizer. At this time they decided against it for some reason. Gyula On Mon, Oct 10, 2022 at 11:57 AM Martin wrote: > Hey, > > that's sad. Is it possible for future Flink Fo

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Thanks everyone for joining the discussion! > Do you have any idea what has happened in the process here? The discussion in this PR [1] shows some details and could be helpful to understand the original motivation of the renaming. We do have a test case for guarding metrics but unfortunaly the

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

On 10/10/2022 11:24, Martijn Visser wrote: Sidenote: metric names are not mentioned in the FLIP process as a public API. Might make sense to have a separate follow-up to add that to the list (I do think we should list them there). That's a general issue we have. There's a lot of things we _ us

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

> I’m with Xintong’s idea to treat numXXXSend as an alias of numXXXOut But that's not possible. If it were that simple there would have never been a need to introduce another metric in the first place. It's a rather fundamental issue with how the new sinks work, in that they emit data to the

Re: videos Flink Forward San Francisco 2022

really very sad - as far as I know this happens for the first time, attitude of new Ververica? Hopefully immerok may resume the open mentality of data artisans. Günter On 10.10.22 11:26, Martijn Visser wrote: Hi Günter, I've understood that only the keynotes were recorded and not the other

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

+1 for reverting these changes in Flink 1.16, so I will cancel 1.16.0-rc1. +1 for `numXXXSend` as the alias of `numXXXOut` in 1.15.3. Best, Xingbo Chesnay Schepler 于2022年10月10日周一 19:13写道： > > I’m with Xintong’s idea to treat numXXXSend as an alias of numXXXOut > > But that's not possible. If it

Re: videos Flink Forward San Francisco 2022

+1 Am 10. Oktober 2022 14:39:28 MESZ schrieb "guenterh.lists" : >really very sad - as far as I know this happens for the first time, attitude >of new Ververica? > >Hopefully immerok may resume the open mentality of data artisans. > >Günter > >On 10.10.22 11:26, Martijn Visser wrote: >> Hi Günter

Re: Window state size with global window and custom trigger

Thanks for the confirmation :) Regards, Alexis. On Sun, 9 Oct 2022, 10:37 Hangxiang Yu, wrote: > Hi, Alexis. > I think you are right. It also applies for a global window with a custom > trigger. > If you apply a ReduceFunction or AggregateFunction, the window state size > usually is smaller tha

Deserialize avro message without reader schema.

Hi, I'm trying to deserialize avro messages from the kafka topic as a consumer. As a kafka consumer, I do not want to hold the reader schema on my end. Is there a way to deserialize avro messages to GenericRecord without providing the reader schema ? Regards, Sucheth Shivakumar website : https:

Re: Deserialize avro message without reader schema.

Hi,What you mean by holding the schema? You don't have to hold it as schema registry does it for you.20:00, 10 октября 2022 г., Sucheth S :Hi,I'm trying to deserialize avro messages from the kafka topic as a consumer. As a kafka consumer, I do not want to hold the reader schema on my end.Is there a

Re: Deserialize avro message without reader schema.

Hi Sucheth, The short answer is no, when deserializing Avro messages you have to provide the schema somehow, either directly or using the Schema Registry. On Mon, Oct 10, 2022 at 10:00 AM Sucheth S wrote: > Hi, > > I'm trying to deserialize avro messages from the kafka topic as a > consumer. >

Preserve rowtime through join

Hey Flinksters, I was wondering if you had any ideas for how to preserve the rowtime across an INNER equi join so that the output can be used in a temporal join. I've attached an example based on the TemporalJoinTest where I'm creating two views by deduplicating underlying streams (to rates_pk an

Data type mapping

I am writing Flink applications in Java and I need to do data type conversions between SQL/Table `DataType` and `TypeInformation`. According to Flink's documentation, type mapping methods in TypeConversions

Re: [DISCUSS] Reverting sink metric name changes made in 1.15

Thanks for the details Chesnay! By “alias” I mean to respect the original definition made in FLIP-33 for numRecordsOut, which is the number of records written to the external system, and keep numRecordsSend as the same value as numRecordsOut for compatibility. I think keeping numRecordsOut for

Question about SQL gateway

Hi, I submit a stream job from the SQL gateway. The stream job keeps outputting results to the SQL gateway. If the SQL gateway restarts or crashes, the stream job will continue running. After the SQL gateway restarts, how to get the results of the steam job? Thanks. Jack