Re:Re: [DISCUSS] The abstraction of cache lookupFunction and cache metric

che in JDBC and Hive >lookup table, the value type is List, since it’s common that a >joining key could mapped to multiple rows. We could add another layer of >abstraction under Cache interface, for example: > >OneToManyCache extends Cache> > >And add interfaces like `a

Re:Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

Hi Qingsheng and devs, Thanks for your heated discussion and redesign to optmize this feature. I just have two comments: 1. How about abtract the LookupCache to a higher level with a common Cache? It will be convenient for devs to use in other place. 2. Does it have any metrics, such as

Re:Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

the existing metric definitions are also >applicable for all-cache case. > >Cheers, > >Qingsheng > > >> On May 15, 2022, at 21:17, zst...@163.com wrote: >> >> Hi Qingsheng and devs, >> >> Thanks for your heated discussion and redesign to optmize th

Re: [DISCUSS] Introduce Hash Lookup Join

Hi Jing, Thanks very much for your FLIP. I have some points: - How shall we deal with CDC data? If there is CDC data in the pipeline, IMHO, shuffle by join key will cause CDC data disorder. Will it be better to use primary key in this case? - If the shuffle keys can be customized when user

Re:Re: [DISCUSS] Introduce Hash Lookup Join

Hi Jing, Thanks for your detail reply. 1) In the last suggestion, hash by primary key is not use for raising the cache hit, but handling with skew of left source. Now that you have 'skew' hint and other discussion about it, I'm looking forward to it. 2) I mean to support user defined partitio

Re:Re: [VOTE] FLIP-204: Introduce Hash Lookup Join

+1 non-binging! Thanks Jing for driving this! Best, Yuan At 2022-01-24 15:32:18, "wenlong.lwl" wrote: >+1 non-binding! > >Thanks for driving this, Jing! > >Best, >Wenlong > >On Mon, 24 Jan 2022 at 16:02, Jingsong Li wrote: > >> +1 (binding) >> >> Thanks Jing! >> >> Best, >> Jingsong >> >> On

[DISCUSS] The abstraction of cache lookupFunction and cache metric

Hi devs, I would like to propose a discussion thread about abstraction of Cache LookupFunction with metrics for cache in connectors to make cache out of box for connector developers. There are multiple LookupFunction implementations in individual connectors [1][2][3][4] so far. At the same ti

Re:Re: [DISCUSS] The abstraction of cache lookupFunction and cache metric

>[1] >https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/groups/SourceReaderMetricGroup.java >[2] >https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/InternalS

Re:Re: [DISCUSS] The abstraction of cache lookupFunction and cache metric

ecordCacheMiss(); >>> ... >>> } >>> >>> You can check SourceReaderMetricGroup[1] and its implementation[2] as a reference. >>> >>> Hope these would be helpful! >>> >>> Best regards, >>> >>> Qingsheng Ren >>>

Re:Re: [DISCUSS] The abstraction of cache lookupFunction and cache metric

class, but it would be great to aggregate these constants into a unified >place. > >Cheers, > >Qingsheng > > >> On Mar 8, 2022, at 10:22, zst...@163.com wrote: >> >> Hi Qingsheng Ren, >> Thanks for your feedback. >> >> >>> 1. It looks l

Re:Re: [VOTE] FLIP-221: Abstraction for lookup source and metric

Thanks Qingsheng for driving this. +1 (non-binding) Best regards, Yuan At 2022-06-23 15:44:55, "Martijn Visser" wrote: >Great work on the FLIP. > >+1 (binding) > >Op do 23 jun. 2022 om 08:07 schreef Leonard Xu : > >> Thanks Qingsheng for driving this work. >> >> +1(binding) >> >> >> B

Re:Re: Re: [ANNOUNCE] New Apache Flink Committers: Qingsheng Ren, Shengkai Fang

Congratulations, Shengkai and Qingsheng! Best Regards, Yuan At 2022-06-24 16:45:25,"godfrey he" , said: >Congrats, Shengkai and Qingsheng! > >Best, >Godfrey > >Yu Li 于2022年6月22日周三 23:55写道： >> >> Congrats and welcome, Qingsheng and Shengkai! >> >> Best Regards, >> Yu >> >> >> On Wed, 22 Jun 20