Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu

2023-04-24 Thread Jing Ge
Congrats! Leonard! Best regards, Jing On Mon, Apr 24, 2023 at 5:53 AM Matthias Pohl wrote: > Congrats, Leonard :) > > On Mon, Apr 24, 2023, 05:17 Yangze Guo wrote: > > > Congratulations, Leonard! > > > > Best, > > Yangze Guo > > > > On Mon, Apr 24, 2023 at 10:05 AM Shuo Cheng wrote: > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren

2023-04-24 Thread Jing Ge
Congrats! Qingsheng! Best regards, Jing On Mon, Apr 24, 2023 at 9:35 AM Zakelly Lan wrote: > Congratulations, Qingsheng! > > Best regards, > Zakelly > > On Mon, Apr 24, 2023 at 11:52 AM Matthias Pohl > wrote: > > > > Congratulations, Qingsheng! :) > > > > On Mon, Apr 24, 2023, 05:17 Yangze

Re: [DISCUSS FLINKSQL PARALLELISM]

2023-04-24 Thread Jing Ge
Hi Green, Since FLIP-292 opened the door to do fine-grained tuning at operator level for Flink SQL jobs, I would also suggest leveraging the compiled json to do further config optimization like Yun Tang already mentioned. We should consider making it(leveraging the compiled json plan) the stand

Re: Re: [DISCUSS] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

2023-04-24 Thread Jing Ge
ingMode parameter, this > provides different atomicity implementations in Batch and Stream modes. > Not only to determine if atomicity is supported, but also to help select > different TwoPhaseCatalogTable implementations to provide different levels > of atomicity! > > Looking forwa

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

2023-04-24 Thread Jing Ge
Thanks Chesnay for working on this. Would you like to share more info about the JDK bug? Best regards, Jing On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler wrote: > As it turns out Kryo isn't a blocker; we ran into a JDK bug. > > On 31/03/2023 08:57, Chesnay Schepler wrote: > > > https://gith

Re: [SUMMARY] Flink 1.18 Release Sync 04/18/2023

2023-04-24 Thread Jing Ge
Hi Qingsheng, Thanks for sharing the summary! Best regards, Jing On Mon, Apr 24, 2023 at 1:50 PM Qingsheng Ren wrote: > Hi devs, > > I'd like to share some highlights in the 1.18 release sync on 04/18/2023 > (Sorry for the late summary!): > > - Feature list: @contributors please add your featu

Re: [VOTE] FLIP-288: Enable Dynamic Partition Discovery by Default in Kafka Source

2023-04-24 Thread Jing Ge
+1(binding) Best regards, Jing On Tue, Apr 25, 2023 at 5:17 AM Rui Fan <1996fan...@gmail.com> wrote: > +1 (binding) > > Best, > Rui Fan > > On Tue, Apr 25, 2023 at 10:06 AM Biao Geng wrote: > > > +1 (non-binding) > > Best, > > Biao Geng > > > > Martijn Visser 于2023年4月24日周一 20:20写道: > > > > > +

Re: [DISCUSS] Planning Flink 2.0

2023-04-25 Thread Jing Ge
Thanks Xingtong and Jark for kicking off and driving the discussion! It is really good to see we finally start talking about Flink 2.0. There are so many great ideas that require breaking API changes and so many tech debts need to be cleaned up. With the Flink 2.0 ahead, we will be more fast-paced

Re: [DISCUSS] Preventing Mockito usage for the new code with Checkstyle

2023-04-25 Thread Jing Ge
This is a great idea, thanks for bringing this up. +1 Also +1 for Junit4. If I am not mistaken, it could only be done after the Junit5 migration is done. @Chesnay thanks for the hint. Do we have any doc about it? If not, it might deserve one. WDYT? Best regards, Jing On Wed, Apr 26, 2023 at 5:1

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-04-27 Thread Jing Ge
Best > Feng > > > > On Thu, Apr 13, 2023 at 8:52 AM Shammon FY wrote: > > > Hi Feng > > > > Thanks for your update. > > > > I found there are two options in `Proposed Changes`, can you put the > > unselected option in `Rejected Alternati

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

2023-04-27 Thread Jing Ge
? Best regards, Jing On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi wrote: > More details about the JDK bug here > https://bugs.openjdk.org/browse/JDK-8277529 > > Related Jira ticket > https://issues.apache.org/jira/browse/FLINK-24998 > > ------ > *From:* J

Re: [DISCUSS] Planning Flink 2.0

2023-04-28 Thread Jing Ge
Hi, As far as I am concerned, it would be great to build two top level categories for Flink 2.0 release. 1. Future - mainly new big features or architecture improvements to achieve visions like streamhouse. This makes the 2.0 release be the 2.0 instead of 1.x release, i.e. the real intention of 2

Re: Re: Re: [DISCUSS] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

2023-04-28 Thread Jing Ge
t; automatically renaming the table name for the user. > > > > -- > > Best regards, > > Mang Zhang > > > > At 2023-04-24 15:41:31, "Jing Ge" wrote: > >Hi Mang, > > > > > > > >Thanks for clarifying it. I am trying to understand

Re: [DISCUSS] EncodingFormat and DecondingFormat provide copy API

2023-04-28 Thread Jing Ge
Hi Tanjialiang, Like we discussed in another thread, please feel free to create a FLIP and start further discussion. In case you need any access right to the wiki page, please let me know, thanks. Best regards, Jing On Sun, Apr 23, 2023 at 4:23 AM tanjialiang wrote: > Hi, devs. > Do anyone ha

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-04-30 Thread Jing Ge
tore to achieve multi-instance > non-conflicting scenarios. What do you think? > > > > Best, > Feng > > On Thu, Apr 27, 2023 at 9:03 PM Jing Ge > wrote: > > > Hi Feng, > > > > Thanks for working on the FLIP. There are still some NIT issues in the > FLIP &g

Re: [DISCUSS] Preventing Mockito usage for the new code with Checkstyle

2023-04-30 Thread Jing Ge
been migrated to junit5. Best regards, Jing On Wed, Apr 26, 2023 at 9:02 AM Panagiotis Garefalakis wrote: > Thanks for bringing this up! +1 for the proposal > > @Jing Ge -- we don't necessarily need to completely migrate to Junit5 (even > though it would be ideal). > We

Re: [DISCUSS] Preventing Mockito usage for the new code with Checkstyle

2023-05-02 Thread Jing Ge
e effort we > could minimize the need for suppressions altogether. > > Cheers, > Panagiotis > > On Sun, Apr 30, 2023 at 6:16 AM Jing Ge > wrote: > > > Thanks @Panagiotis for the hint! Does it mean that those suppressions > need > > to be cleaned up continuously

[SUMMARY] Flink 1.18 Release Sync 05/02/2023

2023-05-02 Thread Jing Ge
Hi devs, I'd like to share highlights synced on 05/02/2023 - Feature list: many features have been added to the list[1]. Double checked new FLIPs that passed votes are already in the list. - CI instabilities: First of all, there are currently 4 Blocker issues. 3 of them have already been assigned

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-05-06 Thread Jing Ge
rve as the default CatalogStore > to save catalogs in memory and replace the functionality of > Mapcatalogs. > The previous plan that kept Mapcatalogs has been moved to > Rejected Alternatives. > > > > Best, > Feng > > On Sun, Apr 30, 2023 at 9:03 PM Jing Ge

Re: [VOTE] Release flink-connector-rabbitmq v3.0.1, release candidate #1

2023-05-06 Thread Jing Ge
+1 - built the source - verified signature - verified hash - contains no compiled binaries - checked tag - web PR looks good Best regards, Jing On Fri, May 5, 2023 at 9:14 AM Khanh Vu wrote: > Sorry, the above report is supposed for flink-connector-gpc-pubsub-3.0.1 > > - > Her

Re: flink documents Korean translation suggestions

2023-05-07 Thread Jing Ge
Hi Tan, Thanks for raising the suggestion. Proposals are always welcome. Flink is very well documented which turns out the translation effort is big. Afaiu, there are limited contributors in the community who could review content in Korean. I am wondering how you will continuously move forward and

Re: [VOTE] FLIP-306: Unified File Merging Mechanism for Checkpoints

2023-05-09 Thread Jing Ge
Hi Zakelly, I saw you sent at least 4 same emails for voting FLIP-306. I guess this one should be the last one and the right one for us to vote right? BTW, based on the sending time, 72 hours means to open the discussion until May 12th. Best regards, Jing On Tue, May 9, 2023 at 8:24 PM Zakelly L

Re: [NOTICE] Flink master branch now uses Maven 3.8.6

2023-05-13 Thread Jing Ge
Great news! We can finally get rid of additional setup to use maven 3.8. Thanks @Chesnay for your effort! Best regards, Jing On Sat, May 13, 2023 at 5:12 AM David Anderson wrote: > Chesnay, thank you for all your hard work on this! > > David > > On Fri, May 12, 2023 at 4:03 PM Chesnay Schepler

Re: [DISCUSS] Release Flink 1.17.1

2023-05-15 Thread Jing Ge
+1 for releasing 1.17.1 Best Regards, Jing On Thu, May 11, 2023 at 10:03 AM Martijn Visser wrote: > +1, thanks for volunteering! > > On Thu, May 11, 2023 at 9:23 AM Xintong Song > wrote: > > > +1 > > > > I'll help with the steps that require PMC privileges. > > > > Best, > > > > Xintong > > >

Re: [DISCUSS] FLIP-229: Prometheus Sink Connector

2023-05-18 Thread Jing Ge
Hi Karthi, Thanks for raising this proposal. It is a common use case to sink metric "data" into downstream Prometheus. The description in the motivation section is more or less misleading the discussion. I would suggest you rephrase it, e.g. metrics (pre)processing via Flink is The current F

Re: [DISCUSS] Release Flink 1.17.1

2023-05-18 Thread Jing Ge
23 at 6:10 AM Lijie Wang > > wrote: > > > > > +1 for the release. > > > > > > Best, > > > Lijie > > > > > > Jing Ge 于2023年5月15日周一 17:07写道: > > > > > > > +1 for releasing 1.17.1 > > > > > > > &

Re: Call for help on the Web UI (In-Place Rescaling)

2023-05-19 Thread Jing Ge
Hi David, Thanks for driving this. I'd like to second Emre, especially the second suggestion. In practice, the parallelism number could be big. Users will be frustrated, if they have to click hundred or even more times in order to reach the wished number. Anyone who will take over the design task,

Re: [DISCUSS] Release Flink 1.17.1

2023-05-19 Thread Jing Ge
y, while preparing the pull request for the blog post. > The specific released time depends on the speed at which they are > completed. VOTE thread will be launched no later than next Monday at the > latest. > > Best regards, > > Weijie > > > Jing Ge 于2023年5月19日周五 14:4

Re: [VOTE] Release 1.16.2, release candidate #1

2023-05-20 Thread Jing Ge
+1(non-binding) - reviewed Jira release notes - built from source - apache repos contain all necessary files - verified signatures - verified hashes - verified tag - reviewed PR Best regards, Jing On Sat, May 20, 2023 at 11:51 AM Yun Tang wrote: > +1 (non-binding) > > > * Verified signatur

Re: Maven plugin to detect issues early on

2023-05-21 Thread Jing Ge
Hi Emre, Thanks for your proposal. It looks very interesting! Please pay attention that most connectors have been externalized. Will your proposed plug be used for building Flink Connectors or Flink itself? Furthermore, it would be great if you could elaborate features wrt best practices so that w

Re: [VOTE] Release 1.17.1, release candidate #1

2023-05-22 Thread Jing Ge
+1(non-binding) - Verified signatures - Verified hashes - Verified tag - repos contain all necessary files - built from source - reviewed the PR Best regards, Jing On Mon, May 22, 2023 at 1:32 PM Yun Tang wrote: > +1 (non-binding) > > > * Verified signatures. > * Start a local standalo

Re: Maven plugin to detect issues early on

2023-05-22 Thread Jing Ge
get community acknowledgement/support for it. I believe working with > the Flink community on this is key as we'd need their support/opinion to do > this the right way and reach more Flink users. > > Thanks > Emre > > On 21/05/2023, 16:48, "Jing Ge" j...@ververica.com

Re: Maven plugin to detect issues early on

2023-05-22 Thread Jing Ge
cc user ML to get more attention, since the plugin will be used by Flink application developers. Best regards, Jing On Mon, May 22, 2023 at 3:32 PM Jing Ge wrote: > Hi Emre, > > Thanks for clarifying it. Afaiac, it is a quite interesting proposal, > especially for Flink job develo

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-05-23 Thread Jing Ge
Hi Yunfeng, Hi Dong Thanks for the informative discussion! It is a rational requirement to set different checkpoint intervals for different sources in a hybridSource. The tiny downside of this proposal, at least for me, is that I have to understand the upper-bound definition of the interval and th

Re: [RESULT][VOTE] Release 1.16.2, release candidate #1

2023-05-24 Thread Jing Ge
t; * Yuxin Tan > > * Yun Tang > > * Jing Ge > > * Xintong Song(binding) > > * Xingbo Huang(binding) > > * Qingsheng Ren(binding) > > * Samrat Deb > > * Hang Ruan > > > There are no disapproving votes. > > > I'll work on the steps to

Re: Zero-Downtime Deployments with Flink Operator

2023-05-24 Thread Jing Ge
Hi Kevin, Thanks for reaching out. This question is not new and has been asked many times previously. I have the same feeling as Gyula. There are ways to provide zero downtime for Flink jobs, but commonly, I doubt if it is really necessary. If there might be some special use cases that really nee

Re: [RESULT] [VOTE] Release 1.17.1, release candidate #1

2023-05-25 Thread Jing Ge
announce that we have unanimously approved this release. > > > > There are 7 approving votes, 3 of which are binding: > > > * Xintong Song(binding) > > * Yuxin Tan > > * Xingbo Huang(binding) > > * Yun Tang > > * Jing Ge > > * Qingsheng Ren(bind

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-05-25 Thread Jing Ge
saying that if there is a source that claims it > > is > > > running at the bounded stage, then Flink will use the " > > > execution.checkpoint.interval.bounded" as the checkpointing interval. > > > > > > Here are the the concerns I have with

Re: [RESULT] [VOTE] Release 1.17.1, release candidate #1

2023-05-25 Thread Jing Ge
t; new image. I estimate that this may take 1-2 days and then I will > officially announce the release of the new version immediately > > Best regards, > > Weijie > > > > Jing Ge 于2023年5月25日周四 19:42写道: > > > Hi Weijie, > > > > Thanks again for driving

Re: [ANNOUNCE] Apache Flink 1.17.1 released

2023-05-26 Thread Jing Ge
Hi Weijie, That is earlier than I expected! Thank you so much for your effort! Best regards, Jing On Fri, May 26, 2023 at 4:44 PM Martijn Visser wrote: > Same here as with Flink 1.16.2, thank you Weijie and those who helped with > testing! > > On Fri, May 26, 2023 at 1:08 PM weijie guo > wrot

Re: [ANNOUNCE] Apache Flink 1.16.2 released

2023-05-26 Thread Jing Ge
Hi Weijie, Thanks again for your effort. I was wondering if there were any obstacles you had to overcome while releasing 1.16.2 and 1.17.1 that could lead us to any improvement wrt the release process and management? Best regards, Jing On Fri, May 26, 2023 at 4:41 PM Martijn Visser wrote: > Th

Re: Questions on checkpointing mechanism for FLIP-27 Source API

2023-05-26 Thread Jing Ge
Hi Hong, Great question! Afaik, it depends on the implementation. Speaking of the "loopback" event sending to the SplitEnumerator, I guess you meant here [1] (It might be good, if you could point out the right position in the source code to help us understand the question better:)), which will end

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-05-26 Thread Jing Ge
Hi Aitozi, Thanks for your proposal. I am not quite sure if I understood your thoughts correctly. You described a special case implementation of the AsyncTableFunction with on public API changes. Would you please elaborate your purpose of writing a FLIP according to the FLIP documentation[1]? Than

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-05-28 Thread Jing Ge
So, in conclusion, no new interface is introduced, but we extend the > ability to support user-defined async table function. > > [1]: > > https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/ > [2]: https://lists.apache.org/thread/qljwd40v5ntz6

Re: [VOTE] Release flink-connector-jdbc v3.1.1, release candidate #1

2023-05-28 Thread Jing Ge
+1 (non-binding) - checked sign - checked hash - checked repos - checked tag - compiled from source - check the web PR Best regards, Jing On Sun, May 28, 2023 at 4:00 PM Benchao Li wrote: > Thanks Martijn, > > - checked signature/checksum [OK] > - downloaded src, compiled from source [OK] > -

Re: Kryo Upgrade: Request FLIP page create access

2023-05-28 Thread Jing Ge
Hi Qingsheng, Could you grant Kurt rights for creating a new FLIP page? Thanks! @Kurt Thanks for reaching out. Please pay attention to the FLIP number you will pick up and keep "Next FLIP number" on [1] up to date. Thanks! Best regards, Jing [1] https://cwiki.apache.org/confluence/display/FLIN

Re: [ANNOUNCE] Apache Flink 1.16.2 released

2023-05-29 Thread Jing Ge
gt; Best regards, > > Weijie > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release > > [2] https://github.com/pypa/twine/issues/997 > > > Jing Ge 于2023年5月27日周六 00:15写道: > > > Hi Weijie, > > > > Thanks again for your effort

Re: Kryo Upgrade: Request FLIP page create access

2023-05-29 Thread Jing Ge
Thanks Qingsheng for taking care of this! Best regards, Jing On Mon, May 29, 2023 at 4:24 AM Qingsheng Ren wrote: > Hi Kurt, > > The permission has been granted. Feel free to reach out if you have any > questions. > > Looking forward to your FLIP! > > Best, > Qingsheng > > On Mon, May 29, 2023

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-05-29 Thread Jing Ge
for further improvement. Best regards, Jing On Mon, May 29, 2023 at 5:31 AM Feng Jin wrote: > Hi all, I would like to update you on the latest progress of the FLIP. > > > Last week, Leonard Xu, HangRuan, Jing Ge, Shammon FY, ShengKai Fang and I > had an offline discussion regard

Re: [SUMMARY] Flink 1.18 Release Sync 05/30/2023

2023-05-30 Thread Jing Ge
Thanks Qingsheng for driving it! @Devs As you might already be aware of, there are many externalizations and new releases of Flink connectors. Once a connector has been externalized successfully, i.e. the related module has been removed in the Flink repo, we will not set a priority higher than maj

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-05-30 Thread Jing Ge
ut, as e2e latency is already high due to the backlog in the > sources > b) if there is no backpressure, that's the only case where the user > actually cares about the frequency of committing records to the output, we > are > using the more frequent checkpointing inter

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-05-31 Thread Jing Ge
Hi Feng, Thanks for the proposal! Very interesting feature. Would you like to update your thoughts described in your previous email about why SupportsTimeTravel has been rejected into the FLIP? This will help readers understand the context (in the future). Since we always directly add overload me

Re: [VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

2023-05-31 Thread Jing Ge
+1(non-binding) - verified sign - verified hash - checked repos - checked tag. NIT: the tag link should be: https://github.com/apache/flink-connector-pulsar/releases/tag/v3.0.1-rc1 - reviewed PR. NIT: left a comment. Best regards, Jing On Wed, May 31, 2023 at 11:16 PM Neng Lu wrote: > +1 > > I

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-01 Thread Jing Ge
ment to a > > CommonExecCorrelate node. So the runtime code should be generated in > > CorrelatedCodeGenerator. > > In CorrelatedCodeGenerator, we will know the TableFunction's Kind of > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE` > > For `FunctionKind

Re: [DISCUSS] Update Flink Roadmap

2023-06-01 Thread Jing Ge
Hi Jark, Thanks for driving it! For point 2, since we are developing 1.18 now, does it make sense to update the roadmap this time while we are releasing 1.18? This discussion thread will be focusing on the Flink 2.0 roadmap, as you mentioned previously. WDYT? Best regards, Jing On Thu, Jun 1, 20

Re: [DISCUSS] Update Flink Roadmap

2023-06-01 Thread Jing Ge
e an updated version of the > current roadmap. > Let's work together on refining the roadmap in this thread. > > Best, > Jark > > On Thu, 1 Jun 2023 at 23:25, Jing Ge wrote: > > > Hi Jark, > > > > Thanks for driving it! For point 2, since we are dev

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-02 Thread Jing Ge
> users have > logic to do some RPC call or query the third-party service which is also IO > intensive. > In these case, we'd like to leverage the async function to improve the > throughput. > > Thanks, > Aitozi. > > Jing Ge 于2023年6月1日周四 22:55写道: > > > Hi

Re: [DISCUSS] FLIP-307: Flink connector Redshift

2023-06-02 Thread Jing Ge
Hi Samrat, Excited to see your proposal. Supporting data warehouses is one of the major tracks for Flink. Thanks for driving it! Happy to see that we reached consensus to prioritize the Sink over Source in the previous discussion. Do you already have any prototype? I'd like to join the reviews. J

Re: [DISCUSS] FLIP-307: Flink connector Redshift

2023-06-05 Thread Jing Ge
Hi Samrat, Thanks for the feedback. I would suggest adding that information into the FLIP. +1 Looking forward to your PR :-) Best regards, Jing On Sat, Jun 3, 2023 at 9:19 PM Samrat Deb wrote: > Hi Jing Ge, > > >>> Do you already have any prototype? I'd like to

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-05 Thread Jing Ge
Hi Shammon, Thanks for driving it! It is a really interesting proposal. Looking forward to the follow-up FLIP for the lineage feature, users will love it :-) There are some inconsistencies in the content. In the very below example, listener.onEvent(CatalogModificationEvent) is called, while in th

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-06 Thread Jing Ge
LIP looks good to me. > > Hope to see the vote thread and following FLIP-314 discussion thread. > > Best, > Leonard > > > On Jun 6, 2023, at 5:04 PM, Shammon FY wrote: > > > > Hi, > > > > Thanks for all the feedback. > > > > For @Jing Ge,

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-06 Thread Jing Ge
Hi Mason, It is a very practical feature that many users are keen to use. Thanks to the previous discussion, the FLIP now looks informative. Thanks for your proposal. One small suggestion is that the attached images are quite small to read if we don't click and enlarge them. Besides that, It is di

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Jing Ge
Hi Ron, Thanks for raising the proposal. It is a very attractive idea! Since the FLIP is a relatively complex one which contains three papers and a design doc. It deserves more time for the discussion to make sure everyone is on the same page. I have a NIT question which will not block your voting

Re: [VOTE] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Jing Ge
+1 Best Regards, Jing On Wed, Jun 7, 2023 at 10:52 AM weijie guo wrote: > +1 (binding) > > Best regards, > > Weijie > > > Jingsong Li 于2023年6月7日周三 15:59写道: > > > +1 > > > > On Wed, Jun 7, 2023 at 3:03 PM Benchao Li wrote: > > > > > > +1, binding > > > > > > Jark Wu 于2023年6月7日周三 14:44写道: > >

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-08 Thread Jing Ge
://www.java2s.com/example/java-src/pkg/java/util/eventobject-85298.html On Thu, Jun 8, 2023 at 7:50 AM Shammon FY wrote: > Hi, > > To @Jing Ge > > Thanks for the clarification. Just out of curiosity, if the context is > not part of the event, why should it be the input parameter of each onE

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-08 Thread Jing Ge
efit the sql optimizer > > topic and observation > > > > For the performance benchmark, I agree with you. As I stated earlier, I > > think this is a new scope of work, we should design it separately, we can > > introduce this improvement in the future. > > > > [1

Call for Presentations: Flink Forward Seattle 2023

2023-06-08 Thread Jing Ge
ward to receiving your submission and welcoming you as a speaker at the Flink Forward Conference. Thank you for your time and consideration. Best regards, -- Jing Ge | Head of Engineering j...@ververica.com <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forw

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-08 Thread Jing Ge
> > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join > > > >.You > > > > can image the look up process as an async RPC call process. > > > > > > > > Let&

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-09 Thread Jing Ge
e option, I updated the FLIP about this you can > refer to > the section of "ConfigOption" and "Rejected Alternatives" > > In the end, for the performance evaluation, I'd like to do some tests and > will update it to the FLIP doc > > Thanks, > Aitozi. &g

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-09 Thread Jing Ge
of > joined result, and each of them will be collected > > > [1]: > > https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52 > > > Thanks, &g

Re: [DISCUSS] FLIP-307: Flink connector Redshift

2023-06-09 Thread Jing Ge
and experiences would be immensely valuable in making this > > decision. > > > > > > [1] > > > https://docs.aws.amazon.com/redshift/latest/dg/t_updating-inserting-using-staging-tables-.html > > [2] https://docs.aws.amazon.com/redshift/latest/dg/merge-exam

Re: [VOTE] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

2023-06-12 Thread Jing Ge
+1(binding) Thanks! Best regards, Jing On Mon, Jun 12, 2023 at 12:01 PM yuxia wrote: > +1 (binding) > Thanks Mang driving it. > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "zhangmang1" > 收件人: "dev" > 发送时间: 星期一, 2023年 6 月 12日 下午 5:31:10 > 主题: [VOTE] FLIP-305: Support atomic for CREATE

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-12 Thread Jing Ge
hink > that's why the > LookupFunction#lookup will return a collection of RowData. > > BTW, I think the behavior of lookup join will not affect the semantic of > the async udtf. > We use the Async TableFunction here and the table function can collect > multiple rows. > > Tha

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-12 Thread Jing Ge
n > > should be a point to be taken care about. Because in the ordered mode, > the > > head element in buffer may affect the > > total memory consumption. > > > > > > Thanks, > > Aitozi. > > > > > > > > Jing Ge 于2023年6月12日周一 20:28写道:

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-13 Thread Jing Ge
Hi Becket, Thanks for driving this important topic! There were many discussions previously that ended up with waiting up for a clear API deprecation process definition. This FLIP will help a lot. I'd like to ask some questions to understand your thoughts. Speaking of the FLIP, *"Always add a "S

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-13 Thread Jing Ge
Jun 13, 2023 at 4:25 PM Chesnay Schepler wrote: > On 13/06/2023 12:50, Jing Ge wrote: > > One major issue we have, afaiu, is caused by the lack of > housekeeping/house > > cleaning, there are many APIs that were marked as deprecated a few years > > ago and still don't g

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

2023-06-13 Thread Jing Ge
Hi yuxia, Thanks for your proposal and sorry for the late reply. The FLIP is in good shape. If I am not mistaken, Everything, that a stored procedure could do, could also be done by a Flink job. The current stored procedure design is to empower Catalogs to provide users some commonly used logics/f

Re: Re: [VOTE] FLIP-311: Support Call Stored Procedure

2023-06-13 Thread Jing Ge
+1(binding) Best Regards, Jing On Tue, Jun 13, 2023 at 9:03 AM Mang Zhang wrote: > +1 (no-binding) > > > > > -- > > Best regards, > Mang Zhang > > > > > > 在 2023-06-13 13:19:31,"Lincoln Lee" 写道: > >+1 (binding) > > > >Best, > >Lincoln Lee > > > > > >Jingsong Li 于2023年6月13日周二 10:07写道: > > > >>

Re: [VOTE] FLIP-294: Support Customized Catalog Modification Listener

2023-06-14 Thread Jing Ge
+1 (binding) Best Regards, Jing On Wed, Jun 14, 2023 at 4:07 PM Benchao Li wrote: > +1 (binding) > > Shammon FY 于2023年6月14日周三 19:52写道: > > > Hi all: > > > > Thanks for all the feedback for FLIP-294: Support Customized Catalog > > Modification Listener [1]. I would like to start a vote for it a

Re: [VOTE] FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

2023-06-14 Thread Jing Ge
+1 (binding) Best Regards, Jing On Wed, Jun 14, 2023 at 3:28 PM Rui Fan <1996fan...@gmail.com> wrote: > +1(binding) > > Best, > Rui Fan > > On Wed, Jun 14, 2023 at 16:24 Hang Ruan wrote: > > > +1 (non-binding) > > > > Thanks for Feng driving it. > > > > Best, > > Hang > > > > Feng Jin 于2023年6月

Re: [VOTE] FLIP-246: Dynamic Kafka Source (originally Multi Cluster Kafka Source)

2023-06-16 Thread Jing Ge
+1 (binding) Best regards, Jing On Thu, Jun 15, 2023 at 7:55 PM Mason Chen wrote: > Hi all, > > Thank you to everyone for the feedback on FLIP-246 [1]. Based on the > discussion thread [2], we have come to a consensus on the design and are > ready to take a vote to contribute this to Flink. > >

Re: [VOTE] FLIP-287: Extend Sink#InitContext to expose TypeSerializer, ObjectReuse and JobID

2023-06-16 Thread Jing Ge
+1(binding) Best Regards, Jing On Fri, Jun 16, 2023 at 10:10 AM Lijie Wang wrote: > +1 (binding) > > Thanks for driving it, Joao. > > Best, > Lijie > > Joao Boto 于2023年6月16日周五 15:53写道: > > > Hi all, > > > > Thank you to everyone for the feedback on FLIP-287[1]. Based on the > > discussion thre

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-17 Thread Jing Ge
i.e. If there is a cascading backwards > incompatible change in the connectors due to a backwards incompatible > change in the core, and as a result a longer migration period is required, > then I think we should postpone the removal of source code. But in general, > we should be able to

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-17 Thread Jing Ge
Hi All, The @Public -> @PublicEvolving proposed by Xintong is a great idea. Especially, after he suggest @PublicRetired, i.e. @PublicEvolving --(2 minor release)--> @Public --> @deprecated --(1 major release)--> @PublicRetired. It will provide a lot of flexibility without breaking any rules we had

Re: [NOTICE] Experimental Java 17 support now available on master

2023-06-18 Thread Jing Ge
Hi Kurt, Thanks for your contribution. I am a little bit confused about the email title, since your PR[1] is not merged into the master yet. I guess, with "Experimental Java 17 support", you meant it is available on your branch which is based on the master. If I am not mistaken, there is no vote

Re: [VOTE] FLIP-308: Support Time Travel

2023-06-19 Thread Jing Ge
+1(binding) On Mon, Jun 19, 2023 at 1:57 PM Benchao Li wrote: > +1 (binding) > > Lincoln Lee 于2023年6月19日周一 19:40写道: > > > +1 (binding) > > > > Best, > > Lincoln Lee > > > > > > yuxia 于2023年6月19日周一 19:30写道: > > > > > +1 (binding) > > > Thanks Feng driving it. > > > > > > Best regards, > > > Yux

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

2023-06-19 Thread Jing Ge
Thanks for your reply. Nice feature! Best regards, Jing On Wed, Jun 14, 2023 at 3:11 AM yuxia wrote: > Yes, you're right. > > Best regards, > Yuxia > > ----- 原始邮件 - > 发件人: "Jing Ge" > 收件人: "dev" > 发送时间: 星期三, 2023年 6 月 14日 上午 4:46:

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-19 Thread Jing Ge
Hi Lijie, Thanks for your proposal. It is a really nice feature. I'd like to ask a few questions to understand your thoughts. Afaiu, the runtime Filter will only be Injected when the gap between the build data size and prob data size is big enough. Let's make an extreme example. If the small tabl

[DISCUSS] Graduate the FileSink to @PublicEvolving

2023-06-20 Thread Jing Ge
Hi all, The FileSink has been marked as @Experimental[1] since Oct. 2020. According to FLIP-197[2], I would like to propose to graduate it to @PublicEvloving in the upcoming 1.18 release. On the other hand, as a related topic, FileSource was marked as @PublicEvolving[3] 3 years ago. It deserves a

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-20 Thread Jing Ge
t seem to affect anything, and the user is > also likely to change the value of the size. One question, how do you think > we should make it easier for users to control the filter injection? > > > [1]: > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-20 Thread Jing Ge
> >>>>>> choice rather than upgrading to 2.0, while satisfying the > >>>> 2-minor-release > >>>>>> migration period. > >>>>>> > >>>>>> I think my major point is, we should not carry APIs deprecated in a > >>&

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-21 Thread Jing Ge
ow the simple way, the subsequent gradual optimization. The first > step may be that we can verify the reasonableness of current formula by > TPC-DS case. > > Best, > Ron > > Jing Ge 于2023年6月20日周二 19:46写道: > > > Hi Ron, > > > > Thanks for the clarification. That

Re: [DISCUSS] Graduate the FileSink to @PublicEvolving

2023-06-22 Thread Jing Ge
le#issue/FLINK-30238 > > Based on prior discussion, I believe this could lead to data loss with > FileSink. > > > > On Tue, Jun 20, 2023, 5:41 AM Jing Ge wrote: > > > Hi all, > > > > The FileSink has been marked as @Experimental[1] since Oct. 2020. > > Ac

Re: [VOTE] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-23 Thread Jing Ge
+1(binding) Best Regards, Jing On Fri, Jun 23, 2023 at 5:50 PM Lijie Wang wrote: > Hi all, > > Thanks for all the feedback about the FLIP-324: Introduce Runtime Filter > for Flink Batch Jobs[1]. This FLIP was discussed in [2]. > > I'd like to start a vote for it. The vote will be open for at le

Re: [DISCUSS] Graduate the FileSink to @PublicEvolving

2023-06-26 Thread Jing Ge
te to files without > incurring the risk of data loss. > > I'm sure I'd feel better about things if there were an ongoing effort to > address this FileSink issue and/or a commitment that StreamingFileSink > would not be removed until this issue is addressed. > >

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-26 Thread Jing Ge
> > > > > 3. Maintain the DataStream API throughout 2.X and remove it until 3.x. > > But > > > there's no need to assume that 2.X is a short version, it's still a > > normal > > > major version. > > > > > > Best, > > > Jingson

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-06-26 Thread Jing Ge
Hi Paul, Thanks for driving it and thank you all for the informative discussion! The FLIP is in good shape now. As described in the FLIP, SQL Driver will be mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway in application mode and 2. external system integration. Would you like

Re: [DISCUSS] FLIP-303: Support REPLACE TABLE AS SELECT statement

2023-06-27 Thread Jing Ge
Hi Yuxia, Thanks for the proposal. Many engines like Snowflake, Databricks support it. +1 "3:Check the atomicity is enabled, it requires both the options table.rtas-ctas.atomicity-enabled is set to true and the corresponding table sink implementation SupportsStaging." Typo? "Option" instead of "

Re: [VOTE] FLIP-303: Support REPLACE TABLE AS SELECT statement

2023-06-28 Thread Jing Ge
+1(binding) On Wed, Jun 28, 2023 at 1:51 PM Mang Zhang wrote: > +1 (no-binding) > > > -- > > Best regards, > Mang Zhang > > > > > > At 2023-06-28 17:48:15, "yuxia" wrote: > >Hi everyone, > >Thanks for all the feedback about FLIP-303: Support REPLACE TABLE AS > SELECT statement[1]. Based on the

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-06-28 Thread Jing Ge
Hi Shammon, Thanks for your proposal. After reading the FLIP, I'd like to ask some questions to make sure we are on the same page. Thanks! 1. TableColumnLineageRelation#sinkColumn() should return TableColumnLineageEntity instead of String, right? 2. Since LineageRelation already contains all inf

  1   2   3   4   5   6   7   8   >