Re: [VOTE] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

2023-10-10 Thread Shammon FY
+1(binding), good job! Best, Shammon FY On Wed, Oct 11, 2023 at 10:18 AM Benchao Li wrote: > +1 (binding) > > Rui Fan <1996fan...@gmail.com> 于2023年10月11日周三 10:17写道: > > > > +1(binding) > > > > Best, > > Rui > > > > On Wed, Oct 11,

Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources

2023-10-08 Thread Shammon FY
+1 (binding) On Mon, Oct 9, 2023 at 11:12 AM Benchao Li wrote: > +1 (binding) > > Zhanghao Chen 于2023年10月9日周一 10:20写道: > > > > Hi All, > > > > Thanks for all the feedback on FLIP-367: Support Setting Parallelism for > Table/SQL Sources [1][2]. > > > > I'd like to start a vote for FLIP-367. The

[RESULT][VOTE] FLIP-314: Support Customized Job Lineage Listener

2023-10-06 Thread Shammon FY
(binding) - 曹帝胄 - Chen Zhanghao - Rui Fan(binding) - Yuepeng Pan There is no disapproving vote. Best, Shammon FY [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener [2] https://lists.apache.org/thread/dxdqjc0dd8rf1vbdg755zo1n2zj1tj8d

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

2023-10-06 Thread Shammon FY
ch as ANY/SLOTS for RM and ANY/TASKS for slot pool? I want this because we would like to add a new slot to TM strategy such as SLOTS_NUM in the future for OLAP to improve the performance for olap jobs, which will use TASKS strategy for task to slot. cc Guoyangze Best, Shammon FY On Fri, Oct 6, 2023

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

2023-09-27 Thread Shammon FY
, Shammon FY On Thu, Sep 28, 2023 at 12:08 PM Rui Fan <1996fan...@gmail.com> wrote: > Hi Zhu Zhu, > > Thanks for your feedback here! > > You are right, user needs to set 2 options: > - cluster.evenly-spread-out-slots=true > - slot.sharing-strategy=TASK_BALANCED_PRE

Re: [VOTE] FLIP-362: Support minimum resource limitation

2023-09-25 Thread Shammon FY
+1(binding), thanks for the proposal. Best, Shammon FY On Mon, Sep 25, 2023 at 11:20 PM Jing Ge wrote: > +1(binding) Thanks! > > Best regards, > Jing > > On Mon, Sep 25, 2023 at 3:48 AM Chen Zhanghao > wrote: > > > Thanks for driving this. +1 (non-binding) &

[VOTE] FLIP-314: Support Customized Job Lineage Listener

2023-09-24 Thread Shammon FY
/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener [2] https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc Best, Shammon FY

Re: FW: RE: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-09-24 Thread Shammon FY
20:56,Leonard Xu 写道: > > > > Thanks Shammon for the informations, the comment makes the lifecycle > clearer. > > +1 > > > > > > Best, > > Leonard > > > > > >> On Sep 18, 2023, at 7:54 PM, Shammon FY wrote: > >> > >&g

Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

2023-09-21 Thread Shammon FY
+1 for the proposal, thanks for driving. Bet, Shammon FY On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo wrote: > Thanks for driving this, +1 for the proposal. > > Best, > Yangze Guo > > > On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang > wrote: > > > > Hi Junrui,

Re: [Discuss] FLIP-362: Support minimum resource limitation

2023-09-20 Thread Shammon FY
Hi, I agree that `minimum resource limitation` will bring values for flink session clusters, but for `Application Mode`, is it useful for streaming and batch jobs? Is it necessary for us to not support the application mode, rather than relying on the default value 0? Best, Shammon FY On Thu

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

2023-09-19 Thread Shammon FY
Thanks for all the valuable feedback, helm charts / templates sound good to me too. It will bring a lot of convenience to the production deployment of Sql-Gateway, looking forward to this, thanks Best, Shammon FY On Wed, Sep 20, 2023 at 10:01 AM Yangze Guo wrote: > Thanks for the reply, Gy

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-09-18 Thread Shammon FY
existing transformations, the added lineage information will also be deleted. Please help to review it again, and If there are no more concerns about FLIP -314[1], I would like to start voting later, thanks. cc @Leonard Best, Shammon FY On Mon, Jul 17, 2023 at 3:43 PM Shammon FY wrote: > Hi d

Re: [DISCUSS] FLIP-360: Merging ExecutionGraphInfoStore and JobResultStore into a single component

2023-09-17 Thread Shammon FY
nents into one interface such as `CompletedJobStore` may be a little strange. What do you think? Best, Shammon FY On Fri, Sep 8, 2023 at 8:08 PM Gyula Fóra wrote: > Hi Matthias! > > Thank you for the detailed proposal, overall I am in favor of making this > unification to simpl

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

2023-09-17 Thread Shammon FY
add a Helm chart for sql-gateway, or should we need to add this feature to the flink operator? Can you help give the conclusion? Thank you very much @Gyula [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/olap_quickstart/ Best, Shammon FY On Sun, Sep 17, 2023 at 2:02 PM

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

2023-09-14 Thread Shammon FY
er pod? I think maybe we need a script like `kubernetes-sql-gatewah.sh` to start `sql-gateway` pods with a flink image, what do you think? Best, Shammon FY On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang wrote: > Hi, I've experiment this feature on K8S recently, here is some of my trial

Re: [DISCUSS] Flink annotation strategy/consensus

2023-09-10 Thread Shammon FY
be marked as `@Public`? I find it is difficult to determine this. Is `@PublicEvolving` really necessary? Should we directly remove `@PublicEvolving` and use `@Public` instead? I think it would be simpler. Best, Shammon FY On Mon, Sep 11, 2023 at 11:05 AM Becket Qin wrote: > Hi Jing, > >

Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

2023-09-10 Thread Shammon FY
are not annotated as `Public`. So do you want to annotate them as `Plublic`? If not, I think you may need to move them from `Public Interfaces` to `Proposed Changes`. Best, Shammon FY On Sat, Sep 9, 2023 at 12:11 PM Chen Zhanghao wrote: > Hi Devs, > > I would like to start a discussio

Re: [DISCUSS] Update Flink Roadmap

2023-08-14 Thread Shammon FY
Hi Jark, Sounds good and I would love to, thanks! I will involve you and Xingtong on the document after updating. Best, Shammon FY On Mon, Aug 14, 2023 at 10:39 PM Jark Wu wrote: > Hi Shammon, > > Sure, could you help to draft a subsection about this in the google doc? > &g

Re: [DISCUSS] How about adding OLAP to Flink Roadmap?

2023-08-14 Thread Shammon FY
Song could you please help to add the flink olap related subsection to the doc [2] ? Thanks very much! [1] https://lists.apache.org/thread/szdr4ngrfcmo7zko4917393zbqhgw0v5 [2] https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit Best, Shammon FY On Mon, Aug 14, 2023

Re: [DISCUSS] Update Flink Roadmap

2023-08-14 Thread Shammon FY
for Flink? Best, Shammon FY On Mon, Aug 14, 2023 at 8:03 PM Jark Wu wrote: > Thank you everyone for helping polish the roadmap [1]. > > I think I have addressed all the comments and we have included all ongoing > parts of Flink. > Please feel free to take a last look. I'm

[DISCUSS] How about adding OLAP to Flink Roadmap?

2023-08-07 Thread Shammon FY
Best, Shammon FY

Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu

2023-08-03 Thread Shammon FY
Congratulations, Weihua! Best, Shammon FY On Fri, Aug 4, 2023 at 12:33 PM Jing Ge wrote: > congrats! Weihua! > > Best regards, > Jing > > On Fri, Aug 4, 2023 at 12:20 PM Matt Wang wrote: > > > Congratulations Weihua ~ > > > > > > -- > > >

Re: [ANNOUNCE] New Apache Flink Committer - Hong Teoh

2023-08-03 Thread Shammon FY
Congratulations, Hong! Best, Shammon FY On Fri, Aug 4, 2023 at 12:33 PM Jing Ge wrote: > congrats! Hong! > > Best regards, > Jing > > On Fri, Aug 4, 2023 at 11:48 AM Qingsheng Ren wrote: > > > Congratulations and welcome aboard, Hong! > > > > Best, >

Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl

2023-08-03 Thread Shammon FY
Congratulations, Matthias! On Fri, Aug 4, 2023 at 1:13 PM Samrat Deb wrote: > Congrats, Matthias! > > > On Fri, 4 Aug 2023 at 10:13 AM, Benchao Li wrote: > > > Congratulations, Matthias! > > > > Jing Ge 于2023年8月4日周五 12:35写道: > > > > > Congrats! Matthias! > > > > > > Best regards, > > > Jing >

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-07-17 Thread Shammon FY
Hi devs, Thanks for all the valuable feedback. If there are no more concerns about FLIP-314[1], I would like to start voting later, thanks. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener Best, Shammon FY On Wed, Jul 12, 2023 at 11

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-07-11 Thread Shammon FY
job and remove previous methods in `DataStreamSource` and `DataStreamSink`. Looking forward to your feedback, thanks. Best, Shammon FY

Re: [ANNOUNCE] Flink 1.18 Feature Freeze Extended until July 24th, 2023

2023-07-06 Thread Shammon FY
Thanks for driving the release and sharing the update, looking forward to 1.18 Best, Shammon FY On Fri, Jul 7, 2023 at 10:56 AM Yun Tang wrote: > Thanks for driving this release and sharing the update on the feature > freeze extension. > > >

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-07-06 Thread Shammon FY
Thanks Jing, sounds good to me. I have updated the FLIP and renamed the lineage related classes to `LineageGraph`, `LineageVertex` and `LineageEdge` and keep it consistent with the job definition in Flink. Best, Shammon FY On Thu, Jul 6, 2023 at 8:25 PM Jing Ge wrote: > Hi Shammon, > &g

Re: [DISCUSS] FLIP-325: Support configuring end-to-end allowed latency

2023-07-05 Thread Shammon FY
latMap/Filter Operator is not a good idea which makes data no longer flowing. Best, Shammon FY On Thu, Jul 6, 2023 at 1:54 AM Piotr Nowojski wrote: > Hi, > > Thanks for this proposal, this is a very much needed thing that should be > addressed in Flink. > > I think there is one

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-05 Thread Shammon FY
Congratulations! Best, Shammon FY On Wed, Jul 5, 2023 at 2:38 PM Paul Lam wrote: > Congrats and cheers! > > Best, > Paul Lam > > > 2023年7月4日 18:04,Benchao Li 写道: > > > > Congratulations! > > > > Feng Jin 于2023年7月4日周二 16:17写道: > >

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-07-05 Thread Shammon FY
Hi devs, Thanks for all the feedback. I have discussed with @QingSheng Ren off-line to confirm some questionable points in the FLIP. Thanks for his valuable inputs and I have updated the FLIP according to our discussion. Looking forward to your feedback, thanks, Best, Shammon FY On Wed, Jul

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-07-05 Thread Shammon FY
think `Entity` and `Relation` are nicer for lineage, what do you think of it? Best, Shammon FY On Thu, Jun 29, 2023 at 4:21 AM Jing Ge wrote: > Hi Shammon, > > Thanks for your proposal. After reading the FLIP, I'd like to ask > some questions to make sure we are on the same p

Re: [DISCUSS] FLIP-325: Support configuring end-to-end allowed latency

2023-06-29 Thread Shammon FY
receives all `FlushEvent` from upstream with the same ID or do flush for each `FlushEvent`? Can you give more detailed proposal about that? We also have a demand for this piece, thanks Best, Shammon FY On Thu, Jun 29, 2023 at 4:35 PM Martijn Visser wrote: > Hi Dong and Yunfeng, > &g

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-06-27 Thread Shammon FY
IP later and please help to review it, thanks >2. it is relatively easy to obtain column lineage through Calcite MetaQuery API It's helpful if Calcite already has some column lineage in meta, I think we can discuss and give the proposal in the column lineage FLIP Best, Shammon FY On We

Re: [DISCUSS] Persistent SQL Gateway

2023-06-27 Thread Shammon FY
s are successfully submitted, the lifecycle is no longer managed by SQL gateway. I don't know if it covers your usage scenario. Could you describe yours for us to test and confirm? Best, Shammon FY On Tue, Jun 27, 2023 at 6:43 PM Ferenc Csaky wrote: > Hi Jark, > > In the current impl

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-06-25 Thread Shammon FY
next step. So we have comprehensively considered the table lineage and column lineage interfaces here, and defined these two interfaces together clearly [1] https://github.com/FangYongs/flink/commit/d4bfe57e7a5315b790e79b8acef8b11e82c9187c Best, Shammon FY On Sun, Jun 25, 2023 at 4:17 PM Y

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-06-25 Thread Shammon FY
nd update lineage for `INSERT` jobs in our streaming & batch ETL, and display the `override` information on the UI. Best, Shammon FY On Tue, Jun 20, 2023 at 6:19 PM Qingsheng Ren wrote: > Hi Shammon, > > Thanks for starting this FLIP! Data lineage is a very important topic, > w

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-06-19 Thread Shammon FY
Hi devs, Is there any comment or feedback for this FLIP? Hope to hear from you, thanks Best, Shammon FY On Tue, Jun 6, 2023 at 8:22 PM Shammon FY wrote: > Hi devs, > > I would like to start a discussion on FLIP-314: Support Customized Job > Lineage Listener[1] which is the next s

[RESULT][VOTE] FLIP-294: Support Customized Catalog Modification Listener

2023-06-19 Thread Shammon FY
for participating! [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Catalog+Modification+Listener [2] https://lists.apache.org/thread/5f9806kw1q01536wzwgx1psh7jhmfmjw Best, Shammon FY

Re: [VOTE] FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

2023-06-15 Thread Shammon FY
Thanks Feng for the FLIP. +1(non-binding) Best, Shammon FY On Thu, Jun 15, 2023 at 6:30 PM Leonard Xu wrote: > Thanks Feng for driving this FLIP forward. > > +1(binding) > > Best, > Leonard > > > On Jun 15, 2023, at 1:42 PM, Dong Lin wrote: > > > &g

[VOTE] FLIP-294: Support Customized Catalog Modification Listener

2023-06-14 Thread Shammon FY
objection or an insufficient number of votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Catalog+Modification+Listener [2] https://lists.apache.org/thread/185mbcwnpokfop4xcb22r9bgfp2m68fx Best, Shammon FY

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-13 Thread Shammon FY
Hi all, Thanks for all the valuable feedback. If there's no further question or concern, I will start voting for FLIP-294 [1] later. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Catalog+Modification+Listener Best, Shammon FY On Fri, Jun 9, 2023

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-08 Thread Shammon FY
Thanks Jing, it makes sense to me and I have updated the FLIP Best, Shammon FY On Thu, Jun 8, 2023 at 11:15 PM Jing Ge wrote: > Hi Shammon, > > If we take a look at the JDK Event design as a reference, we can even add > an Object into the event [1]. Back to the CatalogModif

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-07 Thread Shammon FY
ication Listener" will be more suitable, I'll update the title to it. Best, Shammon FY On Thu, Jun 8, 2023 at 12:25 PM Mason Chen wrote: > Hi Shammon, > > FLIP generally looks good and I'm excited to see this feature. > > 1. I'm also curious about default imp

[DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-06-06 Thread Shammon FY
+Listener [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener Best, Shammon FY

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-06 Thread Shammon FY
e path for running per-job/per-session jobs. The listener will be created in `TableEnvironment` and `SqlGateway` which can a `CatalogManager` with the listener. Best, Shammon FY On Tue, Jun 6, 2023 at 3:33 PM David Morávek wrote: > Hi, > > Thanks for the FLIP! Data lineage is an importan

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-05 Thread Shammon FY
Hi devs: Thanks for all the feedback, and if there are no more comments, I will start a vote on FLIP-294 [1] later. Thanks again. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener Best, Shammon FY On Tue, Jun 6, 2023 at 1:40 PM Shammon

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-05 Thread Shammon FY
Hi Martijn, Thanks for your attention, I will soon initiate a discussion about FLIP-314. Best, Shammon FY On Fri, Jun 2, 2023 at 2:55 AM Martijn Visser wrote: > Hi Shammon, > > Just wanted to chip-in that I like the overall FLIP. Will be interesting to > see the follow-up discus

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-06-01 Thread Shammon FY
connector such as iceberg/paimon can create sources from the `CatalogBaseTable` directly without the need to get the snapshot ID from `CatalogTable.getSnapshot()`. What do you think of it? Best, Shammon FY On Thu, Jun 1, 2023 at 7:22 AM Jing Ge wrote: > Hi Feng, > > Thanks for the

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-31 Thread Shammon FY
x27;s better to stay consistent with `Catalog.alterDatabase`. What do you think? Best, Shammon FY On Thu, Jun 1, 2023 at 10:25 AM yuxia wrote: > Hi, Shammon. > I mean do we need to contain the new database after alter in > AlterDatabaseEvent? So that the listener can know what has been mod

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-05-31 Thread Shammon FY
= catalogStore.get(catalogName);` in the `CatalogManager` should be `Optional optionalDescriptor = catalogStore.get(catalogName);` Best, Shammon FY On Wed, May 31, 2023 at 2:24 PM liu ron wrote: > Hi, Feng > > Thanks for driving this FLIP, this proposal is very useful for catalog > man

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-30 Thread Shammon FY
Hi yuxia Thanks for your input. The `AlterDatabaseEvent` extends `DatabaseModificationEvent` which has the original database. Best, Shammon FY On Wed, May 31, 2023 at 2:24 PM yuxia wrote: > Thanks Shammon for driving it. > The FLIP generally looks good to me. I only have one question.

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-30 Thread Shammon FY
andle the exceptions too. If users fail to create a listener and throw an exception, Flink will throw the exception too. Best, Shammon FY On Wed, May 31, 2023 at 11:36 AM liu ron wrote: > Hi, Shammon > > Thanks for driving this FLIP, It will enforce the Flink metadata capability > from

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-05-30 Thread Shammon FY
le. Should sql driver support dynamic parameters and specify config file? Best, Shammon FY On Tue, May 30, 2023 at 9:57 PM Weihua Hu wrote: > Thanks Paul for the proposal. > > +1 for this. It is valuable in improving ease of use. > > I have a few questions. > - Is SQLRunner the

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-30 Thread Shammon FY
stener. As mentioned above, Flink will not roll back the DDL even when the listener throws an exception. I think permission verification is another issue and can be discussed separately. Best, Shammon FY On Tue, May 30, 2023 at 1:07 AM Feng Jin wrote: > Hi, Shammon > > Thanks for dr

[DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-05-26 Thread Shammon FY
Hi devs, We would like to bring up a discussion about FLIP-294: Support Customized Job Meta Data Listener[1]. We have had several discussions with Jark Wu, Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions and interfaces, and thanks for their valuable advice. The overall job and

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-05-25 Thread Shammon FY
? Best, Shammon FY On Fri, May 26, 2023 at 1:20 PM Benchao Li wrote: > Regarding the implementation, did you consider the pushdown abilities > compatible, e.g., projection pushdown, filter pushdown, partition pushdown. > Since `Snapshot` is not handled much in existing rules, I have

Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu

2023-04-23 Thread Shammon FY
Congratulations, Leonard! Best, Shammon FY On Sun, Apr 23, 2023 at 5:07 PM Xianxun Ye wrote: > Congratulations, Leonard! > > Best regards, > > Xianxun > > > 2023年4月23日 09:10,Lincoln Lee 写道: > > > > Congratulations, Leonard! > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren

2023-04-23 Thread Shammon FY
Congratulations, Qingsheng! Best, Shammon FY On Sun, Apr 23, 2023 at 4:40 PM Weihua Hu wrote: > Congratulations, Qingsheng! > > Best, > Weihua > > > On Sun, Apr 23, 2023 at 3:53 PM Yun Tang wrote: > > > Congratulations, Qingsh

Re: [VOTE] FLIP-288: Enable Dynamic Partition Discovery by Default in Kafka Source

2023-04-22 Thread Shammon FY
+1 (non-binding) Best, Shammon FY On Sun, Apr 23, 2023 at 10:35 AM Qingsheng Ren wrote: > Thanks for pushing this FLIP forward, Hongshun! > > +1 (binding) > > Best, > Qingsheng > > On Fri, Apr 21, 2023 at 2:52 PM Hongshun Wang > wrote: > > > Dear Flink D

Re: [DISCUSS] FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source

2023-04-18 Thread Shammon FY
if we always use EARLIEST for new discovery partitions. Best, Shammon FY On Tue, Apr 18, 2023 at 4:59 PM Hongshun Wang wrote: > Hi Shammon, > > Thank you for your advice.I have carefully considered whether to show this > in SQL DDL. Therefore, I carefully studied whether it is

Re: Re: [VOTE] FLIP-302: Support TRUNCATE TABLE statement in batch mode

2023-04-17 Thread Shammon FY
+1 (no-binding) Best, Shammon FY On Tue, Apr 18, 2023 at 12:56 PM Jacky Lau wrote: > +1 (no-binding) > > Best, > Jacky Lau > > Jingsong Li 于2023年4月18日周二 11:57写道: > > > +1 > > > > On Tue, Apr 18, 2023 at 9:39 AM Aitozi wrote: > > > > >

Re: [DISCUSS] FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source

2023-04-14 Thread Shammon FY
`. Do SQL jobs need to support this feature? Best, Shammon FY On Wed, Apr 12, 2023 at 2:27 PM Hongshun Wang wrote: > Hi everyone, > > I have already modified FLIP-288 to provide a > newDiscoveryOffsetsInitializer in the KafkaSourceBuilder and > KafkaSourceEnumerator

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-04-12 Thread Shammon FY
Hi Feng Thanks for your update. I found there are two options in `Proposed Changes`, can you put the unselected option in `Rejected Alternatives`? I think this may help us better understand your proposal Best, Shammon FY On Thu, Apr 13, 2023 at 4:49 AM Jing Ge wrote: > Hi Feng, > &g

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-04-05 Thread Shammon FY
g the catalog in `Map catalogs` and `CatalogStore catalogStore` together when it is registered or added in `CatalogManager`? Best, Shammon FY On Tue, Apr 4, 2023 at 5:22 PM Feng Jin wrote: > Thank you for your reply. I am very sorry for the misunderstanding caused > by my deviation from t

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-04-04 Thread Shammon FY
that conclusion. What about the proposal of the discussion in thread [1] ? [1] https://lists.apache.org/thread/9bnjblgd9wvrl75lkm84oo654c4lqv70 Best, Shammon FY On Tue, Apr 4, 2023 at 3:41 PM yuxia wrote: > Thanks Feng for driving this FLIP. > I have few comments: > 1: The mechani

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

2023-03-30 Thread Shammon FY
ed to understand the differences between the two `registerCatalog`? 3. How about supporting multiple catalogs in one `xml` or `json` file? Best, Shammon FY On Thu, Mar 30, 2023 at 2:21 PM Feng Jin wrote: > Hi everyone, > > I would like to start a discuss on FLIP-295: Support persistenc

Re: [DISCUSS] FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source

2023-03-29 Thread Shammon FY
OFFSET and Flink validate it. Then we can use the same strategy to apply to the newly discovered partition, I think this will be nice too Best, Shammon FY On Thu, Mar 30, 2023 at 12:29 PM Leonard Xu wrote: > Thanks Hongshun and Shammon for driving the FLIP! > > > > *2. Clar

Re: [DISCUSS] FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source

2023-03-27 Thread Shammon FY
Hi hongshun Thanks for updating, the FLIP's pretty good and looks good to me! Best, Shammon FY On Tue, Mar 28, 2023 at 11:16 AM Hongshun Wang wrote: > Hi Shammon, > > > Thanks a lot for your advise. I agree with your opinion now. It seems that > I forgot to consider

Re: [DISCUSS] FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source

2023-03-27 Thread Shammon FY
it? Best, Shammon FY On Fri, Mar 24, 2023 at 11:23 AM Hongshun Wang wrote: > "If all new messages in old partitions should be consumed, all new messages > in new partitions should also be consumed." > > Sorry, I wrote the last sentence incorrectly. > > On Fri, Ma

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Shammon FY
Congratulations! Best, Shammon FY On Mon, Mar 27, 2023 at 11:37 PM Samrat Deb wrote: > congratulations > > Bests, > Samrat > On Mon, 27 Mar 2023 at 8:24 PM, Alexander Fedulov < > alexander.fedu...@gmail.com> wrote: > > > Great to see this, congratulations!

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-03-24 Thread Shammon FY
Thanks Jing and Gordon, I have closed the pr https://github.com/apache/flink/pull/21965 and will open a new one for kafka connector Best, shammon FY On Saturday, March 25, 2023, Ran Tao wrote: > Thank you Gordon and all the people who have worked on the externalized > kafka implemen

Re: [DISCUSS] FLIP-292: Support configuring state TTL at operator level for Table API & SQL programs

2023-03-23 Thread Shammon FY
@Shuo's idea that for complex cases, users can combine hits and `json plan` to configure `TTL on Operator` better. What do you think? Thanks Best, Shammon FY On Thu, Mar 23, 2023 at 9:58 PM Shuo Cheng wrote: > Correction: “users can set 'scan.startup.mode' for kafka connec

Re: [DISCUSS] FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source

2023-03-23 Thread Shammon FY
t the TIMESTAMP with 2023-03-23 16:00:00 for their jobs. If a partition is added during this period, jobs will generate “surprising” data. What do you think of it? Best, Shammon FY On Tue, Mar 21, 2023 at 6:58 PM Hongshun Wang wrote: > Hi, Hang, > > Thanks for your advice. > > When

Re: [DISCUSS] FLIP-302: Support TRUNCATE TABLE statement

2023-03-22 Thread Shammon FY
these two operations in the truncate syntax? Or support adding options in `with`? Best, Shammon FY On Wed, Mar 22, 2023 at 9:13 PM yuxia wrote: > Hi, devs. > > I'd like to start a discussion about FLIP-302: Support TRUNCATE TABLE > statement [1]. > > The TRUNCATE T

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

2023-03-20 Thread Shammon FY
simple string collection for tags in listeners, such as key-value? Best, Shammon FY On Mon, Mar 20, 2023 at 7:48 PM Leonard Xu wrote: > Hi,Panagiotis > > > Thank you for kicking off this discussion. Overall, the proposed feature of > this FLIP makes sense to me. We have also di

[DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

2023-03-17 Thread Shammon FY
location. And then user can do more statistics according to different dimensions. 2. Users may want to save results in listener, and then they can get the historical results even jabmanager failover. Can we provide a unified implementation for data storage requirements? Best, shammon FY On

[RESULT][VOTE] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

2023-03-17 Thread Shammon FY
, Shammon FY

Re: [VOTE] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

2023-03-16 Thread Shammon FY
Hi everyone I'm closing this vote now. I will follow up with the result in another email. Thanks Best, Shammon FY On Thu, Mar 16, 2023 at 10:37 AM ConradJam wrote: > Hang Ruan ruanhang1...@gmail.com 通过 > <https://support.google.com/mail/answer/1311182?hl=zh-CN>“flink.apache.

Re: [DISCUSS] Enabling dynamic partition discovery by default in Kafka source

2023-03-15 Thread Shammon FY
Hi Hongshun Agree with @Etienne. And can you describe this process and the problems in a figure? Thanks Best, Shammon FY On Wed, Mar 15, 2023 at 5:15 PM Etienne Chauchot wrote: > Hi, > > Why not track this in a FLIP and a ticket and link this discussion thread. > > My 2 ce

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

2023-03-13 Thread Shammon FY
r is an implementation > to > > interact with Flink, so in theory, it would be doable to use > > Flink-JDBC-Driver in Flink-JDBC-Connector if we add it as a dialect. > > > > However, I don't see a scenario which requires this yet. All in all, it > > would be

[VOTE] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

2023-03-12 Thread Shammon FY
Hi Devs, I'd like to start the vote on FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway [1]. The FLIP was discussed in thread [2], and it aims to introduce Flink Jdbc Driver module in Flink. The vote will last for at least 72 hours (03/16, 15:00 UTC+8) unless there is an objection or insuff

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

2023-03-12 Thread Shammon FY
Congratulations, Yuxia On Mon, Mar 13, 2023 at 12:11 PM Jingsong Li wrote: > Congratulations, Yuxia! > > On Mon, Mar 13, 2023 at 11:49 AM Juntao Hu wrote: > > > > Congratulations, Yuxia! > > > > Best, > > Juntao > > > > > > Wencong Liu 于2023年3月13日周一 11:33写道: > > > > > Congratulations, Yuxia! >

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

2023-03-12 Thread Shammon FY
Best regards, > Jing > > [1] > > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/ > > On Fri, Mar 3, 2023 at 8:56 AM Shammon FY wrote: > > > Hi, > > > > Thanks jinsong. I think implementing JDBC for batch mode first soun

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

2023-03-09 Thread Shammon FY
Thanks weihua, +1 (non-binding) Best, Shammon On Fri, Mar 10, 2023 at 10:32 AM Xintong Song wrote: > +1 (binding) > > Best, > > Xintong > > > > On Thu, Mar 9, 2023 at 1:28 PM Weihua Hu wrote: > > > Hi Everyone, > > > > I would like to start the vote on FLIP-298: Unifying the Implementation > >

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

2023-03-06 Thread Shammon FY
; > > >> - There used to be a defaut slot manager implementation, which is > > > > >> non-declarative and has been removed now. The two features, > > > declarative > > > > / > > > > >> reactive resource management an

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

2023-03-02 Thread Shammon FY
integration for JDBC and streaming... > > Do you need to use JDBC in streaming mode? Or do we just implement > JDBC for batch only first? > > Best, > Jingsong > > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY wrote: > > > > Hi > > > > Thanks for the

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

2023-03-02 Thread Shammon FY
r `Public Interface` part, only > > `FlinkResultSet.getRowKind` is a true new interface, others are just > > implementations. > > > > If the user does not cast into a FlinkResultSet, will there be serious > > consequences here (RowKind is ignored)? > > > > Best,

Re: [VOTE] FLIP-291: Externalized Declarative Resource Management

2023-03-01 Thread Shammon FY
+1 (non-binding) Best, Shammon On Wed, Mar 1, 2023 at 4:51 PM Roman Khachatryan wrote: > +1 (binding) > > Thanks David, and everyone involved :) > > Regards, > Roman > > > On Wed, Mar 1, 2023 at 8:01 AM Gyula Fóra wrote: > > > +1 (binding) > > > > Looking forward to this :) > > > > Gyula > >

[DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

2023-03-01 Thread Shammon FY
Hi devs, I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway[1]. FLIP-275[2] supports remote sql client based on gateway, users can interact with gateway by flink console. However, for users who create session clusters with Flink, they'd like to use Jdbc Dri

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

2023-02-28 Thread Shammon FY
Hi Thanks for starting this work weihua, I think unifying DeclarativeSlotManager and FineGrainedSlotManager is valuable. I agree with @Matthias and @John that we need a way to ensure that DeclarativeSlotManager's capabilities are fully covered by FineGrainedSlotManager 1. For their functional di

[DISCUSS] Deprecate deserialize method in DeserializationSchema

2023-02-27 Thread Shammon FY
Hi devs Currently there are two deserialization methods in `DeserializationSchema` 1. `T deserialize(byte[] message)`, only deserialize one record from binary, if there is no record it should return null. 2. `void deserialize(byte[] message, Collector out)`, supports deserializing none, one or mul

Re: [Discuss] Some questions on flink-table-store micro benchmark

2023-02-27 Thread Shammon FY
Hi jingsong Getting rid of JMH is a good idea. For the second point, how can we track the performance changes of the micro benchmark? What do you think? Best, Shammon On Mon, Feb 27, 2023 at 10:57 AM Jingsong Li wrote: > Thanks Yun. > > Another way is we can get rid of JMH, something like Spar

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

2023-02-22 Thread Shammon FY
Hi kui Thanks for your answer and +1 to yuxia too > we should not bind the watermark-related options to a connector to ensure semantic clarity. In my opinion, adding watermark-related options to a connector is much more clear. Currently users can define simple watermark strategy in DDL, adding m

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

2023-02-21 Thread Shammon FY
Hi kui Thanks for initializing this discussion. I have two questions 1. What will happen if we add watermark related options in `the connector options`? Will the connector ignore these options or throw an exception? How can we support this? 2. If one table is used by two operators with different

Re: Proposal to improve the Apache Flink website

2023-02-20 Thread Shammon FY
Hi +1 too for this proposal Compared with the beginning, Flink has good applications in many directions now. The new website can give users better guidance for streaming/batch/olap/ML and so on. Looking forward to the new website :) Best, Shammon On Tue, Feb 21, 2023 at 2:24 PM Gyula Fóra wr

Re: [ANNOUNCE] New Apache Flink Committer - Rui Fan

2023-02-20 Thread Shammon FY
Congratulations, Rui! Best, Shammon On Tue, Feb 21, 2023 at 1:40 PM Sergey Nuyanzin wrote: > Congratulations, Rui! > > On Tue, Feb 21, 2023 at 4:53 AM Weihua Hu wrote: > > > Congratulations, Rui! > > > > Best, > > Weihua > > > > > > On Tue, Feb 21, 2023 at 11:28 AM Biao Geng wrote: > > > > >

Re: [ANNOUNCE] New Apache Flink Committer - Anton Kalashnikov

2023-02-20 Thread Shammon FY
Congratulations, Anton! Best, Shammon On Tue, Feb 21, 2023 at 1:41 PM Sergey Nuyanzin wrote: > Congratulations, Anton! > > On Tue, Feb 21, 2023 at 4:53 AM Weihua Hu wrote: > > > Congratulations, Anton! > > > > Best, > > Weihua > > > > > > On Tue, Feb 21, 2023 at 11:22 AM weijie guo > > wrote:

Re: [DISCUSS] Extract core autoscaling algorithm as new SubModule in flink-kubernetes-operator

2023-02-19 Thread Shammon FY
the two core > > > requirements, but the operator itself > > > cannot be used for Flink jobs running on YARN. Those core requirements > > are > > > solved through other mechanisms in the case of YARN. But the main > problem > > > for us is *how to* > &g

Re: [DISCUSS] Extract core autoscaling algorithm as new SubModule in flink-kubernetes-operator

2023-02-16 Thread Shammon FY
gt; > > On Thu, Feb 16, 2023 at 7:37 PM Samrat Deb > wrote: > > > > > Hi Max , > > > If you are fine and aligned with the same thought , since this is going > > to > > > be very useful to us, we are ready to help / contribute additional work > > > requ

  1   2   >