Re: [VOTE] FLIP-288: Enable Dynamic Partition Discovery by Default in Kafka Source

+1 (non-binding) Best, Shammon FY On Sun, Apr 23, 2023 at 10:35 AM Qingsheng Ren wrote: > Thanks for pushing this FLIP forward, Hongshun! > > +1 (binding) > > Best, > Qingsheng > > On Fri, Apr 21, 2023 at 2:52 PM Hongshun Wang > wrote: > > > Dear Flink D

Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren

Congratulations, Qingsheng! Best, Shammon FY On Sun, Apr 23, 2023 at 4:40 PM Weihua Hu wrote: > Congratulations, Qingsheng! > > Best, > Weihua > > > On Sun, Apr 23, 2023 at 3:53 PM Yun Tang wrote: > > > Congratulations, Qingsh

Re: [ANNOUNCE] New Apache Flink PMC Member - Leonard Xu

Congratulations, Leonard! Best, Shammon FY On Sun, Apr 23, 2023 at 5:07 PM Xianxun Ye wrote: > Congratulations, Leonard! > > Best regards, > > Xianxun > > > 2023年4月23日 09:10，Lincoln Lee 写道： > > > > Congratulations, Leonard! > >

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

? Best, Shammon FY On Fri, May 26, 2023 at 1:20 PM Benchao Li wrote: > Regarding the implementation, did you consider the pushdown abilities > compatible, e.g., projection pushdown, filter pushdown, partition pushdown. > Since `Snapshot` is not handled much in existing rules, I have

[DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

Hi devs, We would like to bring up a discussion about FLIP-294: Support Customized Job Meta Data Listener[1]. We have had several discussions with Jark Wu, Leonard Xu, Dong Lin, Qingsheng Ren and Poorvank about the functions and interfaces, and thanks for their valuable advice. The overall job and

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

stener. As mentioned above, Flink will not roll back the DDL even when the listener throws an exception. I think permission verification is another issue and can be discussed separately. Best, Shammon FY On Tue, May 30, 2023 at 1:07 AM Feng Jin wrote: > Hi, Shammon > > Thanks for dr

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

le. Should sql driver support dynamic parameters and specify config file? Best, Shammon FY On Tue, May 30, 2023 at 9:57 PM Weihua Hu wrote: > Thanks Paul for the proposal. > > +1 for this. It is valuable in improving ease of use. > > I have a few questions. > - Is SQLRunner the

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

andle the exceptions too. If users fail to create a listener and throw an exception, Flink will throw the exception too. Best, Shammon FY On Wed, May 31, 2023 at 11:36 AM liu ron wrote: > Hi, Shammon > > Thanks for driving this FLIP, It will enforce the Flink metadata capability > from

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

Hi yuxia Thanks for your input. The `AlterDatabaseEvent` extends `DatabaseModificationEvent` which has the original database. Best, Shammon FY On Wed, May 31, 2023 at 2:24 PM yuxia wrote: > Thanks Shammon for driving it. > The FLIP generally looks good to me. I only have one question.

Re: [DISCUSS] FLIP 295: Support persistence of Catalog configuration and asynchronous registration

= catalogStore.get(catalogName);` in the `CatalogManager` should be `Optional optionalDescriptor = catalogStore.get(catalogName);` Best, Shammon FY On Wed, May 31, 2023 at 2:24 PM liu ron wrote: > Hi, Feng > > Thanks for driving this FLIP, this proposal is very useful for catalog > man

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

x27;s better to stay consistent with `Catalog.alterDatabase`. What do you think? Best, Shammon FY On Thu, Jun 1, 2023 at 10:25 AM yuxia wrote: > Hi, Shammon. > I mean do we need to contain the new database after alter in > AlterDatabaseEvent? So that the listener can know what has been mod

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

connector such as iceberg/paimon can create sources from the `CatalogBaseTable` directly without the need to get the snapshot ID from `CatalogTable.getSnapshot()`. What do you think of it? Best, Shammon FY On Thu, Jun 1, 2023 at 7:22 AM Jing Ge wrote: > Hi Feng, > > Thanks for the

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

Hi Martijn, Thanks for your attention, I will soon initiate a discussion about FLIP-314. Best, Shammon FY On Fri, Jun 2, 2023 at 2:55 AM Martijn Visser wrote: > Hi Shammon, > > Just wanted to chip-in that I like the overall FLIP. Will be interesting to > see the follow-up discus

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

Hi devs: Thanks for all the feedback, and if there are no more comments, I will start a vote on FLIP-294 [1] later. Thanks again. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener Best, Shammon FY On Tue, Jun 6, 2023 at 1:40 PM Shammon

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

e path for running per-job/per-session jobs. The listener will be created in `TableEnvironment` and `SqlGateway` which can a `CatalogManager` with the listener. Best, Shammon FY On Tue, Jun 6, 2023 at 3:33 PM David Morávek wrote: > Hi, > > Thanks for the FLIP! Data lineage is an importan

[DISCUSS] FLIP-314: Support Customized Job Lineage Listener

+Listener [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Job+Meta+Data+Listener Best, Shammon FY

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

ication Listener" will be more suitable, I'll update the title to it. Best, Shammon FY On Thu, Jun 8, 2023 at 12:25 PM Mason Chen wrote: > Hi Shammon, > > FLIP generally looks good and I'm excited to see this feature. > > 1. I'm also curious about default imp

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

Thanks Jing, it makes sense to me and I have updated the FLIP Best, Shammon FY On Thu, Jun 8, 2023 at 11:15 PM Jing Ge wrote: > Hi Shammon, > > If we take a look at the JDK Event design as a reference, we can even add > an Object into the event [1]. Back to the CatalogModif

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

Hi all, Thanks for all the valuable feedback. If there's no further question or concern, I will start voting for FLIP-294 [1] later. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Catalog+Modification+Listener Best, Shammon FY On Fri, Jun 9, 2023

[VOTE] FLIP-294: Support Customized Catalog Modification Listener

objection or an insufficient number of votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Catalog+Modification+Listener [2] https://lists.apache.org/thread/185mbcwnpokfop4xcb22r9bgfp2m68fx Best, Shammon FY

Re: [VOTE] FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

Thanks Feng for the FLIP. +1(non-binding) Best, Shammon FY On Thu, Jun 15, 2023 at 6:30 PM Leonard Xu wrote: > Thanks Feng for driving this FLIP forward. > > +1(binding) > > Best, > Leonard > > > On Jun 15, 2023, at 1:42 PM, Dong Lin wrote: > > > &g

[RESULT][VOTE] FLIP-294: Support Customized Catalog Modification Listener

for participating! [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Catalog+Modification+Listener [2] https://lists.apache.org/thread/5f9806kw1q01536wzwgx1psh7jhmfmjw Best, Shammon FY

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

Hi devs, Is there any comment or feedback for this FLIP? Hope to hear from you, thanks Best, Shammon FY On Tue, Jun 6, 2023 at 8:22 PM Shammon FY wrote: > Hi devs, > > I would like to start a discussion on FLIP-314: Support Customized Job > Lineage Listener[1] which is the next s

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

nd update lineage for `INSERT` jobs in our streaming & batch ETL, and display the `override` information on the UI. Best, Shammon FY On Tue, Jun 20, 2023 at 6:19 PM Qingsheng Ren wrote: > Hi Shammon, > > Thanks for starting this FLIP! Data lineage is a very important topic, > w

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

next step. So we have comprehensively considered the table lineage and column lineage interfaces here, and defined these two interfaces together clearly [1] https://github.com/FangYongs/flink/commit/d4bfe57e7a5315b790e79b8acef8b11e82c9187c Best, Shammon FY On Sun, Jun 25, 2023 at 4:17 PM Y

Re: [DISCUSS] Persistent SQL Gateway

s are successfully submitted, the lifecycle is no longer managed by SQL gateway. I don't know if it covers your usage scenario. Could you describe yours for us to test and confirm? Best, Shammon FY On Tue, Jun 27, 2023 at 6:43 PM Ferenc Csaky wrote: > Hi Jark, > > In the current impl

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

IP later and please help to review it, thanks >2. it is relatively easy to obtain column lineage through Calcite MetaQuery API It's helpful if Calcite already has some column lineage in meta, I think we can discuss and give the proposal in the column lineage FLIP Best, Shammon FY On We

Re: [DISCUSS] FLIP-325: Support configuring end-to-end allowed latency

receives all `FlushEvent` from upstream with the same ID or do flush for each `FlushEvent`? Can you give more detailed proposal about that? We also have a demand for this piece, thanks Best, Shammon FY On Thu, Jun 29, 2023 at 4:35 PM Martijn Visser wrote: > Hi Dong and Yunfeng, > &g

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

think `Entity` and `Relation` are nicer for lineage, what do you think of it? Best, Shammon FY On Thu, Jun 29, 2023 at 4:21 AM Jing Ge wrote: > Hi Shammon, > > Thanks for your proposal. After reading the FLIP, I'd like to ask > some questions to make sure we are on the same p

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

Hi devs, Thanks for all the feedback. I have discussed with @QingSheng Ren off-line to confirm some questionable points in the FLIP. Thanks for his valuable inputs and I have updated the FLIP according to our discussion. Looking forward to your feedback, thanks, Best, Shammon FY On Wed, Jul

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Congratulations! Best, Shammon FY On Wed, Jul 5, 2023 at 2:38 PM Paul Lam wrote: > Congrats and cheers! > > Best, > Paul Lam > > > 2023年7月4日 18:04，Benchao Li 写道： > > > > Congratulations! > > > > Feng Jin 于2023年7月4日周二 16:17写道： > >

Re: [DISCUSS] FLIP-325: Support configuring end-to-end allowed latency

latMap/Filter Operator is not a good idea which makes data no longer flowing. Best, Shammon FY On Thu, Jul 6, 2023 at 1:54 AM Piotr Nowojski wrote: > Hi, > > Thanks for this proposal, this is a very much needed thing that should be > addressed in Flink. > > I think there is one

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

Thanks Jing, sounds good to me. I have updated the FLIP and renamed the lineage related classes to `LineageGraph`, `LineageVertex` and `LineageEdge` and keep it consistent with the job definition in Flink. Best, Shammon FY On Thu, Jul 6, 2023 at 8:25 PM Jing Ge wrote: > Hi Shammon, > &g

Re: [ANNOUNCE] Flink 1.18 Feature Freeze Extended until July 24th, 2023

Thanks for driving the release and sharing the update, looking forward to 1.18 Best, Shammon FY On Fri, Jul 7, 2023 at 10:56 AM Yun Tang wrote: > Thanks for driving this release and sharing the update on the feature > freeze extension. > > >

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

job and remove previous methods in `DataStreamSource` and `DataStreamSink`. Looking forward to your feedback, thanks. Best, Shammon FY

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

Hi devs, Thanks for all the valuable feedback. If there are no more concerns about FLIP-314[1], I would like to start voting later, thanks. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener Best, Shammon FY On Wed, Jul 12, 2023 at 11

Re: [ANNOUNCE] New Apache Flink PMC Member - Matthias Pohl

Congratulations, Matthias! On Fri, Aug 4, 2023 at 1:13 PM Samrat Deb wrote: > Congrats, Matthias! > > > On Fri, 4 Aug 2023 at 10:13 AM, Benchao Li wrote: > > > Congratulations, Matthias! > > > > Jing Ge 于2023年8月4日周五 12:35写道： > > > > > Congrats! Matthias! > > > > > > Best regards, > > > Jing >

Re: [ANNOUNCE] New Apache Flink Committer - Hong Teoh

Congratulations, Hong! Best, Shammon FY On Fri, Aug 4, 2023 at 12:33 PM Jing Ge wrote: > congrats! Hong! > > Best regards, > Jing > > On Fri, Aug 4, 2023 at 11:48 AM Qingsheng Ren wrote: > > > Congratulations and welcome aboard, Hong! > > > > Best, >

Re: [ANNOUNCE] New Apache Flink Committer - Weihua Hu

Congratulations, Weihua! Best, Shammon FY On Fri, Aug 4, 2023 at 12:33 PM Jing Ge wrote: > congrats! Weihua! > > Best regards, > Jing > > On Fri, Aug 4, 2023 at 12:20 PM Matt Wang wrote: > > > Congratulations Weihua ~ > > > > > > -- > > >

[DISCUSS] How about adding OLAP to Flink Roadmap?

Best, Shammon FY

Re: [DISCUSS] Update Flink Roadmap

for Flink? Best, Shammon FY On Mon, Aug 14, 2023 at 8:03 PM Jark Wu wrote: > Thank you everyone for helping polish the roadmap [1]. > > I think I have addressed all the comments and we have included all ongoing > parts of Flink. > Please feel free to take a last look. I'm

Re: [DISCUSS] How about adding OLAP to Flink Roadmap?

Song could you please help to add the flink olap related subsection to the doc [2] ? Thanks very much! [1] https://lists.apache.org/thread/szdr4ngrfcmo7zko4917393zbqhgw0v5 [2] https://docs.google.com/document/d/12BDiVKEsY-f7HI3suO_IxwzCmR04QcVqLarXgyJAb7c/edit Best, Shammon FY On Mon, Aug 14, 2023

Re: [DISCUSS] Update Flink Roadmap

Hi Jark, Sounds good and I would love to, thanks! I will involve you and Xingtong on the document after updating. Best, Shammon FY On Mon, Aug 14, 2023 at 10:39 PM Jark Wu wrote: > Hi Shammon, > > Sure, could you help to draft a subsection about this in the google doc? > &g

Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

are not annotated as `Public`. So do you want to annotate them as `Plublic`? If not, I think you may need to move them from `Public Interfaces` to `Proposed Changes`. Best, Shammon FY On Sat, Sep 9, 2023 at 12:11 PM Chen Zhanghao wrote: > Hi Devs, > > I would like to start a discussio

Re: [DISCUSS] Flink annotation strategy/consensus

be marked as `@Public`? I find it is difficult to determine this. Is `@PublicEvolving` really necessary? Should we directly remove `@PublicEvolving` and use `@Public` instead? I think it would be simpler. Best, Shammon FY On Mon, Sep 11, 2023 at 11:05 AM Becket Qin wrote: > Hi Jing, > >

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

er pod? I think maybe we need a script like `kubernetes-sql-gatewah.sh` to start `sql-gateway` pods with a flink image, what do you think? Best, Shammon FY On Fri, Sep 15, 2023 at 10:02 AM Xiaolong Wang wrote: > Hi, I've experiment this feature on K8S recently, here is some of my trial

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

add a Helm chart for sql-gateway, or should we need to add this feature to the flink operator? Can you help give the conclusion? Thank you very much @Gyula [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/olap_quickstart/ Best, Shammon FY On Sun, Sep 17, 2023 at 2:02 PM

Re: [DISCUSS] FLIP-360: Merging ExecutionGraphInfoStore and JobResultStore into a single component

nents into one interface such as `CompletedJobStore` may be a little strange. What do you think? Best, Shammon FY On Fri, Sep 8, 2023 at 8:08 PM Gyula Fóra wrote: > Hi Matthias! > > Thank you for the detailed proposal, overall I am in favor of making this > unification to simpl

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

existing transformations, the added lineage information will also be deleted. Please help to review it again, and If there are no more concerns about FLIP -314[1], I would like to start voting later, thanks. cc @Leonard Best, Shammon FY On Mon, Jul 17, 2023 at 3:43 PM Shammon FY wrote: > Hi d

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

Thanks for all the valuable feedback, helm charts / templates sound good to me too. It will bring a lot of convenience to the production deployment of Sql-Gateway, looking forward to this, thanks Best, Shammon FY On Wed, Sep 20, 2023 at 10:01 AM Yangze Guo wrote: > Thanks for the reply, Gy

Re: [Discuss] FLIP-362: Support minimum resource limitation

Hi, I agree that `minimum resource limitation` will bring values for flink session clusters, but for `Application Mode`, is it useful for streaming and batch jobs? Is it necessary for us to not support the application mode, rather than relying on the default value 0? Best, Shammon FY On Thu

Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

+1 for the proposal, thanks for driving. Bet, Shammon FY On Fri, Sep 22, 2023 at 12:41 PM Yangze Guo wrote: > Thanks for driving this, +1 for the proposal. > > Best, > Yangze Guo > > > On Fri, Sep 22, 2023 at 11:59 AM Lijie Wang > wrote: > > > > Hi Junrui,

Re: FW: RE: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

20:56，Leonard Xu 写道： > > > > Thanks Shammon for the informations, the comment makes the lifecycle > clearer. > > +1 > > > > > > Best, > > Leonard > > > > > >> On Sep 18, 2023, at 7:54 PM, Shammon FY wrote: > >> > >&g

[VOTE] FLIP-314: Support Customized Job Lineage Listener

/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener [2] https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc Best, Shammon FY

Re: [VOTE] FLIP-362: Support minimum resource limitation

+1(binding), thanks for the proposal. Best, Shammon FY On Mon, Sep 25, 2023 at 11:20 PM Jing Ge wrote: > +1(binding) Thanks! > > Best regards, > Jing > > On Mon, Sep 25, 2023 at 3:48 AM Chen Zhanghao > wrote: > > > Thanks for driving this. +1 (non-binding) &

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

, Shammon FY On Thu, Sep 28, 2023 at 12:08 PM Rui Fan <1996fan...@gmail.com> wrote: > Hi Zhu Zhu, > > Thanks for your feedback here! > > You are right, user needs to set 2 options: > - cluster.evenly-spread-out-slots=true > - slot.sharing-strategy=TASK_BALANCED_PRE

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

ch as ANY/SLOTS for RM and ANY/TASKS for slot pool? I want this because we would like to add a new slot to TM strategy such as SLOTS_NUM in the future for OLAP to improve the performance for olap jobs, which will use TASKS strategy for task to slot. cc Guoyangze Best, Shammon FY On Fri, Oct 6, 2023

[RESULT][VOTE] FLIP-314: Support Customized Job Lineage Listener

(binding) - 曹帝胄 - Chen Zhanghao - Rui Fan(binding) - Yuepeng Pan There is no disapproving vote. Best, Shammon FY [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener [2] https://lists.apache.org/thread/dxdqjc0dd8rf1vbdg755zo1n2zj1tj8d

Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources

+1 (binding) On Mon, Oct 9, 2023 at 11:12 AM Benchao Li wrote: > +1 (binding) > > Zhanghao Chen 于2023年10月9日周一 10:20写道： > > > > Hi All, > > > > Thanks for all the feedback on FLIP-367: Support Setting Parallelism for > Table/SQL Sources [1][2]. > > > > I'd like to start a vote for FLIP-367. The

Re: [VOTE] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

+1(binding), good job! Best, Shammon FY On Wed, Oct 11, 2023 at 10:18 AM Benchao Li wrote: > +1 (binding) > > Rui Fan <1996fan...@gmail.com> 于2023年10月11日周三 10:17写道： > > > > +1(binding) > > > > Best, > > Rui > > > > On Wed, Oct 11,

[DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Hi devs: I'd like to start a discussion about FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store[1]. In the whole data stream processing, there are consistency problems such as how to manage the dependencies of multiple jobs and tables, how to define and handle E2E dela

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

https://cwiki.apache.org/confluence/display/FLINK/FLIP-182%3A+Support+watermark+alignment+of+FLIP-27+Sources > > 5. Given the MetaService role, it feels like this is introducing a tight > dependency between Flink and the Table Store. How pluggable is this > solution, given the cha

Re: [DISCUSS] FLIP-275: Support Remote SQL Client Based on SQL Gateway

Hi zelin Thanks for driving this discussion. I notice that the sql-client will interact with sql-gateway by `REST Client` in the `Executor` in the FLIP, how about introducing jdbc-sdk for sql-gateway? Then the sql-client can connect the gateway with jdbc-sdk, on the other hand, the other applica

[DISCUSSION] Scan mode in Table Store for Flink Stream and Batch job

Hi devs: I'm an engineer from ByteDance, and here I'd link to discuss "Scan mode in Table Store for Flink Stream and Batch job". Users can execute Flink Steam and Batch jobs on Table Store. In Table Store 0.2 there're two items which determine how the Stream and Batch jobs' sources read data: Sta

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

of the system after adding > the global checkpoint, and additional (single?) point of failure. > 5. Such a design would also not work if we ever wanted to have task local > checkpoints. > > All in all, it seems to me like actually the watermarks and even time are > the better conce

Re: [DISCUSSION] Scan mode in Table Store for Flink Stream and Batch job

atest-full`, >> `compacted-full`. Regardless of the runtime-mode, the two choices >> force full range to read. >> >> I think solution 3 is a compromise solution. It can also ensure the >> availability of default values. Conceptually, it can at least explain >

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

> cross-pipeline barrier alignment, which is the mechanism a subset of > currently supported checkpointing implementations happen to be using. Is > that correct? > > My biggest concern is that tying this with a "side-effect" of the > checkpointing mechanism could block

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

as to produce some data, downstream can not process it, > downstream can not process this data yet > 2. checkpoint 42 is triggered on the upstream job > 3. checkpoint 42 is completed on the upstream job, data processed since > last checkpoint has been committed > 4. upstream job can con

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

point of time already fixed > order), they are committed and will never change. > > > However, from the perspective of implementation complexity, I personally > think using Checkpoint in the first phase makes sense, what do you think? > > Maybe I'm missing something, but I

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

always check what are the committed watermarks, > select the watermark value (for example min across all used tables), and > ask every table: please give me all of the data up until the selected > watermark. Or give me all tables in the version for the selected watermark. > >

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

wid Wysakowicz about this > subject. We have only briefly chatted and exchanged some thoughts/ideas, > but unfortunately we were not able to finish the discussions before the > holiday season/vacations. Can we get back to this topic in January? > > Best, > Piotrek > > pt., 16

Re: flink add multiple sink in sequence

Hi @Great I think the two sinks in your example are equivalent and independent. If there are some logical relationships between two sinks, you may need to create a new combined sink and do it yourself. On Thu, Jan 5, 2023 at 11:48 PM Great Info wrote: > > I have a stream from Kafka, after readi

Re: [DISCUSS] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

Hi @Jing Ge Thanks for the proposal and sorry for joining the discussion late. I met the same problem when I performed a batch query on sql-client and it could not print the whole line of results. I think the FLIP is nice, just one comment: Can users set the parameter dynamically on sql-client wi

Re: [VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

+1 (no-binding) Best, Shammon On Thu, Jan 12, 2023 at 8:11 PM Shengkai Fang wrote: > +1(binding) > > Best, > Shengkai > > Jark Wu 于2023年1月12日周四 19:22写道： > > > +1 (binding) > > Thank you for driving this effort. > > > > Best, > > Jark > > > > > 2023年1月9日 15:46，Jing Ge 写道： > > > > > > Hi, > >

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

20, 2022 at 10:01 AM Shammon FY wrote: > Hi Piotr, > > Thanks for your syncing. I will update the FLIP later and keep this > discussion open. Looking forward to your feedback, thanks > > > Best, > Shammon > > > On Mon, Dec 19, 2022 at 10:45 PM Piotr Nowojski > wr

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

porting auditability use cases. To that latter point, it seems nice to > provide not only a mechanism to query arbitrary snapshots, but also to > define a TTL/GC model that allows users to keep hourly snapshots for N > hours, daily snapshots for N days, weekly snapshots for N weeks, and th

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

opportunity to maintain a manifest of historical snapshots > and > > allow people to query as of old points in time. That can be valuable for > > individuals answering data questions, building products, and crucially > > supporting auditability use cases. To that latter point, it

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

tore > > in the first phase, and may consider other storage in the future > > I'm not sure if I follow. Generally speaking, why do we need MetaService at > all? Why can we only support writes to and reads from TableStore, and not > any source/sink that implements some specifi

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

Hi David Thanks for initiating this discussion. I think declaring job resource requirements by REST API is very valuable. I just left some comments as followed 1) How about adding some more information such as vertex type (SOURCE/MAP/JOIN and .etc) in the response of `get jobs resource-requiremen

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

er > themselves in `MetaService`, > > it needs to serially check whether they write to the same table > > Why do we need to do that? Only to disallow this? To forbid writing from > two jobs into a single table? If so, can we not push this responsibility > down to the connector? Li

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

uestion above). > > > Thanks, everyone, for your initial thoughts! > > Best, > D. > > On Tue, Feb 7, 2023 at 4:39 AM weijie guo > wrote: > > > Thanks David for driving this. This is a very valuable work, especially > for > > cloud native environment. >

Re: [ANNOUNCE] New Apache Flink Committer - Weijie Guo

Congratulations, Weijie! Best, Shammon On Mon, Feb 13, 2023 at 12:03 PM Geng Biao wrote: > Congrats, weijie! > Best, > Biao Geng > > 获取 Outlook for iOS > > 发件人: Lijie Wang > 发送时间: Monday, February 13, 2023 11:55:56 AM > 收件人: dev@flink.ap

Re: [ANNOUNCE] New Apache Flink Committer - Jing Ge

Congratulations Jing! Best, Shammon On Tue, Feb 14, 2023 at 4:00 PM weijie guo wrote: > Congratulations Jing! > > Best regards, > > Weijie > > > Jingsong Li 于2023年2月14日周二 15:54写道： > > > Congratulations Jing! > > > > Best, > > Jingsong > > > > On Tue, Feb 14, 2023 at 3:50 PM godfrey he wrote:

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

nt in this thread if you have any questions later. THX Best, Shammon On Tue, Feb 7, 2023 at 11:53 AM Shammon FY wrote: > Hi Piotr > > Thanks for your feedback. > > > - stateless operators, could completely ignore the issue and process the > records normally, as they are doin

Re: [ANNOUNCE] New Apache Flink PMC Member - Dong Lin

Congratulations, Dong! Best, Shammon On Thu, Feb 16, 2023 at 4:27 PM weijie guo wrote: > Congratulations Dong! Well deserved~ > > Best regards, > > Weijie > > > Samrat Deb 于2023年2月16日周四 16:12写道： > > > Congratulations , Dong! > > > > On Thu, Feb 16, 2023 at 1:39 PM Jing Ge > > wrote: > > > >

Re: [DISCUSS] Extract core autoscaling algorithm as new SubModule in flink-kubernetes-operator

Hi Samrat Do you mean to create an independent module for flink scaling in flink-k8s-operator? How about creating a project such as `flink-auto-scaling` which is completely independent? Besides resource managers such as k8s and yarn, we can do more things in the project, for example, updating conf

Re: [DISCUSS] Extract core autoscaling algorithm as new SubModule in flink-kubernetes-operator

gt; > > On Thu, Feb 16, 2023 at 7:37 PM Samrat Deb > wrote: > > > > > Hi Max , > > > If you are fine and aligned with the same thought , since this is going > > to > > > be very useful to us, we are ready to help / contribute additional work > > > requ

Re: [DISCUSS] Extract core autoscaling algorithm as new SubModule in flink-kubernetes-operator

the two core > > > requirements, but the operator itself > > > cannot be used for Flink jobs running on YARN. Those core requirements > > are > > > solved through other mechanisms in the case of YARN. But the main > problem > > > for us is *how to* > &g

Re: [ANNOUNCE] New Apache Flink Committer - Anton Kalashnikov

Congratulations, Anton! Best, Shammon On Tue, Feb 21, 2023 at 1:41 PM Sergey Nuyanzin wrote: > Congratulations, Anton! > > On Tue, Feb 21, 2023 at 4:53 AM Weihua Hu wrote: > > > Congratulations, Anton! > > > > Best, > > Weihua > > > > > > On Tue, Feb 21, 2023 at 11:22 AM weijie guo > > wrote:

Re: [ANNOUNCE] New Apache Flink Committer - Rui Fan

Congratulations, Rui! Best, Shammon On Tue, Feb 21, 2023 at 1:40 PM Sergey Nuyanzin wrote: > Congratulations, Rui! > > On Tue, Feb 21, 2023 at 4:53 AM Weihua Hu wrote: > > > Congratulations, Rui! > > > > Best, > > Weihua > > > > > > On Tue, Feb 21, 2023 at 11:28 AM Biao Geng wrote: > > > > >

Re: Proposal to improve the Apache Flink website

Hi +1 too for this proposal Compared with the beginning, Flink has good applications in many directions now. The new website can give users better guidance for streaming/batch/olap/ML and so on. Looking forward to the new website :) Best, Shammon On Tue, Feb 21, 2023 at 2:24 PM Gyula Fóra wr

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

Hi kui Thanks for initializing this discussion. I have two questions 1. What will happen if we add watermark related options in `the connector options`? Will the connector ignore these options or throw an exception? How can we support this? 2. If one table is used by two operators with different

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

Hi kui Thanks for your answer and +1 to yuxia too > we should not bind the watermark-related options to a connector to ensure semantic clarity. In my opinion, adding watermark-related options to a connector is much more clear. Currently users can define simple watermark strategy in DDL, adding m

Re: [Discuss] Some questions on flink-table-store micro benchmark

Hi jingsong Getting rid of JMH is a good idea. For the second point, how can we track the performance changes of the micro benchmark? What do you think? Best, Shammon On Mon, Feb 27, 2023 at 10:57 AM Jingsong Li wrote: > Thanks Yun. > > Another way is we can get rid of JMH, something like Spar

[DISCUSS] Deprecate deserialize method in DeserializationSchema

Hi devs Currently there are two deserialization methods in `DeserializationSchema` 1. `T deserialize(byte[] message)`, only deserialize one record from binary, if there is no record it should return null. 2. `void deserialize(byte[] message, Collector out)`, supports deserializing none, one or mul

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

Hi Thanks for starting this work weihua, I think unifying DeclarativeSlotManager and FineGrainedSlotManager is valuable. I agree with @Matthias and @John that we need a way to ensure that DeclarativeSlotManager's capabilities are fully covered by FineGrainedSlotManager 1. For their functional di

[DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

Hi devs, I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway[1]. FLIP-275[2] supports remote sql client based on gateway, users can interact with gateway by flink console. However, for users who create session clusters with Flink, they'd like to use Jdbc Dri

Re: [VOTE] FLIP-291: Externalized Declarative Resource Management

+1 (non-binding) Best, Shammon On Wed, Mar 1, 2023 at 4:51 PM Roman Khachatryan wrote: > +1 (binding) > > Thanks David, and everyone involved :) > > Regards, > Roman > > > On Wed, Mar 1, 2023 at 8:01 AM Gyula Fóra wrote: > > > +1 (binding) > > > > Looking forward to this :) > > > > Gyula > >

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

r `Public Interface` part, only > > `FlinkResultSet.getRowKind` is a true new interface, others are just > > implementations. > > > > If the user does not cast into a FlinkResultSet, will there be serious > > consequences here (RowKind is ignored)? > > > > Best,

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

integration for JDBC and streaming... > > Do you need to use JDBC in streaming mode? Or do we just implement > JDBC for batch only first? > > Best, > Jingsong > > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY wrote: > > > > Hi > > > > Thanks for the

  1   2   >