[DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-03 Thread David Morávek
Hi everyone, This FLIP [1] introduces a new REST API for declaring resource requirements for the Adaptive Scheduler. There seems to be a clear need for this API based on the discussion on the "Reworking the Rescale API" [2] thread. Before we get started, this work is heavily based on the prototyp

Re: [DISCUSS] FLIP-278: Hybrid Source Connector

2023-02-03 Thread Ran Tao
Hi, Martijn. i have updated the flip about table api & switched start timestamp. thanks. Martijn Visser 于2022年12月16日周五 16:59写道: > Hi Ran, > > For completeness, this is a new thread that was already previously started > at https://lists.apache.org/thread/xptn2ddzj34q9f5vtbfb62lsybmvcwjq. I'm > li

Re: [DISCUSS] FLIP-278: Hybrid Source Connector

2023-02-03 Thread Ran Tao
Hi, all. i have updated flip-278[1]. I think all problems or comments has been addressed. 1.about option prefix, we use identifiers. 2.table api implementation and demo 3.about switched dynamic position (hybrid source use it auto switch from previous to next source) More details can be found at d

Re: Reworking the Rescale API

2023-02-03 Thread Chesnay Schepler
My primary concern here is to be able to rescale with upfront reservation of resources before restarting the job, so the job doesn't get stuck in case of resource constraints. Not sure I follow. The AS only rescales when it has already acquired the slots that it needs. > This is a blocker fr

[jira] [Created] (FLINK-30894) Introduce Serializer to serialize internal data structure

2023-02-03 Thread Feng Wang (Jira)
Feng Wang created FLINK-30894: - Summary: Introduce Serializer to serialize internal data structure Key: FLINK-30894 URL: https://issues.apache.org/jira/browse/FLINK-30894 Project: Flink Issue Typ

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-03 Thread ConradJam
Hi David: Thank you for drive this flip, which helps less flink shutdown time for this flip, I would like to make a few idea on share - when the number of "slots" is insufficient, can we can stop users rescaling or throw something to tell user "less avaliable slots to upgrade, please c

[jira] [Created] (FLINK-30895) SlotSharingSlotAllocator may waste slots

2023-02-03 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-30895: Summary: SlotSharingSlotAllocator may waste slots Key: FLINK-30895 URL: https://issues.apache.org/jira/browse/FLINK-30895 Project: Flink Issue Type:

Re: Need help how to use Table API to join two Kafka streams

2023-02-03 Thread yuxia
Hi, Amir. May look like using scala code: val t1 = tableEnv.executeSql("CREATE TEMPORARY TABLE s1 (id int, ssn string) WITH ('connector' = 'kafka', ...); val t2 = tableEnv.executeSql("CREATE TEMPORARY TABLE s2 (id int, ssn string) WITH ('connector' = 'kafka', ...); // you will need to rename th

[jira] [Created] (FLINK-30896) Reduce usage of CatalogViewImpl in planner

2023-02-03 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-30896: --- Summary: Reduce usage of CatalogViewImpl in planner Key: FLINK-30896 URL: https://issues.apache.org/jira/browse/FLINK-30896 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-03 Thread Matthias Pohl
Thanks David for creating this FLIP. It sounds promising and useful to have. Here are some thoughts from my side (some of them might be rather a follow-up and not necessarily part of this FLIP): - I'm wondering whether it makes sense to add some kind of resource ID to the REST API. This would give

[jira] [Created] (FLINK-30897) Avoid timeouts in JUnit tests

2023-02-03 Thread lincoln lee (Jira)
lincoln lee created FLINK-30897: --- Summary: Avoid timeouts in JUnit tests Key: FLINK-30897 URL: https://issues.apache.org/jira/browse/FLINK-30897 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-30898) Do not include and build examples in operator image

2023-02-03 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-30898: -- Summary: Do not include and build examples in operator image Key: FLINK-30898 URL: https://issues.apache.org/jira/browse/FLINK-30898 Project: Flink Issue Type: N

Re: [VOTE] FLIP-285: Refactoring LeaderElection to make Flink support multi-component leader election out-of-the-box

2023-02-03 Thread David Morávek
Thanks for the detailed FLIP, Matthias; this will simplify the HA code-base significantly. +1 (binding) Best, D. On Tue, Jan 31, 2023 at 5:22 AM Yang Wang wrote: > +1 (Binding) > > Best, > Yang > > ConradJam 于2023年1月31日周二 12:09写道: > > > +1 non-binding > > > > Matthias Pohl 于2023年1月25日周三 17:3

Re: [VOTE] FLIP-285: Refactoring LeaderElection to make Flink support multi-component leader election out-of-the-box

2023-02-03 Thread Matthias Pohl
Thanks for your participation. I'm closing this vote and will announce the results in a separate email. Matthias On Fri, Feb 3, 2023 at 3:26 PM David Morávek wrote: > Thanks for the detailed FLIP, Matthias; this will simplify the HA code-base > significantly. > > +1 (binding) > > Best, > D. > >

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2023-02-03 Thread Piotr Nowojski
Hi Shammon, Thanks for pushing the topic further. I'm not sure how this new proposal is supposed to be working? How should timestamp barrier interplay with event time and watermarks? Or is timestamp barrier supposed to completely replace watermarks? > stateful and temporal operators should align

[RESULT][VOTE] FLIP-285: Refactoring LeaderElection to make Flink support multi-component leader election out-of-the-box

2023-02-03 Thread Matthias Pohl
The vote on FLIP-285 is done. The proposal is approved in [1] after the discussion happened in [2]. There are 6 approving votes, 3 of which are binding: * Chesnay Schepler (binding) * Rui Fan (non-binding) * Weijie Guo (non-binding) * Conrad Jam (non-binding) * Yang Wang (binding) * David Morávek

Re: Need help how to use Table API to join two Kafka streams

2023-02-03 Thread Amir Hossein Sharifzadeh
Thank you very much, Yuxia! ssn stands for the social security number here (it was just an example but it can be any field). Best, Amir On Fri, Feb 3, 2023 at 5:45 AM yuxia wrote: > Hi, Amir. > May look like using scala code: > > val t1 = tableEnv.executeSql("CREATE TEMPORARY TABLE s1 (id int

Re: [DISCUSS] Incorporate performance regression monitoring into routine process

2023-02-03 Thread David Anderson
+1 I don't have anything substantive to add, but I want to express how pleased I am to see this conversation happening. David On Thu, Feb 2, 2023 at 5:09 AM Martijn Visser wrote: > Hi all, > > +1 for the overall proposal. My feedback matches with what Matthias > has already provided earlier. >

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

2023-02-03 Thread Konstantin Knauf
Hi everyone, if I am not mistaken of the sinks mentioned by Joao Kafka, Kinesis & Kinesis already use the Sink2 API. How were those implemented without exposing the ExecutionConfig? Best, Konstantin Am Mi., 1. Feb. 2023 um 12:28 Uhr schrieb Lijie Wang < wangdachui9...@gmail.com>: > +1 for Opt

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

2023-02-03 Thread Konstantin Knauf
Hi everyone, sorry for joining the discussion late. 1) Is there an option to deprecate SinkFunction in Flink 1.17 while leaving SinkV2 @PublicEvolving in Flink 1.17. We then aim to make SinkV2 @Public in and remove SinkFunction in Flink 1.18. @PublicEvolving are intended for public use. So, I don