Re: [DISCUSS] Planning Flink 2.0

2023-04-28 Thread John Roesler
Hi all, Great discussion! For what it's worth, my experience has been that Semantic Versioning is really the best way to think about major releases. It can occasionally be nice to have a big "marketing release" to celebrate a major improvement, but coupling the concept of big improvements to m

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-17 Thread John Roesler
Hi Becket, Thanks for this FLIP! Having a deprecation process is really important. I understand some people’s concerns about the additional burden for project maintainers, but my personal experience with Kafka has been that it’s very liveable and that it’s well worth the benefit to users. In fa

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-18 Thread John Roesler
start with the current status. Hopefully after we are > more comfortable with the maintenance overhead of deprecated APIs, we can > then have a stronger guarantee for Experimental / PublicEvolving APIs. > > Thanks, > > Jiangjie (Becket) Qin > > > > On Sun, Jun 18, 2023 at

Re: [VOTE] FLIP-287: Extend Sink#InitContext to expose TypeSerializer, ObjectReuse and JobID

2023-06-19 Thread John Roesler
Thanks for the FLIP, Joao! I'm +1 (non-binding) -John On 2023/06/19 15:54:49 Leonard Xu wrote: > +1(binding) > > Best, > Leonard > > > On Jun 19, 2023, at 8:12 PM, Martijn Visser > > wrote: > > > > +1 (binding) > > > > On Mon, Jun 19, 2023 at 4:58 AM Yuepeng Pan wrote: > > > >> +1 (non-b

Re: [VOTE] FLIP-308: Support Time Travel

2023-06-19 Thread John Roesler
Thanks for this FLIP, Feng! I've been following along, and I'm +1 (non-binding). Thanks, -John On 2023/06/19 15:50:48 Leonard Xu wrote: > +1(binding) > > Best, > Leonard > > > On Jun 19, 2023, at 8:53 PM, Jing Ge wrote: > > > > +1(binding) > > > > On Mon, Jun 19, 2023 at 1:57 PM Benchao Li

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-19 Thread John Roesler
> > > > > > I see your point, that users would feel surprised if they find things > > no > > > > longer work when upgrading to another 2.x minor release. However, I'd > > > like > > > > to point out that PublicEvolving APIs would have

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-20 Thread John Roesler
o explain this itself is an > indicator that it might be confusing for users. > > > Becket, on the other hand, prefers option 2. From my understanding, his > major point is that a quick major version bump causes barely any actual > lose on users, while in option 1 not providi

Re: [VOTE] FLIP-321: introduce an API deprecation process

2023-07-06 Thread John Roesler
+1 (non-binding) Thanks for the FLIP! -John On Mon, Jul 3, 2023, at 22:21, Jingsong Li wrote: > +1 binding > > On Tue, Jul 4, 2023 at 10:40 AM Zhu Zhu wrote: >> >> +1 (binding) >> >> Thanks, >> Zhu >> >> ConradJam 于2023年7月3日周一 22:39写道: >> > >> > +1 (no-binding) >> > >> > Matthias Pohl 于2023年7月

Re: [DISCUSS] FLIP-278: Hybrid Source Connector

2022-12-18 Thread John Roesler
Hello all, Thanks for the FLIP, Ran! The HybridSource is a really cool feature, and I was glad to see a proposal to expose it in the Table and SQL APIs. My main question is also about the switching control (question 2). It seems like the existing Kafka connector has all the options you'd want

Re: [DISCUSS] Allow source readers extending SourceReaderBase to override numRecordsIn report logic

2023-01-04 Thread John Roesler
Hi Wencong, Thanks for the proposal! I agree that we should fix this disconnect between the metric and what is actually happening in those source connectors. My instincts agree with Dong's. Adding a configuration option in order to tune the relationship between a superclass and a subclass seems

Re: [DISCUSS] Allow source readers extending SourceReaderBase to override numRecordsIn report logic

2023-01-06 Thread John Roesler
Thanks for the replies, Dong and Wencong! That’s a good point about the overhead of the extra method. Is the desire to actually deprecate that metric in a user-facing way, or just to deprecate the private counter mechanism? It seems like if the desire is to deprecate the existing private counte

Re: [DISCUSS] Allow source readers extending SourceReaderBase to override numRecordsIn report logic

2023-01-07 Thread John Roesler
ill not increment the counter if a subclass uses the newly added > constructor. > > Cheers, > Dong > > > On Sat, Jan 7, 2023 at 4:47 AM John Roesler wrote: > >> Thanks for the replies, Dong and Wencong! >> >> That’s a good point about the overhead of the extra metho

Re: [DISCUSS] Enabling dynamic partition discovery by default in Kafka source

2023-01-13 Thread John Roesler
Thanks for this proposal, Qingsheng! If you want to be a little conservative with the default, 5 minutes might be better than 30 seconds. The equivalent config in Kafka seems to be metadata.max.age.ms (https://kafka.apache.org/documentation/#consumerconfigs_metadata.max.age.ms), which has a de

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2023-01-27 Thread John Roesler
Hello Shammon and all, Thanks for this FLIP! I've been working toward this kind of global consistency across large scale data infrastructure for a long time, and it's fantastic to see a high-profile effort like this come into play. I have been lurking in the discussion for a while and delaying

Re: [ANNOUNCE] New Apache Flink Committer - Anton Kalashnikov

2023-02-20 Thread John Roesler
Congratulations, Anton! -John On Mon, Feb 20, 2023, at 08:18, Piotr Nowojski wrote: > Hi, everyone > > On behalf of the PMC, I'm very happy to announce Anton Kalashnikov as a new > Flink Committer. > > Anton has been very active for almost two years already, authored and > reviewed many PRs over t

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-20 Thread John Roesler
Thanks for the FLIP, David! I just had one small question. IIUC, the REST API PUT request will go through the new DispatcherGateway method to be handled. Then, after validation, the dispatcher would call the new JobMasterGateway method to actually update the job. Which component will write th

Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements

2023-02-24 Thread John Roesler
Hello Ran, Thanks for the FLIP! Do you mind if we revisit the topic of doing this by adding an Information schema? The SHOW approach requires modifying the parser/language for every gap we identify. On the flip side, an Information schema is much easier to discover and remember how to use, and

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

2023-02-28 Thread John Roesler
b2c4975d25b05eb4161617af0d704a85ff7b1cad19d3c817c12f1e29cR1151 > > Best, > D. > > On Tue, Feb 21, 2023 at 12:24 AM John Roesler wrote: > >> Thanks for the FLIP, David! >> >> I just had one small question. IIUC, the REST API PUT request will go >> through the

Re: [VOTE] FLIP-291: Externalized Declarative Resource Management

2023-02-28 Thread John Roesler
Thanks for the FLIP, David! I’m +1 (non-binding) -John On Tue, Feb 28, 2023, at 07:46, David Morávek wrote: > Hi Everyone, > > I want to start the vote on FLIP-291: Externalized Declarative Resource > Management [1]. The FLIP was discussed in this thread [2]. > > The goal of the FLIP is to enabl

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

2023-02-28 Thread John Roesler
Thanks for the FLIP, Weihua! I’ve read the FLIP, and it sounds good to me. We need to avoid proliferating alternative implementations wherever possible. I have just a couple of comments: 1. I share Matthias’s concern about ensuring the behavior is really the same. One suggestion I’ve used for t

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

2023-03-01 Thread John Roesler
gt;> > 1. For their functional differences, can you give some detailed tests to >> > verify that the new FineGrainedSlotManager has these capabilities? This >> can >> > effectively verify the new functions >> > >> > 2. I'm worried that many functions

Re: [Vote] FLIP-298: Unifying the Implementation of SlotManager

2023-03-09 Thread John Roesler
Thanks, Weihua! I’m +1 (non-binding) -John On Thu, Mar 9, 2023, at 21:02, Shammon FY wrote: > Thanks weihua, +1 (non-binding) > > Best, > Shammon > > On Fri, Mar 10, 2023 at 10:32 AM Xintong Song wrote: > >> +1 (binding) >> >> Best, >> >> Xintong >> >> >> >> On Thu, Mar 9, 2023 at 1:28 PM Weihu

Re: [DISCUSS] EXACTLY_ONCE delivery semantics for upsert-kafka connector

2023-04-11 Thread John Roesler
Hi Jark, I hope you don’t mind if I chime in. You have a good point that the sequence of upserts will eventually converge to the correct value under the at-least-once delivery guarantee, but it can still be important to avoid passing on uncommitted results. Some thoughts, numbered for referenc

Re: [DISCUSS] Allow sharing (RocksDB) memory between slots

2022-11-09 Thread John Roesler
Hi Roman, Thanks for the proposal! This will make scheduling a lot more flexible for our use case. Just a quick follow-up question about the number of new configs we’re adding here. It seems like the proposed configs provide a lot of flexibility, but at the expense of added complexity. It see

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-24 Thread John Roesler
Hi Max, Thanks for the FLIP! I’ve been curious about one one point. I can imagine some good reasons for it but wonder what you have in mind. What’s the reason to add auto scaling to the Operator instead of to the JobManager? It seems like adding that capability to the JobManager would be a big

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-24 Thread John Roesler
to integrate with user job upgrades etc. > > These are some of the main things that come to my mind :) > > Having it in the operator ties this logic to Kubernetes itself but we feel > that an autoscaler is mostly relevant in an elastic cloud environment > anyways. > > Cheers,

[jira] [Created] (FLINK-29962) Exclude Jamon 2.3.1

2022-11-09 Thread John Roesler (Jira)
John Roesler created FLINK-29962: Summary: Exclude Jamon 2.3.1 Key: FLINK-29962 URL: https://issues.apache.org/jira/browse/FLINK-29962 Project: Flink Issue Type: Improvement