[DISCUSS] Looking for maintainers for Cassandra connector or decide to remove connector

2022-01-21 Thread Martijn Visser
Hi everyone, We're looking for community members, who would like to maintain Flink's Cassandra connector [1] going forward. The connector currently is only available as a Sink for DataStream users and the original contributors are unable to work on further improvements. An overview of some of the

[jira] [Created] (FLINK-25744) Support native savepoints (w/o modifying the statebackend specific snapshot strategies)

2022-01-21 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-25744: -- Summary: Support native savepoints (w/o modifying the statebackend specific snapshot strategies) Key: FLINK-25744 URL: https://issues.apache.org/jira/browse/FLINK-25744

[jira] [Created] (FLINK-25745) Support RocksDB incremental native savepoints

2022-01-21 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-25745: -- Summary: Support RocksDB incremental native savepoints Key: FLINK-25745 URL: https://issues.apache.org/jira/browse/FLINK-25745 Project: Flink Issue Type:

Re: Flink native k8s integration vs. operator

2022-01-21 Thread Alexis Sarda-Espinosa
Hi Robert, I agree with you, I mean that's why I was writing a K8s operator, but the restriction wasn't decided by me, it was imposed on me. I guess my thinking was rather that an operator wouldn't necessarily supersede standalone+reactive, at least not in my case, but that certainly doesn't me

Developing multiple Flink Pieplines

2022-01-21 Thread Hussein El Ghoul
Dear Community, I hope this email finds you well. I am a beginner and I had some points that needed clarification. I am working on multiple flink pipelines, and need to deploy those pipelines on kubernetes but I am not sure if I should deploy these pipelines in one repository (one application

[DISCUSS] Looking for maintainers for NiFi connector or decide to remove connector

2022-01-21 Thread Martijn Visser
Hi everyone, We're looking for community members, who would like to maintain Flink's NiFi connector [1] going forward. The connector currently is available as a Source and Sink for DataStream users and it hasn't been maintained in the past couple of years. An overview of some of the things that a

[DISCUSS] Deprecate/remove Twitter connector

2022-01-21 Thread Martijn Visser
Hi everyone, I would like to discuss deprecating Flinks' Twitter connector [1]. This was one of the first connectors that was added to Flink, which could be used to access the tweets from Twitter. Given the evolution of Flink over Twitter, I don't think that: * Users are still using this connecto

Re: [DISCUSS] Future of Per-Job Mode

2022-01-21 Thread Konstantin Knauf
Thanks Thomas & Biao for your feedback. Any additional opinions on how we should proceed with per job-mode? As you might have guessed, I am leaning towards proposing to deprecate per-job mode. On Thu, Jan 13, 2022 at 5:11 PM Thomas Weise wrote: > Regarding session mode: > > ## Session Mode > *

[jira] [Created] (FLINK-25746) Failed to run ITCase locally with IDEA under flink-orc and flink-parquet module

2022-01-21 Thread Jane Chan (Jira)
Jane Chan created FLINK-25746: - Summary: Failed to run ITCase locally with IDEA under flink-orc and flink-parquet module Key: FLINK-25746 URL: https://issues.apache.org/jira/browse/FLINK-25746 Project: Fl

Re: [VOTE] FLIP-201: Persist local state in working directory

2022-01-21 Thread Till Rohrmann
Thanks a lot for your votes. I am closing the vote now and will post the result as a separate email thread. Cheers, Till On Fri, Jan 21, 2022 at 5:06 AM Yang Wang wrote: > +1 (binding) > > This would be useful especially for K8s users. > > Best, > Yang > > David Morávek 于2022年1月20日周四 18:19写道:

[RESULT][VOTE] FLIP-201: Persist local state in working directory

2022-01-21 Thread Till Rohrmann
Hi everyone, I am happy to announce that FLIP-201 [1] has been accepted by this vote [2]. * Etienne Chauchot (non-binding) * Till Rohrmann (binding) * Matthias Pohl (binding) * Chesnay Schepler (binding) * David Moravek (non-binding) * Yang Wang (binding) [1] https://cwiki.apache.org/confluence/

Re: [DISCUSS] FLIP-203: Incremental savepoints

2022-01-21 Thread David Morávek
As per offline discussion with Piotr: we have some ways to deal with possible incompatibilities (hiding them behind a feature flag, new state backend implementation). Having faster adoption of the new Flink releases is more valuable in this context, than a possible overhead on the implementation si

[jira] [Created] (FLINK-25747) UdfStreamOperatorCheckpointingITCase hangs on AZP

2022-01-21 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-25747: - Summary: UdfStreamOperatorCheckpointingITCase hangs on AZP Key: FLINK-25747 URL: https://issues.apache.org/jira/browse/FLINK-25747 Project: Flink Issue Typ

[jira] [Created] (FLINK-25748) Website misses some Repositories

2022-01-21 Thread Konstantin Knauf (Jira)
Konstantin Knauf created FLINK-25748: Summary: Website misses some Repositories Key: FLINK-25748 URL: https://issues.apache.org/jira/browse/FLINK-25748 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-25749) YARNSessionFIFOSecuredITCase.testDetachedMode fails on AZP

2022-01-21 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-25749: - Summary: YARNSessionFIFOSecuredITCase.testDetachedMode fails on AZP Key: FLINK-25749 URL: https://issues.apache.org/jira/browse/FLINK-25749 Project: Flink

[jira] [Created] (FLINK-25750) Performance regression on 20.01.2021 in globalWindow and stateBackend benchmarks

2022-01-21 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-25750: - Summary: Performance regression on 20.01.2021 in globalWindow and stateBackend benchmarks Key: FLINK-25750 URL: https://issues.apache.org/jira/browse/FLINK-25750

Re: [VOTE] FLIP-205: Support cache in DataStream for Batch Processing

2022-01-21 Thread Dian Fu
+1 (binding) Thanks for driving this. Looking forward to this feature. Regards, Dian On Thu, Jan 20, 2022 at 5:57 PM Gen Luo wrote: > +1 (non-binding) > > On Thu, Jan 20, 2022 at 3:26 PM Yun Gao > wrote: > > > +1 (binding) > > > > Thanks Xuannan for driving this! > > > > Best, > > Yun > > > >

[jira] [Created] (FLINK-25751) flink-table tests exit with code 137

2022-01-21 Thread Niklas Semmler (Jira)
Niklas Semmler created FLINK-25751: -- Summary: flink-table tests exit with code 137 Key: FLINK-25751 URL: https://issues.apache.org/jira/browse/FLINK-25751 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-25752) Create a migration e2e test

2022-01-21 Thread Igal Shilman (Jira)
Igal Shilman created FLINK-25752: Summary: Create a migration e2e test Key: FLINK-25752 URL: https://issues.apache.org/jira/browse/FLINK-25752 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-25753) KafkaTableITCase.testStartFromGroupOffsetsLatest fails on AZP

2022-01-21 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-25753: - Summary: KafkaTableITCase.testStartFromGroupOffsetsLatest fails on AZP Key: FLINK-25753 URL: https://issues.apache.org/jira/browse/FLINK-25753 Project: Flink

[jira] [Created] (FLINK-25754) Drop unused private class org.apache.flink.connector.elasticsearch.table.KeyExtractor.ColumnWithIndex

2022-01-21 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-25754: --- Summary: Drop unused private class org.apache.flink.connector.elasticsearch.table.KeyExtractor.ColumnWithIndex Key: FLINK-25754 URL: https://issues.apache.org/jira/browse/FL

Re:Re: [VOTE] FLIP-205: Support cache in DataStream for Batch Processing

2022-01-21 Thread Yuepeng Pan
+1 (non-binding) Thanks for driving this. Looking forward to this feature. Best, Roc. At 2022-01-21 17:22:36, "Dian Fu" wrote: >+1 (binding) > >Thanks for driving this. Looking forward to this feature. > >Regards, >Dian > >On Thu, Jan 20, 2022 at 5:57 PM Gen Luo wrote: > >> +1 (non-bindin

Re: [VOTE] Release flink-shaded 15.0, release candidate #2

2022-01-21 Thread Chesnay Schepler
+1 (binding) On 19/01/2022 17:03, Till Rohrmann wrote: +1 (binding) * Checked checksums and signatures * Looked at the diff between this RC and 14.0 and everything looked good Cheers, Till On Wed, Jan 19, 2022 at 2:30 PM Piotr Nowojski wrote: +1 (binding) I checked the diff between 14.0..

Re: [VOTE] Release flink-shaded 15.0, release candidate #2

2022-01-21 Thread Chesnay Schepler
The vote time has passed and we have unanimously approved this release. binding votes: * piotr * till * chesnay On 21/01/2022 15:30, Chesnay Schepler wrote: +1 (binding) On 19/01/2022 17:03, Till Rohrmann wrote: +1 (binding) * Checked checksums and signatures * Looked at the diff between thi

[VOTE] FLIP-211: Kerberos delegation token framework

2022-01-21 Thread Gabor Somogyi
Hi devs, I would like to start the vote for FLIP-211 [1], which was discussed and reached a consensus in the discussion thread [2]. The vote will be open for at least 72h, unless there is an objection or not enough votes. BR, G [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+K

Re: [VOTE] FLIP-211: Kerberos delegation token framework

2022-01-21 Thread Junfan Zhang
+1 Thanks for driving this. By the way does the process of voting is only for Flink committer and not contributors? I’m not familiar and it’s my first voting. Best, JunFan. On Jan 21, 2022, 10:57 PM +0800, Gabor Somogyi , wrote: > Hi devs, > > I would like to start the vote for FLIP-211 [1], whi

[jira] [Created] (FLINK-25755) Introduce shared interface for adaptive scheduler state transitions

2022-01-21 Thread Jira
David Morávek created FLINK-25755: - Summary: Introduce shared interface for adaptive scheduler state transitions Key: FLINK-25755 URL: https://issues.apache.org/jira/browse/FLINK-25755 Project: Flink

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

2022-01-21 Thread David Morávek
Hi Gabor, thanks for drafting the FLIP, I think having a solid Kerberos support is crucial for many enterprise deployments. I have multiple questions regarding the implementation (note that I have very limited knowledge of Kerberos): 1) If I understand it correctly, we'll only obtain tokens in t

[jira] [Created] (FLINK-25756) Dedicated Opensearch connectors

2022-01-21 Thread Andriy Redko (Jira)
Andriy Redko created FLINK-25756: Summary: Dedicated Opensearch connectors Key: FLINK-25756 URL: https://issues.apache.org/jira/browse/FLINK-25756 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

2022-01-21 Thread Gabor Somogyi
1. One thing is important, token management is planned to be done generically within Flink and not scattered in RM specific code. JobManager has a DelegationTokenManager which obtains tokens time-to-time (if configured properly). JM knows which TaskManagers are in place so it can distribute it to a

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

2022-01-21 Thread Gabor Somogyi
Oh and one more thing. I'm planning to add this feature in small chunk of PRs because security is super hairy area. That way reviewers can be more easily obtains the concept. On Fri, 21 Jan 2022, 18:03 David Morávek, wrote: > Hi Gabor, > > thanks for drafting the FLIP, I think having a solid Ker

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

2022-01-21 Thread David Morávek
Hi Gabor, 1. One thing is important, token management is planned to be done > generically within Flink and not scattered in RM specific code. JobManager > has a DelegationTokenManager which obtains tokens time-to-time (if > configured properly). JM knows which TaskManagers are in place so it can >

[jira] [Created] (FLINK-25757) Fix this security issue related to this exploit: https://www.exploit-db.com/exploits/48978

2022-01-21 Thread Michael Bailey (Jira)
Michael Bailey created FLINK-25757: -- Summary: Fix this security issue related to this exploit: https://www.exploit-db.com/exploits/48978 Key: FLINK-25757 URL: https://issues.apache.org/jira/browse/FLINK-25757

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

2022-01-21 Thread Gabor Somogyi
> I think we might both mean something different by the RM. You feel it well, I've not specified these terms well in the explanation. RM I meant resource management framework. JobManager is the Flink class. This means that inside JM instance there will be a DTM instance, so they would have the sam

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

2022-01-21 Thread David Morávek
> > JobManager is the Flink class. There is no such class in Flink. The closest thing to the JobManager is a ClusterEntrypoint. The cluster entrypoint spawns new RM Runner & Dispatcher Runner that start participating in the leader election. Once they gain leadership they spawn the actual underlyi

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

2022-01-21 Thread Gabor Somogyi
Here is the exact class, I'm from mobile so not had a look at the exact class name: https://github.com/gaborgsomogyi/flink/blob/8ab75e46013f159778ccfce52463e7bc63e395a9/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java#L176 That keeps track of TMs where the tokens can be