[jira] [Created] (FLINK-23325) Bump flink-shaded-netty to 4.1.65

2021-07-09 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-23325: Summary: Bump flink-shaded-netty to 4.1.65 Key: FLINK-23325 URL: https://issues.apache.org/jira/browse/FLINK-23325 Project: Flink Issue Type: Improve

[jira] [Created] (FLINK-23326) Bump flink-shaded-guava to 30.1.1

2021-07-09 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-23326: Summary: Bump flink-shaded-guava to 30.1.1 Key: FLINK-23326 URL: https://issues.apache.org/jira/browse/FLINK-23326 Project: Flink Issue Type: Improve

Re: Job Recovery Time on TM Lost

2021-07-09 Thread Till Rohrmann
Gen is right with his explanation why the dead TM discovery can be faster with Flink < 1.12. Concerning flaky TaskManager connections: 2.1 I think the problem is that the receiving TM does not know the container ID of the sending TM. It only knows its address. But this is something one could impr

Re: [DISCUSS] Lifecycle of ShuffleMaster and its Relationship with JobMaster and PartitionTracker

2021-07-09 Thread Yingjie Cao
Hi Zhu, Thanks for the reply. > One question which might be out of the scope is that whether we should do similar things for ShuffleEnvironment? I agree we should also consider fatal error handling for ShuffleEnvironment eventually. > I think the proposal does not conflict with this target. One

[jira] [Created] (FLINK-23327) Spark Structured Streaming runner sources should be put back to main src folder

2021-07-09 Thread Etienne Chauchot (Jira)
Etienne Chauchot created FLINK-23327: Summary: Spark Structured Streaming runner sources should be put back to main src folder Key: FLINK-23327 URL: https://issues.apache.org/jira/browse/FLINK-23327

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-09 Thread Jun Qin
Congratulations, Yuan! > On Jul 8, 2021, at 11:49 AM, Jingsong Li wrote: > > Congratulations Yuan! > > Best, > Jingsong > > On Thu, Jul 8, 2021 at 5:43 PM Arvid Heise wrote: > >> Yay! >> >> On Thu, Jul 8, 2021 at 10:02 AM Jiayi Liao >> wrote: >> >>> Congratulations Yuan! >>> >>> Best, >>

[jira] [Created] (FLINK-23328) UDF Improvement

2021-07-09 Thread waywtdcc (Jira)
waywtdcc created FLINK-23328: Summary: UDF Improvement Key: FLINK-23328 URL: https://issues.apache.org/jira/browse/FLINK-23328 Project: Flink Issue Type: Improvement Components: Table S

[jira] [Created] (FLINK-23329) Bump flink-shaded to 14.0

2021-07-09 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-23329: Summary: Bump flink-shaded to 14.0 Key: FLINK-23329 URL: https://issues.apache.org/jira/browse/FLINK-23329 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-23330) Deprecate toAppendStream and toRetractStream

2021-07-09 Thread Timo Walther (Jira)
Timo Walther created FLINK-23330: Summary: Deprecate toAppendStream and toRetractStream Key: FLINK-23330 URL: https://issues.apache.org/jira/browse/FLINK-23330 Project: Flink Issue Type: Sub-

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-09 Thread Benchao Li
Congratulations Yuan~ Jun Qin 于2021年7月9日周五 下午4:27写道: > Congratulations, Yuan! > > > On Jul 8, 2021, at 11:49 AM, Jingsong Li wrote: > > > > Congratulations Yuan! > > > > Best, > > Jingsong > > > > On Thu, Jul 8, 2021 at 5:43 PM Arvid Heise wrote: > > > >> Yay! > >> > >> On Thu, Jul 8, 2021 at

[DISCUSS] FLIP-182: Watermark alignment

2021-07-09 Thread Piotr Nowojski
Hey! A couple of weeks ago me and Arvid Heise played around with an idea to address a long standing issue of Flink: lack of watermark/event time alignment between different parallel instances of sources, that can lead to ever growing state size for downstream operators like WindowOperator. We had

[jira] [Created] (FLINK-23331) FileReadingWatermarkITCase.testWatermarkEmissionWithChaining fails on Azure

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23331: --- Summary: FileReadingWatermarkITCase.testWatermarkEmissionWithChaining fails on Azure Key: FLINK-23331 URL: https://issues.apache.org/jira/browse/FLINK-23331 Project: Fl

[jira] [Created] (FLINK-23332) Update flink-training exercises gradle version

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23332: --- Summary: Update flink-training exercises gradle version Key: FLINK-23332 URL: https://issues.apache.org/jira/browse/FLINK-23332 Project: Flink Issue Type: Impr

[jira] [Created] (FLINK-23333) Improve gradle setup for flink-training exercises

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-2: --- Summary: Improve gradle setup for flink-training exercises Key: FLINK-2 URL: https://issues.apache.org/jira/browse/FLINK-2 Project: Flink Issue Type: I

[jira] [Created] (FLINK-23334) Move 'application' implementation decision to subprojects

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23334: --- Summary: Move 'application' implementation decision to subprojects Key: FLINK-23334 URL: https://issues.apache.org/jira/browse/FLINK-23334 Project: Flink Issue

[jira] [Created] (FLINK-23335) Add a separate 'runSolution' gradle task

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23335: --- Summary: Add a separate 'runSolution' gradle task Key: FLINK-23335 URL: https://issues.apache.org/jira/browse/FLINK-23335 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-23336) Use the same log4j version as in Flink 1.13

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23336: --- Summary: Use the same log4j version as in Flink 1.13 Key: FLINK-23336 URL: https://issues.apache.org/jira/browse/FLINK-23336 Project: Flink Issue Type: Improve

[jira] [Created] (FLINK-23337) Properly use the 'shadow' plugin and remove flinkShadowJar

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23337: --- Summary: Properly use the 'shadow' plugin and remove flinkShadowJar Key: FLINK-23337 URL: https://issues.apache.org/jira/browse/FLINK-23337 Project: Flink Issu

[jira] [Created] (FLINK-23338) Use Spotless for flink-training exercises as well

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23338: --- Summary: Use Spotless for flink-training exercises as well Key: FLINK-23338 URL: https://issues.apache.org/jira/browse/FLINK-23338 Project: Flink Issue Type: I

[jira] [Created] (FLINK-23339) Disable flink-training exercises in Scala by default

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23339: --- Summary: Disable flink-training exercises in Scala by default Key: FLINK-23339 URL: https://issues.apache.org/jira/browse/FLINK-23339 Project: Flink Issue Type

[jira] [Created] (FLINK-23340) Improve development instructions for flink-training exercises

2021-07-09 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-23340: --- Summary: Improve development instructions for flink-training exercises Key: FLINK-23340 URL: https://issues.apache.org/jira/browse/FLINK-23340 Project: Flink

[DISCUSS] FLIP-183: Dynamic buffer size adjustment

2021-07-09 Thread Anton,Kalashnikov
Hey! There is a wish to decrease amount of in-flight data which can improve aligned checkpoint time(fewer in-flight data to process before checkpoint can complete) and improve the behaviour and performance of unaligned checkpoints (fewer in-flight data that needs to be persisted in every unal

Re: [DISCUSS] FLIP-182: Watermark alignment

2021-07-09 Thread Seth Wiesman
+1 In my opinion, this limitation is perfectly fine for the MVP. Watermark alignment is a long-standing issue and this already moves the ball so far forward. I don't expect this will cause many issues in practice, as I understand it the FileSource always processes one split at a time, and in my e

Re: [DISCUSS] Incrementally deprecating the DataSet API

2021-07-09 Thread Etienne Chauchot
Hi Seth, Thanks for your comment. That is perfect then ! If the community agrees to give precedence to the table API connectors over the DataStream connectors as discussed here, all the features are already available then. I will not need to port my latest additions to table connectors then

Re: [DISCUSS] Incrementally deprecating the DataSet API

2021-07-09 Thread Seth Wiesman
Happy to help clarify, and certainly, I don't think anyone would reject the contribution if you wanted to add that functionality. The important point we want to communicate is that the table API has matured to a level where we do not see it as supplementary to DataStream but as an equal first-class

[jira] [Created] (FLINK-23341) AkkaRpcActorTest.testOnStopFutureCompletionDirectlyTerminatesAkkaRpcActor fails on azure

2021-07-09 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-23341: - Summary: AkkaRpcActorTest.testOnStopFutureCompletionDirectlyTerminatesAkkaRpcActor fails on azure Key: FLINK-23341 URL: https://issues.apache.org/jira/browse/FLINK-2334

Re: [DISCUSS] FLIP-183: Dynamic buffer size adjustment

2021-07-09 Thread Piotr Nowojski
Hey, Thanks for writing this down. +1 from my side. Best, Piotrek pt., 9 lip 2021 o 17:00 Anton,Kalashnikov napisał(a): > Hey! > > There is a wish to decrease amount of in-flight data which can improve > aligned checkpoint time(fewer in-flight data to process before > checkpoint can complete)

[jira] [Created] (FLINK-23342) State ownership: compute shared state on recovery

2021-07-09 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-23342: - Summary: State ownership: compute shared state on recovery Key: FLINK-23342 URL: https://issues.apache.org/jira/browse/FLINK-23342 Project: Flink I

[jira] [Created] (FLINK-23343) State ownership: prevent shared state deletion on TM

2021-07-09 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-23343: - Summary: State ownership: prevent shared state deletion on TM Key: FLINK-23343 URL: https://issues.apache.org/jira/browse/FLINK-23343 Project: Flink

[jira] [Created] (FLINK-23344) State ownership: track and discard private state (rocks, heap)

2021-07-09 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-23344: - Summary: State ownership: track and discard private state (rocks, heap) Key: FLINK-23344 URL: https://issues.apache.org/jira/browse/FLINK-23344 Project: Fli

[RESULT] [VOTE] FLIP-172: Support custom transactional.id prefix in FlinkKafkaProducer

2021-07-09 Thread Wenhao Ji
Hi everyone, I am happy to announce that FLIP-172 [1] is approved. The vote [2] is now closed. There were five +1 votes, three of them were binding: - Dawid Wysakowicz (binding) - Piotr Nowojski (binding) - Arvid Heise (binding) - Yuan Mei (non-binding) - Daniel Lorych (non-binding) There was no

Re: [DISCUSS] FLIP-172: Support custom transactional.id prefix in FlinkKafkaProducer

2021-07-09 Thread Wenhao Ji
Thank everyone for participating in this discussion! The improvement has been accepted now. Can someone kindly assign this ticket FLINK-22452 [1] to me? I would like to start to implement this feature. [1] https://issues.apache.org/jira/browse/FLINK-22452 Thanks, Wenhao On Mon, Jun 28, 2021 at 6