Re: [DISCUSS] Watermark propagation with Sink API

2021-05-17 Thread Arvid Heise
Hi Eron, I think this is a useful addition for storage systems that act as pass-through for Flink to reduce recovery time. It is only useful if you combine it with regional fail-over as only a small part of the pipeline is restarted. A couple of thoughts on the implications: 1. Watermarks are clo

[jira] [Created] (FLINK-22691) On Aliyun ECS, docker deployed flink (1.13.0), which was scanned to Mirai DDOS Trojan

2021-05-17 Thread Magina.Shan (Jira)
Magina.Shan created FLINK-22691: --- Summary: On Aliyun ECS, docker deployed flink (1.13.0), which was scanned to Mirai DDOS Trojan Key: FLINK-22691 URL: https://issues.apache.org/jira/browse/FLINK-22691 P

[jira] [Created] (FLINK-22690) kerberos integration with flink,kerberos tokens will expire

2021-05-17 Thread kevinsun (Jira)
kevinsun created FLINK-22690: Summary: kerberos integration with flink,kerberos tokens will expire Key: FLINK-22690 URL: https://issues.apache.org/jira/browse/FLINK-22690 Project: Flink Issue T

[jira] [Created] (FLINK-22689) Table API Documentation Row-Based Operations Example Fails

2021-05-17 Thread Yunfeng Zhou (Jira)
Yunfeng Zhou created FLINK-22689: Summary: Table API Documentation Row-Based Operations Example Fails Key: FLINK-22689 URL: https://issues.apache.org/jira/browse/FLINK-22689 Project: Flink Is

[jira] [Created] (FLINK-22688) Root Exception can not be shown on Web UI in Flink 1.13.0

2021-05-17 Thread Gary Wu (Jira)
Gary Wu created FLINK-22688: --- Summary: Root Exception can not be shown on Web UI in Flink 1.13.0 Key: FLINK-22688 URL: https://issues.apache.org/jira/browse/FLINK-22688 Project: Flink Issue Type: B

[DISCUSS] Watermark propagation with Sink API

2021-05-17 Thread Eron Wright
I would like to propose an enhancement to the Sink API, the ability to receive upstream watermarks. I'm aware that the sink context provides the current watermark for a given record. I'd like to be able to write a sink function that is invoked whenever the watermark changes. Out of scope would

Re: Helm chart for Flink

2021-05-17 Thread Austin Cawley-Edwards
Sorry @dev list for the pings – I misread which thread I was replying to. @Pedro Silva – the dev@flink.apache.org list is used for developers working on Flink itself. Please use u...@flink.apache.org for future questions :) Best, Austin On Mon, May 17, 2021 at 3:58 PM Austin Cawley-Edwards < au

Re: Helm chart for Flink

2021-05-17 Thread Austin Cawley-Edwards
Hi Pedro, There is currently no official Kubernetes Operator for Flink and, by extension, there is no official Helm chart. It would be relatively easy to create a chart for simply deploying standalone Flink resources via the Kubernetes manifests described here[1], though it would leave out the abi

[jira] [Created] (FLINK-22687) unified the org.apache.flink.runtime.jobmaster.slotpool.SlotPool and org.apache.flink.runtime.jobmaster.slotpool.DeclarativeSlotPool

2021-05-17 Thread lqjacklee (Jira)
lqjacklee created FLINK-22687: - Summary: unified the org.apache.flink.runtime.jobmaster.slotpool.SlotPool and org.apache.flink.runtime.jobmaster.slotpool.DeclarativeSlotPool Key: FLINK-22687 URL: https://issues.apach

Re: [DISCUSS] Releasing Flink 1.13.1

2021-05-17 Thread Arvid Heise
I'd also like to have a fix for https://issues.apache.org/jira/browse/FLINK-22686 that was reported today. On Mon, May 17, 2021 at 11:52 AM Timo Walther wrote: > Hi Konstantin, > > thanks for starting the discussion. From the Table API side, we also > fixed a couple of critical issues already th

[jira] [Created] (FLINK-22686) Incompatible subtask mappings while resuming from unaligned checkpoints

2021-05-17 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-22686: --- Summary: Incompatible subtask mappings while resuming from unaligned checkpoints Key: FLINK-22686 URL: https://issues.apache.org/jira/browse/FLINK-22686 Project: Flink

Helm chart for Flink

2021-05-17 Thread Pedro Silva
Hello, Does flink have an official Helm Chart? I haven't been able to find any, the closest most up-to-date one seems to be https://github.com/GoogleCloudPlatform/flink-on-k8s-operator. Is this correct or is there a more mature and/or recommeded helm chart to use? Thank you.

[jira] [Created] (FLINK-22685) Write data to hive table in batch mode throws FileNotFoundException.

2021-05-17 Thread Moses (Jira)
Moses created FLINK-22685: - Summary: Write data to hive table in batch mode throws FileNotFoundException. Key: FLINK-22685 URL: https://issues.apache.org/jira/browse/FLINK-22685 Project: Flink Issue

Re: [DISCUSS] Releasing Flink 1.13.1

2021-05-17 Thread Timo Walther
Hi Konstantin, thanks for starting the discussion. From the Table API side, we also fixed a couple of critical issues already that justify releasing a 1.13.1 asap. Personally, I would like to include https://issues.apache.org/jira/browse/FLINK-22666 that fixes some last issues with the Scal

Re: [VOTE] Release 1.12.4, release candidate #1

2021-05-17 Thread Xingbo Huang
+1 (non-binding) - Verified checksums and signatures - Check python package contents - Pip install python wheel package - Run python udf job in python shell Best, Xingbo Robert Metzger 于2021年5月17日周一 下午4:07写道: > +1 (binding) > > - Checked diff to 1.12.3: > > https://github.com/apache/flink/comp

[jira] [Created] (FLINK-22684) Add the ability to ignore in-flight data on recovery

2021-05-17 Thread Anton Kalashnikov (Jira)
Anton Kalashnikov created FLINK-22684: - Summary: Add the ability to ignore in-flight data on recovery Key: FLINK-22684 URL: https://issues.apache.org/jira/browse/FLINK-22684 Project: Flink

[jira] [Created] (FLINK-22683) The total Flink/process memory of memoryConfiguration in /taskmanagers can be null or incorrect value

2021-05-17 Thread Yangze Guo (Jira)
Yangze Guo created FLINK-22683: -- Summary: The total Flink/process memory of memoryConfiguration in /taskmanagers can be null or incorrect value Key: FLINK-22683 URL: https://issues.apache.org/jira/browse/FLINK-22683

[jira] [Created] (FLINK-22682) Checkpoint interval too large for higher DOP

2021-05-17 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-22682: --- Summary: Checkpoint interval too large for higher DOP Key: FLINK-22682 URL: https://issues.apache.org/jira/browse/FLINK-22682 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-22681) how can i get flink job status from prometheus(PrometheusPushGateway)

2021-05-17 Thread yuwei zhan (Jira)
yuwei zhan created FLINK-22681: -- Summary: how can i get flink job status from prometheus(PrometheusPushGateway) Key: FLINK-22681 URL: https://issues.apache.org/jira/browse/FLINK-22681 Project: Flink

[jira] [Created] (FLINK-22680) An IndexOutOfBoundsException is thrown out after apply WatermarkAssignerChangelogNormalizeTransposeRule

2021-05-17 Thread Andy (Jira)
Andy created FLINK-22680: Summary: An IndexOutOfBoundsException is thrown out after apply WatermarkAssignerChangelogNormalizeTransposeRule Key: FLINK-22680 URL: https://issues.apache.org/jira/browse/FLINK-22680

Re: [DISCUSS] Releasing Flink 1.13.1

2021-05-17 Thread Robert Metzger
Thanks a lot for starting the discussion about the release. I'd like to include https://issues.apache.org/jira/browse/FLINK-22266 as well (I forgot to set the fixVersion accordingly). It's an important fix for the stop with savepoint operation of the adaptive scheduler. On Mon, May 17, 2021 at 10:

Re: [VOTE] Release 1.12.4, release candidate #1

2021-05-17 Thread Robert Metzger
+1 (binding) - Checked diff to 1.12.3: https://github.com/apache/flink/compare/release-1.12.3-rc1...release-1.12.4-rc1 - checked some jars in the staging repo - checked contents of flink-dist jar of 2.11 release - Submitted demo job On Sun, May 16, 2021 at 8:40 PM Dawid Wysakowicz wrote: > +1

[DISCUSS] Releasing Flink 1.13.1

2021-05-17 Thread Konstantin Knauf
Hi everyone, I would like to start discussing Flink 1.13.1. There are already quite a few critical fixes merged, specifically: * https://issues.apache.org/jira/browse/FLINK-22555 (LGPL-2.1 files in flink-python jars) * https://issues.apache.org/jira/browse/FLINK-17170 (Cannot stop streaming job w

Re: [DISCUSS]FLIP-150: Introduce Hybrid Source

2021-05-17 Thread Nicholas Jiang
Hi Thomas, Sorry for later reply for your POC. I have reviewed the based abstract implementation of your pull request: https://github.com/apache/flink/pull/15924. IMO, for the switching mechanism, this level of abstraction is not concise enough, which doesn't make connector contribution easier.

[jira] [Created] (FLINK-22679) code optimization:Transformation.equals

2021-05-17 Thread huzeming (Jira)
huzeming created FLINK-22679: Summary: code optimization:Transformation.equals Key: FLINK-22679 URL: https://issues.apache.org/jira/browse/FLINK-22679 Project: Flink Issue Type: Improvement