[jira] [Created] (FLINK-30731) Translate "Set Operations" page of "Querys" into Chinese

2023-01-18 Thread chenhaiyang (Jira)
chenhaiyang created FLINK-30731: --- Summary: Translate "Set Operations" page of "Querys" into Chinese Key: FLINK-30731 URL: https://issues.apache.org/jira/browse/FLINK-30731 Project: Flink Issue

Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job

2023-01-18 Thread Biao Liu
Hi Martijn & Jing, Thanks for feedback! Currently, SinkFunction is in a subtle circumstance. Like Jing pointed out, SinkFunction is still marked as public. Technically, according to the Flink Bylaws[1], the decision should be approved through an official voting. Although many of the community mai

Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job

2023-01-18 Thread Yun Tang
Hi Biao, I think it's time to deprecate the SinkFunction and it would be fine if you could drive to launch the discussion. BTW, we might make it done in flink-1.17 release with deprecating SourceFunction[1] together. [1] https://issues.apache.org/jira/browse/FLINK-28045 Best Yun Tang ___

Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job

2023-01-18 Thread Martijn Visser
Hi Biao, Just to clarify, I understand your point of view and will not block your FLIP :) It would indeed be great to achieve that both the SinkFunction and SourceFunction will be marked as deprecated in 1.17 like Yun Tang pointed out. Best regards, Martijn Op wo 18 jan. 2023 om 09:34 schreef

[jira] [Created] (FLINK-30732) "Running HA (hashmap, sync) end-to-end test" failed with exit code 1

2023-01-18 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-30732: - Summary: "Running HA (hashmap, sync) end-to-end test" failed with exit code 1 Key: FLINK-30732 URL: https://issues.apache.org/jira/browse/FLINK-30732 Project: Flink

[jira] [Created] (FLINK-30733) The AzureCI Slack bot does not list all failed builds in the Slack channel

2023-01-18 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-30733: - Summary: The AzureCI Slack bot does not list all failed builds in the Slack channel Key: FLINK-30733 URL: https://issues.apache.org/jira/browse/FLINK-30733 Project:

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

2023-01-18 Thread Chesnay Schepler
You're misunderstanding the problem. Metric groups form a tree with each group providing certain metadata. E.g., on the taskmanager we have a TM metric group that provides info about the TM, that has child task metric groups, that have child operator metric groups etc. The operator metric grou

Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job

2023-01-18 Thread Jing Ge
Hi, I think it will be confusing for users that the older API is deprecated but the related new API is not graduated yet which ends up with the awkward situation that none proper API could be used in production. In our case, e.g. the sinkv2.Sink should be marked as @public before the SinkFunction

[jira] [Created] (FLINK-30734) KBinsDiscretizer handles Double.NaN incorrectly

2023-01-18 Thread Fan Hong (Jira)
Fan Hong created FLINK-30734: Summary: KBinsDiscretizer handles Double.NaN incorrectly Key: FLINK-30734 URL: https://issues.apache.org/jira/browse/FLINK-30734 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-30735) Translate "Top-N" page of "Querys" into Chinese

2023-01-18 Thread chenhaiyang (Jira)
chenhaiyang created FLINK-30735: --- Summary: Translate "Top-N" page of "Querys" into Chinese Key: FLINK-30735 URL: https://issues.apache.org/jira/browse/FLINK-30735 Project: Flink Issue Type: Sub

Re: [Discuss] Checkout the CURRENT_SNAPSHOT_VERSION branch of the flink-benchmarks repo during each major release

2023-01-18 Thread Yun Tang
Since we already had reached an agreement more than two years ago, and there were no objections here, I will then update the create-release guide. Thanks to everyone who participated in. Best Yun Tang From: Lijie Wang Sent: Tuesday, January 17, 2023 22:50 To: de

Re: [DISCUSS] FLIP-285: Refactoring the leader election code base in Flink

2023-01-18 Thread Chesnay Schepler
There are a lot of good things in this, and until the Extension bit I'm fully on board. With the extension, how does the leader contender get access to the LeaderElection? I would've assumed that LEService returns a LeaderElection when register is called, but according to the diagram this met

Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job

2023-01-18 Thread Martijn Visser
Hi Jing, That's a valid point but it brings up a discussion on the promotion of the Sink V2 API in general. I'll open a separate discussion thread for this. Best regards, Martijn Op wo 18 jan. 2023 om 11:01 schreef Jing Ge : > Hi, > > I think it will be confusing for users that the older API

[DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

2023-01-18 Thread Martijn Visser
Hi all, While discussing FLIP-281 [1] the discussion also turned to the SinkFunction and the SinkV2 API. For a broader discussion I'm opening up a separate discussion thread. As Yun Tang has mentioned in that discussion thread, it would be a good time to deprecate the SinkFunction to avoid the ne

[jira] [Created] (FLINK-30736) Exclude generated sources from checkstyle

2023-01-18 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-30736: Summary: Exclude generated sources from checkstyle Key: FLINK-30736 URL: https://issues.apache.org/jira/browse/FLINK-30736 Project: Flink Issue Type:

[jira] [Created] (FLINK-30737) Setup

2023-01-18 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-30737: Summary: Setup Key: FLINK-30737 URL: https://issues.apache.org/jira/browse/FLINK-30737 Project: Flink Issue Type: Technical Debt Reporter

[jira] [Created] (FLINK-30738) Disable verbose mode

2023-01-18 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-30738: Summary: Disable verbose mode Key: FLINK-30738 URL: https://issues.apache.org/jira/browse/FLINK-30738 Project: Flink Issue Type: Sub-task C

Re: [DISCUSS] FLIP-285: Refactoring the leader election code base in Flink

2023-01-18 Thread Matthias Pohl
Thanks for participating in the discussion, Yang & Chesnay. LeaderElection interface extension gave me a headache as well. I added it initially because I thought it would be of more value. But essentially, it doesn't help but make the code harder to understand (as your questions rightfully point ou

[jira] [Created] (FLINK-30739) SqlGatewayRestEndpointStatementITCase failed with NullPointer

2023-01-18 Thread Yun Tang (Jira)
Yun Tang created FLINK-30739: Summary: SqlGatewayRestEndpointStatementITCase failed with NullPointer Key: FLINK-30739 URL: https://issues.apache.org/jira/browse/FLINK-30739 Project: Flink Issue

Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)

2023-01-18 Thread Dawid Wysakowicz
It makes sense from my side. Could you, just for completeness, extend it with the info what will be the compression unit? If understand it correctly you envision each state to be compressed separately. Best, Dawid On 17/01/2023 15:47, Etienne Chauchot wrote: Hi everyone, I just published

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

2023-01-18 Thread Yun Tang
SinkV2 was introduced in Flink-1.15 and annotated as @PublicEvolving from the 1st day [1]. From FLIP-197, we can promote it to @Public since it already existed with two releases. And I didn't find a FLIP to discuss the process to deprecate APIs, considering the SinkFunction has actually been sta

[jira] [Created] (FLINK-30740) INSERT to Kafka does not work when Kafka config {{auto.create.topics.enabled}} is set to false

2023-01-18 Thread Bruno Cadonna (Jira)
Bruno Cadonna created FLINK-30740: - Summary: INSERT to Kafka does not work when Kafka config {{auto.create.topics.enabled}} is set to false Key: FLINK-30740 URL: https://issues.apache.org/jira/browse/FLINK-30740

[jira] [Created] (FLINK-30741) IPv6HostnamesITCase should create remote rpc service

2023-01-18 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-30741: Summary: IPv6HostnamesITCase should create remote rpc service Key: FLINK-30741 URL: https://issues.apache.org/jira/browse/FLINK-30741 Project: Flink

[jira] [Created] (FLINK-30742) Create E2E for GlueCatalog

2023-01-18 Thread Samrat Deb (Jira)
Samrat Deb created FLINK-30742: -- Summary: Create E2E for GlueCatalog Key: FLINK-30742 URL: https://issues.apache.org/jira/browse/FLINK-30742 Project: Flink Issue Type: Sub-task Repo

[jira] [Created] (FLINK-30743) Improve Kubernetes HA Services docs

2023-01-18 Thread Wolfgang Buchner (Jira)
Wolfgang Buchner created FLINK-30743: Summary: Improve Kubernetes HA Services docs Key: FLINK-30743 URL: https://issues.apache.org/jira/browse/FLINK-30743 Project: Flink Issue Type: Impro

[jira] [Created] (FLINK-30744) Translate "Window Top-N" page of "Querys" into Chinese

2023-01-18 Thread chenhaiyang (Jira)
chenhaiyang created FLINK-30744: --- Summary: Translate "Window Top-N" page of "Querys" into Chinese Key: FLINK-30744 URL: https://issues.apache.org/jira/browse/FLINK-30744 Project: Flink Issue Ty

Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)

2023-01-18 Thread ConradJam
Thanks for driver this flip,I think have some question about this flip - Can we select the compression format, for example? such as snappy or lz0 - If compression format is supported, I think the metadata will probably have to include which compression format and I agree idea with Daw

[jira] [Created] (FLINK-30745) Check-pointing with Azure Data Lake Storage

2023-01-18 Thread Dheeraj Panangat (Jira)
Dheeraj Panangat created FLINK-30745: Summary: Check-pointing with Azure Data Lake Storage Key: FLINK-30745 URL: https://issues.apache.org/jira/browse/FLINK-30745 Project: Flink Issue Typ

[jira] [Created] (FLINK-30746) Flink Table / SQL Column types of query result and sink for do not match, but should

2023-01-18 Thread Colin Williams (Jira)
Colin Williams created FLINK-30746: -- Summary: Flink Table / SQL Column types of query result and sink for do not match, but should Key: FLINK-30746 URL: https://issues.apache.org/jira/browse/FLINK-30746

[jira] [Created] (FLINK-30747) Translate "Window Deduplication" page of "Querys" into Chinese

2023-01-18 Thread chenhaiyang (Jira)
chenhaiyang created FLINK-30747: --- Summary: Translate "Window Deduplication" page of "Querys" into Chinese Key: FLINK-30747 URL: https://issues.apache.org/jira/browse/FLINK-30747 Project: Flink

[jira] [Created] (FLINK-30748) Translate "Window Deduplication" page of "Querys" into Chinese

2023-01-18 Thread chenhaiyang (Jira)
chenhaiyang created FLINK-30748: --- Summary: Translate "Window Deduplication" page of "Querys" into Chinese Key: FLINK-30748 URL: https://issues.apache.org/jira/browse/FLINK-30748 Project: Flink

[jira] [Created] (FLINK-30749) Delegation token provider enabled flag documentation is wrong

2023-01-18 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-30749: - Summary: Delegation token provider enabled flag documentation is wrong Key: FLINK-30749 URL: https://issues.apache.org/jira/browse/FLINK-30749 Project: Flink

Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)

2023-01-18 Thread Dawid Wysakowicz
For the compression formats I'd stick with what is supported for the keyed state, which is either enable or disable compression. The format used for keyed state backend compression is snappy. Best, Dawid On 18/01/2023 15:33, ConradJam wrote: Thanks for driver this flip,I think have some ques

Re: [DISCUSS] FLIP-285: Refactoring the leader election code base in Flink

2023-01-18 Thread Matthias Pohl
After another round of discussion, I came up with a (hopefully) final proposal. The previously discussed approach was still not optimal because the contender ID lived in the LeaderContender even though it is actually LeaderElectionService-internal knowledge. Fixing that helped fix the overall archi

Unequal distribtion of Kafka partitions with respect to topic when reading from multiple topics using KafkaSource API with Flink 1.14.3

2023-01-18 Thread Kathula, Sandeep
Hi, I am using KafkaSource API read from 6 topics within Kafka. Flink version - 1.14.3. Each and every kafka topic my Flink pipeline reads from is having a different load but same number of partitions (lets say 30). For example partition 0 of topic 1 and partition 0 of topic 2 have different

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

2023-01-18 Thread Lijie Wang
Hi Martijn, Thanks for driving this. I have a only concern about the Sink.InitContext. Does the Sink.InitContext will also be changed to @Public ? As described in FLIP-287, currently the Sink.InitContext still lacks some necessary information to migrate existing connectors to new sinks. If it is

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

2023-01-18 Thread Biao Liu
Hi Martijn, Thanks for bringing us this discussion! I think it's time to mark SinkFunction as deprecated. It may help a lot to encourage users to migrate existing sink connectors to the new interface. About the concern of Lijie, I'm not sure if it's OK to compatibly change the interface with @Pu

[jira] [Created] (FLINK-30750) CompactActionITCase.testBatchCompact in table store is unstable

2023-01-18 Thread Shammon (Jira)
Shammon created FLINK-30750: --- Summary: CompactActionITCase.testBatchCompact in table store is unstable Key: FLINK-30750 URL: https://issues.apache.org/jira/browse/FLINK-30750 Project: Flink Issue

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

2023-01-18 Thread Jing Ge
Hi, Thanks Martijn for bringing this to our attention. This discussion deserves a much wider range of attention. because graduation of the Sink API will have an impact on all connectors. For me, briefly speaking, like I already pointed out in another thread [1], it is too hasty and risky to do it

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

2023-01-18 Thread Jing Ge
Hi, There are some typos in my last email, although hopefully I made myself clear with the context, sorry about that: FLIP-179 -> FLIP-197 FlinkKafkaConsumer -> FlinkKafkaProducer Best regards, Jing On Thu, Jan 19, 2023 at 5:30 AM Jing Ge wrote: > Hi, > > Thanks Martijn for bringing this to o

[jira] [Created] (FLINK-30751) Remove references to disableDataSync in RocksDB documentation

2023-01-18 Thread David Christle (Jira)
David Christle created FLINK-30751: -- Summary: Remove references to disableDataSync in RocksDB documentation Key: FLINK-30751 URL: https://issues.apache.org/jira/browse/FLINK-30751 Project: Flink

[jira] [Created] (FLINK-30752) Support 'EXPLAIN PLAN_ADVICE' statement in PyFlink

2023-01-18 Thread Jane Chan (Jira)
Jane Chan created FLINK-30752: - Summary: Support 'EXPLAIN PLAN_ADVICE' statement in PyFlink Key: FLINK-30752 URL: https://issues.apache.org/jira/browse/FLINK-30752 Project: Flink Issue Type: Sub-

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

2023-01-18 Thread Hang Ruan
Hi, chesnay, Thanks for your reply. I still have some doubts about the questions you raised. > Extending the set of ScopeFormats is problematic because it in practice > it breaks the config if users actively rely on it, since there's now > another key that they _must_ set for it to be consistent/