[DISCUSS] Incorporate performance regression monitoring into routine process

Hi devs, I'd like to start a discussion about incorporating performance regression monitoring into the routine process. Flink benchmarks are periodically executed on http://codespeed.dak8s.net:8080 to monitor Flink performance. In late Oct'22, a new slack channel #flink-dev-benchmarks was created

Re: [DISCUSS] Enabling dynamic partition discovery by default in Kafka source

Hi all, Sorry to come into the discussion late--I saw the thread earlier. I'm also +1 for the change in general. I think most users have this turned on by default since the overhead is quite low. A default in the two digit seconds range works well for us. However, I do have two main concerns that

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

Hi all, I'm also interested in this FLIP! A metric group that extends from the jobmanager group resonates well with me too. I prefer choice 1 since that's consistent with how the other metric groups are designed. In addition, in your example, I think you mean to write `parent.addGroup("operator_

Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)

Hi, @Dawid yes each state is compressed separately as with keyed state. I'll add this precision to the FLIP. Best Etienne Le 18/01/2023 à 13:55, Dawid Wysakowicz a écrit : It makes sense from my side. Could you, just for completeness, extend it with the info what will be the compression u

Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)

Hi, In the future we could add new compression algorithms by simply extending /StreamCompressionDecorator/. For now there is only 2 extensions: /UncompressedStreamCompressionDecorator/ and /SnappyStreamCompressionDecorator/. But I agree, I'd stick to /SnappyStreamCompressionDecorator /which

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

Hi Jing, Thanks for your input! 1) I think that we have had more than two release cycles for the Sink V2 API. The first release of the Sink API was introduced with Flink 1.12 which was December 2020, more than 2 years ago. The additional feature that Sink V2 has introduced was the ability to hook

Re: [DISCUSS] Incorporate performance regression monitoring into routine process

Hey Yanfei, Thanks so much for the efforts driving the whole process. It's great to see that the performance benchmarks are indeed useful to help find regressions. This is a discussion thread separated from the original performance benchmark announcement thread [1]. Let's continue here so that mor

Re: [DISCUSS] Release Flink 1.16.1

Hi apologies, Apologies for getting back late on this. There are no more blockers for Flink 1.16.1. I'll kick off the release process. Best regards, Martijn Op di 10 jan. 2023 om 14:21 schreef Jing Ge : > Hi Martijn, > > +1 and thanks for bringing this up. Since all blocking issues have been

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

> First, I do not understand why users have to configure the new scope format, which has a default value. If you don't use scope formats, sure. If you do use scope formats, e.g. to add a common prefix (which is the case for datadog users for example), then the current default in the FLIP is in

[jira] [Created] (FLINK-30753) Py4J cannot acquire Table.explain() method

Yunfeng Zhou created FLINK-30753: Summary: Py4J cannot acquire Table.explain() method Key: FLINK-30753 URL: https://issues.apache.org/jira/browse/FLINK-30753 Project: Flink Issue Type: Improv

[jira] [Created] (FLINK-30754) Fix ExceptionThrowingDelegationTokenProvider/Receiver multi-threaded test issues

Gabor Somogyi created FLINK-30754: - Summary: Fix ExceptionThrowingDelegationTokenProvider/Receiver multi-threaded test issues Key: FLINK-30754 URL: https://issues.apache.org/jira/browse/FLINK-30754 Pr

Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)

Hi, The idea sounds like a nice improvement to complete the feature. I don't have any comments on top of what has been written already above. Bets, Piotrek czw., 19 sty 2023 o 09:57 Etienne Chauchot napisał(a): > Hi, > > In the future we could add new compression algorithms by simply > extendi

[DISCUSS] Release connector-parent 1.0

Hello, I'd like to do the first release of the connector parent pom under Apache. I'd like this to be done before 1.17 so the connectors can make use of the proper artifact. Does anyone have any other suggestions for what we should/could change compared before we do that?

[jira] [Created] (FLINK-30755) Make SinkV2 support speculative execution for batch jobs

Biao Liu created FLINK-30755: Summary: Make SinkV2 support speculative execution for batch jobs Key: FLINK-30755 URL: https://issues.apache.org/jira/browse/FLINK-30755 Project: Flink Issue Type:

[jira] [Created] (FLINK-30756) StreamTableExplainTests.test_explain fails

Matthias Pohl created FLINK-30756: - Summary: StreamTableExplainTests.test_explain fails Key: FLINK-30756 URL: https://issues.apache.org/jira/browse/FLINK-30756 Project: Flink Issue Type: Bug

Re: [DISCUSS] Incorporate performance regression monitoring into routine process

Hi, Thanks for bringing this up! Generally speaking +1 for the proposal. I have only one suggestion for the draft. In the past years, when I was creating performance regression tickets, I was setting the priority to a blocker and I would propose to add this to the instructions and general convent

Re: Stateful Functions with Flink 1.15 and onwards

I thought I'd check in again here, is there any update? If there's anything I can do to help with releasing statefun 3.3, just let me know. Thanks. On Mon, Nov 28, 2022 at 8:12 AM Galen Warren wrote: > Hi Gordon, just checking in. Is there anything I can do to help here? > > On Fri, Nov 4, 2022

[jira] [Created] (FLINK-30757) Ugrade busybox version to a pinned version for operator

Gabor Somogyi created FLINK-30757: - Summary: Ugrade busybox version to a pinned version for operator Key: FLINK-30757 URL: https://issues.apache.org/jira/browse/FLINK-30757 Project: Flink Iss

[VOTE] Release 1.16.1, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 1.16.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

[DISCUSS] Introduce Side Output Functionality to the Table & SQL API

Hi Dev Team, I’m Jeremy, a Solutions Architect supporting the Kinesis Data Analytics for Apache Flink service at AWS and wanted to open a discussion about a particular problem our customers have faced, and how new functionality within the Table and SQL API’s within Apache Flink could help solve

Re: [DISCUSS] Introduce Side Output Functionality to the Table & SQL API

Hi Jeremy, Have you looked into the CURRENT_WATERMARK function [1] which will allow one to operate on late events without resorting to using the DataStream API? Best regards, Martijn [1] https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/functions/systemfunctions/#temporal-func

Re: [DISCUSS] Introduce Side Output Functionality to the Table & SQL API

Thanks for sharing this, Martijn--I will test it out! - Jeremy On 1/19/23, 12:59 PM, "Martijn Visser" mailto:martijnvis...@apache.org>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know

Py4JJavaError

Hi developers: I am trying to run a sample Flink consumer ( https://nightlies.apache.org/flink/flink-docs-release-1.16/api/python//examples/datastream/connectors.html) but I get these error messages (I am running the program on Mac M1, and downgraded my java to jdk 8: java version "1.8.0_351"):

Re: [Discussion] externalize Hive connector

Hi Yuxia, FLINK-30667 aims to address hive-connector dependency on table-planner @internal class . Here is a bit of rationale. ParserImpl in table-planner and HiveParser in hive connector should be able to evolve separately after future externa

[jira] [Created] (FLINK-30758) Remove sql-client.display.max-column-width

Jing Ge created FLINK-30758: --- Summary: Remove sql-client.display.max-column-width Key: FLINK-30758 URL: https://issues.apache.org/jira/browse/FLINK-30758 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-290: Operator state compression (FLINK-30113)

The proposal reads quite reasonable! I do not have additional comments as long as the change can insure backward compatibility. And many thanks to Dawid for catching this! Best Yuan On Thu, Jan 19, 2023 at 6:03 PM Piotr Nowojski wrote: > Hi, > > The idea sounds like a nice improvement to comp