Re: [Discuss] Checkout the CURRENT_SNAPSHOT_VERSION branch of the flink-benchmarks repo during each major release

Thanks for the proposal, Yun! +1 for managing the flink-benchmark repository by branch. I didn't put my eyes on the benchmark repo a lot but having only one branch for testing multiple Flink versions requires a lot of adaptive code I assume. Also this will make it easier to compare performance bet

Re: [DISCUSS] FLIP-264 Extract BaseCoordinatorContext

What's the status and conclusion of this discussion? I have seen the value of exposing OperatorCoordinator because of the powerful RPC calls, some projects are already using it, such as Hudi[1]. But I agree this is a large topic and requires another FLIP. I am also concerned about extracting a Pu

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

Thanks for making this detailed guide, Yanfei! This is quite helpful for release managers to monitor and manage performance regressions. I think it will be great to also document the threshold of alerts sent to the Slack channel, and some related formula used in the test, either in the wiki page o

[jira] [Created] (FLINK-30696) Translate "joins" page of "Querys" into Chinese

chenhaiyang created FLINK-30696: --- Summary: Translate "joins" page of "Querys" into Chinese Key: FLINK-30696 URL: https://issues.apache.org/jira/browse/FLINK-30696 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-30697) Upgrade scalafmt documentation

Matthias Pohl created FLINK-30697: - Summary: Upgrade scalafmt documentation Key: FLINK-30697 URL: https://issues.apache.org/jira/browse/FLINK-30697 Project: Flink Issue Type: Bug Co

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

The proposed ScopeFormat is still problematic for a few reasons. Extending the set of ScopeFormats is problematic because it in practice it breaks the config if users actively rely on it, since there's now another key that they _must_ set for it to be consistent/compatible with their existing

[jira] [Created] (FLINK-30698) SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails

Matthias Pohl created FLINK-30698: - Summary: SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails Key: FLINK-30698 URL: https://issues.apache.org/jira/browse/FLINK-30698 Project: Flink

[jira] [Created] (FLINK-30699) Improve the efficiency of the getRandomString method in the StringUtils class

Bingye Chen created FLINK-30699: --- Summary: Improve the efficiency of the getRandomString method in the StringUtils class Key: FLINK-30699 URL: https://issues.apache.org/jira/browse/FLINK-30699 Project:

Re: [VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

+1 (binding) Best, Godfrey Shammon FY 于2023年1月12日周四 23:27写道： > > +1 (no-binding) > > > Best, > Shammon > > On Thu, Jan 12, 2023 at 8:11 PM Shengkai Fang wrote: > > > +1(binding) > > > > Best, > > Shengkai > > > > Jark Wu 于2023年1月12日周四 19:22写道： > > > > > +1 (binding) > > > Thank you for driving

[jira] [Created] (FLINK-30700) Translate "Deduplication" page of "Querys" into Chinese

chenhaiyang created FLINK-30700: --- Summary: Translate "Deduplication" page of "Querys" into Chinese Key: FLINK-30700 URL: https://issues.apache.org/jira/browse/FLINK-30700 Project: Flink Issue

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

Hi all, After an offline talk with Gunnar I review all connectors on branch release_1.16 (still has virtually all connectors) we can see some of them using ExecutionConfig.. The basic use is for creating serializers: TypeInformation.createSerializer(ExecutionConfig config), and since it is a F

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

Slight correction: Using metrics.scope.jm.job as the default should be safe. On 16/01/2023 10:18, Chesnay Schepler wrote: The proposed ScopeFormat is still problematic for a few reasons. Extending the set of ScopeFormats is problematic because it in practice it breaks the config if users activ

Re: [DISCUSS] FLIP-286: Fix the API stability/scope annotation inconsistency in AbstractStreamOperator

In addition to fixing the annotation consistency formally, I think it is equally, if not more, important to see whether the public APIs we expose tell a good story. For example, if we say StreamConfig should be internal, some fair questions to ask is why our own AbstractStreamOperat

[RESULT][VOTE] FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode

Hi all, I am happy to announce that FLIP-279 Unified the max display column width for SqlClient and Table APi in both Streaming and Batch execMode [1] has been accepted. There are 3 binding votes and 1 non-binding vote [2]: - Jark Wu (binding) - Shengkai Fang (binding) - Shammon FY (non-binding)

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

Hi Jing Ge, Thanks for your response.. Making the review left above about all connectors, I realise that we need the full ExecutionConfig as it is needed to generate the serializer correctly if objectReuse is enabled as we call TypeInformation.createSerializer(ExecutionConfig config) On the Po

Re: [Discuss] Checkout the CURRENT_SNAPSHOT_VERSION branch of the flink-benchmarks repo during each major release

Hi Yun, Thanks for kicking off this discussion, +1 for the proposal. In the past, we occasionally encountered flink-benchmarks compilation failures caused by the code updates in flink repo, like FLINK-28931[1]. Managing the flink-benchmarks repository by release branch can help us handle these cas

[jira] [Created] (FLINK-30701) Remove SimpleTableTestHelper and use sql in SparkITCase

Shammon created FLINK-30701: --- Summary: Remove SimpleTableTestHelper and use sql in SparkITCase Key: FLINK-30701 URL: https://issues.apache.org/jira/browse/FLINK-30701 Project: Flink Issue Type: Sub

[NOTICE] dependency-reduced pom now put in module basedir

Hello, I just merged FLINK-30350 with which the dependency reduced pom by the shade plugin is no longer put into the target directory but the modules basedir instead. This avoids some side-effects where the modules basedir was changed after the shade-plugin ran (to the target/ directory), wh

[NOTICE] Parallel compilation should no longer lock up

Hello, with FLINK-30083 being merged it should now be possible to safely build Flink with the -T switch without it locking up during dependency reduction. On my machine a clean install -T2 (without QA plugins, tests or building the UI) this saved about 20%; YMMV. I'll see if we can't use th

[jira] [Created] (FLINK-30702) Introduce ElasticDialect

Grzegorz Kołakowski created FLINK-30702: --- Summary: Introduce ElasticDialect Key: FLINK-30702 URL: https://issues.apache.org/jira/browse/FLINK-30702 Project: Flink Issue Type: New Featur

[jira] [Created] (FLINK-30703) PulsarOrderedPartitionSplitReaderTest>PulsarPartitionSplitReaderTestBase.consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition fails

Matthias Pohl created FLINK-30703: - Summary: PulsarOrderedPartitionSplitReaderTest>PulsarPartitionSplitReaderTestBase.consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition fails Key: FLINK-30703

Re: [NOTICE] Parallel compilation should no longer lock up

Hi Chesnay, Thanks for the efforts! During development I've seen some of the tests are not multi-threaded compliant. Intended to fix what I'm aware of... G On Mon, Jan 16, 2023 at 2:19 PM Chesnay Schepler wrote: > Hello, > > with FLINK-30083 being merged it should now be possible to safely bu

[jira] [Created] (FLINK-30704) Add S3 delegation token support

Gabor Somogyi created FLINK-30704: - Summary: Add S3 delegation token support Key: FLINK-30704 URL: https://issues.apache.org/jira/browse/FLINK-30704 Project: Flink Issue Type: New Feature

Supplying jar stored at S3 to flink to run the job in kubernetes

I have been following the examples mentioned here: flink-kubernetes-operator_examples . I'm testing this on the local minikube. I have deployed minio for s3 and flink operator. I have my application jar in s3(using minio for t

[jira] [Created] (FLINK-30705) CR reference generator does not respect JsonProperty annotation

Gyula Fora created FLINK-30705: -- Summary: CR reference generator does not respect JsonProperty annotation Key: FLINK-30705 URL: https://issues.apache.org/jira/browse/FLINK-30705 Project: Flink

[jira] [Created] (FLINK-30706) Remove flink-table-common dependency for table store core

Jingsong Lee created FLINK-30706: Summary: Remove flink-table-common dependency for table store core Key: FLINK-30706 URL: https://issues.apache.org/jira/browse/FLINK-30706 Project: Flink Iss

Re: Supplying jar stored at S3 to flink to run the job in kubernetes

Hi, Rahul User support and questions should be sent to the user mailing list ( u...@flink.apache.org) You can resend the issue to the user mailing list with a detailed error log. Best, Weihua On Mon, Jan 16, 2023 at 11:18 PM rahul sahoo wrote: > I have been following the examples mentioned h

[jira] [Created] (FLINK-30707) Let speculative execution takes input number bytes into account when detecting slow tasks

xingbe created FLINK-30707: -- Summary: Let speculative execution takes input number bytes into account when detecting slow tasks Key: FLINK-30707 URL: https://issues.apache.org/jira/browse/FLINK-30707 Project

[jira] [Created] (FLINK-30708) Typo problem of the doc `Determinism In Continuous Queries`

Jie Han created FLINK-30708: --- Summary: Typo problem of the doc `Determinism In Continuous Queries` Key: FLINK-30708 URL: https://issues.apache.org/jira/browse/FLINK-30708 Project: Flink Issue Type

[jira] [Created] (FLINK-30709) NetworkInput#emitNext() should push records to DataOutput in a while loop

Rui Fan created FLINK-30709: --- Summary: NetworkInput#emitNext() should push records to DataOutput in a while loop Key: FLINK-30709 URL: https://issues.apache.org/jira/browse/FLINK-30709 Project: Flink

[jira] [Created] (FLINK-30710) Fix invalid field id for nested type in spark catalog

Shammon created FLINK-30710: --- Summary: Fix invalid field id for nested type in spark catalog Key: FLINK-30710 URL: https://issues.apache.org/jira/browse/FLINK-30710 Project: Flink Issue Type: Bug

Re: Supplying jar stored at S3 to flink to run the job in kubernetes

Do you build your own flink-kubernetes-operator image with the flink-s3-fs plugin bundled[1]? [1]. https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.3/docs/custom-resource/overview/#flinksessionjob-spec-overview Best, Yang Weihua Hu 于2023年1月17日周二 10:47写道： > Hi, Rahul

[jira] [Created] (FLINK-30711) Support dropping partition via Flink action

yuzelin created FLINK-30711: --- Summary: Support dropping partition via Flink action Key: FLINK-30711 URL: https://issues.apache.org/jira/browse/FLINK-30711 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-30712) Update network memory configuration docs

Yuxin Tan created FLINK-30712: - Summary: Update network memory configuration docs Key: FLINK-30712 URL: https://issues.apache.org/jira/browse/FLINK-30712 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-287: Extend Sink#InitContext to expose ExecutionConfig and JobID

Hi Joao, Thanks for driving this FLIP, +1 for exposing a read-only ExecutionConfig, users/developers are not expected to modify the ExecutionConfig in Sink. Can we directly introduce the read-only ExecutionConfig in this FLIP? No need for a separate FLIP, because currently it will only be used in