Re: Streaming data from AWS S3 Source

Hi Neelabh You can use FileSystem Connector . for DataStream [1] for TableAPI [2] . And you need to put necessary dependency to your flink environment. [3] For Flink SQL setup, you can reference `sql getting started module`[4] [1]. https://nightlies.apache.org/flink/flink-docs-master/docs/c

Re: [VOTE] Release 1.16.3, release candidate #1

+1(not-binding) - verified signatures hash - verified hash - checked the tag - checked the repo - verified there are no binaries in the source archive - built from sources Best regards, Jing On Thu, Nov 23, 2023 at 7:44 PM Feng Jin wrote: > +1(non-binding) > > - verified signatures and hashsum

Re: [VOTE] Release 1.17.2, release candidate #1

+1(not-binding) - verified signatures and hash - reviewed the web PR - checked the tag - verified there are no binaries in the source archive - built from sources Best regards, Jing On Fri, Nov 24, 2023 at 4:07 AM weijie guo wrote: > Sorry, it's a non-binding vote. :) > > Best regards, > > Wei

Streaming data from AWS S3 Source

Hey Team, I want to stream data from AWS S3 source as they are being generated by an event to Apache Flink Stream for a data transformation job. I found out about FileSystem SQL Connector but need some ref

回复: Re:Re: [DISCUSS][FLINK-32993] Datagen connector handles length-constrained fields according to the schema definition by default

Thank Jane for providing examples to make discussions clearer. Thank Lincoln and Xuyang for your feedback,I agree with you wholeheartedly that it is better to throw an error instead of ignoring it directly. Extending datagen to generate variable length values is really an excelent idea, I will cr

Re: [VOTE] Release 1.17.2, release candidate #1

Sorry, it's a non-binding vote. :) Best regards, Weijie weijie guo 于2023年11月24日周五 11:05写道： > +1 (binding) > > - Verified signatures and hashsums > - Verified POM/NOTICE filess > - Built from sources. > - Reviewed the web PR. > > Best regards, > > Weijie > > > Yuxin Tan 于2023年11月24日周五 10:18写道

Re: [VOTE] Release 1.17.2, release candidate #1

+1 (binding) - Verified signatures and hashsums - Verified POM/NOTICE filess - Built from sources. - Reviewed the web PR. Best regards, Weijie Yuxin Tan 于2023年11月24日周五 10:18写道： > +1 (non-binding) > > - Verified signatures and hashsums > - Verified there are no binaries in the source archive

[jira] [Created] (FLINK-33636) Implement JdbcAutoScalerEventHandler

Rui Fan created FLINK-33636: --- Summary: Implement JdbcAutoScalerEventHandler Key: FLINK-33636 URL: https://issues.apache.org/jira/browse/FLINK-33636 Project: Flink Issue Type: Sub-task Com

[jira] [Created] (FLINK-33635) ElasticSearch connector main branch can not compile

Weijie Guo created FLINK-33635: -- Summary: ElasticSearch connector main branch can not compile Key: FLINK-33635 URL: https://issues.apache.org/jira/browse/FLINK-33635 Project: Flink Issue Type: B

Re: [VOTE] Release 1.17.2, release candidate #1

+1 (non-binding) - Verified signatures and hashsums - Verified there are no binaries in the source archive - Built Flink from sources - Reviewed the web PR Best, Yuxin Feng Jin 于2023年11月24日周五 02:20写道： > +1(non-binding) > > - verified signatures and hashsums > - Verified there are no binaries

Re: [VOTE] Release flink-connector-opensearch v1.1.0, release candidate #1

Hi Danny, Thanks for starting a RC for this. >From the looks of the staged POMs for 1.1.0-1.18, the flink versions for Flink dependencies still point to 1.17.1. My understanding is that this is fine, as those provided scope dependencies (e.g. flink-streaming-java) will have their versions overwr

Re: [VOTE] Release flink-connector-opensearch v1.1.0, release candidate #1

+1 (non-binding) - downloaded artifacts - built from source - verified checksums and signatures - reviewed web pr On Mon, Nov 6, 2023 at 5:31 PM Ryan Skraba wrote: > Hello! +1 (non-binding) Thanks for the release! > > I've validated the source for the RC1: > * flink-connector-opensearch-1.1.0-

Re: [VOTE] Release 1.16.3, release candidate #1

+1(non-binding) - verified signatures and hashsums - Verified there are no binaries in the source archive - Built Flink from sources - Started local standalone cluster, submitted sql job using sql-client Best, Feng On Thu, Nov 23, 2023 at 11:00 PM Hang Ruan wrote: > +1 (non-binding) > > - veri

Re: [VOTE] Release 1.17.2, release candidate #1

+1(non-binding) - verified signatures and hashsums - Verified there are no binaries in the source archive - Built Flink from sources - Started local standalone cluster, submitted sql job using sql-client Best, Feng On Thu, Nov 23, 2023 at 11:08 PM Hang Ruan wrote: > +1 (non-binding) > > - ver

Re: [VOTE] Release 1.17.2, release candidate #1

+1 (non-binding) - verified signatures - verified hashsums - Verified there are no binaries in the source archive - reviewed the web PR - built Flink from sources Best, Hang Jiabao Sun 于2023年11月23日周四 22:09写道： > Thanks for driving this release. > > +1(non-binding) > > - Checked the tag in git >

Re: [VOTE] Release 1.16.3, release candidate #1

+1 (non-binding) - verified signatures - verified hashsums - Verified there are no binaries in the source archive - reviewed the Release Note - reviewed the web PR Best, Hang Jiabao Sun 于2023年11月23日周四 22:15写道： > Thanks for driving this release. > > +1 (non-binding) > > - Checked the tag in git

[jira] [Created] (FLINK-33634) Add Conditions to Flink CRD's Status field

Tony Garrard created FLINK-33634: Summary: Add Conditions to Flink CRD's Status field Key: FLINK-33634 URL: https://issues.apache.org/jira/browse/FLINK-33634 Project: Flink Issue Type: Improv

[jira] [Created] (FLINK-33633) Automatic creation of RBAC for instances of Flink Deployments

Tony Garrard created FLINK-33633: Summary: Automatic creation of RBAC for instances of Flink Deployments Key: FLINK-33633 URL: https://issues.apache.org/jira/browse/FLINK-33633 Project: Flink

[jira] [Created] (FLINK-33632) Add custom mutator plugin

Tony Garrard created FLINK-33632: Summary: Add custom mutator plugin Key: FLINK-33632 URL: https://issues.apache.org/jira/browse/FLINK-33632 Project: Flink Issue Type: Improvement C

Re: [VOTE] Release 1.16.3, release candidate #1

Thanks for driving this release. +1 (non-binding) - Checked the tag in git - Verified signatures and hashsums - Verified there are no binaries in the source archive - Built Flink from sources Best, Jiabao > 2023年11月23日 21:55，Sergey Nuyanzin 写道： > > +1 (non-binding) > > - Downloaded artifacts

Re: [VOTE] Release 1.17.2, release candidate #1

Thanks for driving this release. +1(non-binding) - Checked the tag in git - Verified signatures and hashsums - Verified there are no binaries in the source archive - Built Flink from sources Best, Jiabao > 2023年11月21日 20:46，Matthias Pohl 写道： > > +1 (binding) > > * Downloaded artifacts > * B

Re: [VOTE] Release 1.16.3, release candidate #1

+1 (non-binding) - Downloaded artifacts - Built Flink from sources - Verified checksums & signatures - Verified pom/NOTICE files - reviewed the web PR On Thu, Nov 23, 2023 at 1:28 PM Leonard Xu wrote: > +1 (binding) > > - verified signatures > - verified hashsums > - checked that all POM files

Discussion: [FLINK-33609] Take into account the resource limit specified in the pod template

Hello everyone, I've encountered an issue while using the flink open source kubernetes operator for Flink deployment. Despite setting resource limits in the pod template, it appears that these limits are not considered during TaskManager (TM) pod deployment. Upon code investigation, it seems the l

[jira] [Created] (FLINK-33631) Remove HA ConfigMap label setup

Matthias Pohl created FLINK-33631: - Summary: Remove HA ConfigMap label setup Key: FLINK-33631 URL: https://issues.apache.org/jira/browse/FLINK-33631 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-33630) CoordinationResponse should be wrapped by SerializedValue in TaskOperatorEventGateway and JobMasterOperatorEventGateway

Qingsheng Ren created FLINK-33630: - Summary: CoordinationResponse should be wrapped by SerializedValue in TaskOperatorEventGateway and JobMasterOperatorEventGateway Key: FLINK-33630 URL: https://issues.apache.org/

Re: [VOTE] Release 1.16.3, release candidate #1

+1 (binding) - verified signatures - verified hashsums - checked that all POM files point to the same version 1.16.3 - started SQL Client, used MySQL CDC connector to read changelog from database , the result is expected - reviewed the web PR, left minor comment Best, Leonard > 2023年11月21日 下午

Re: [DISCUSS] Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

We had a longer discussion with Gordon yesterday. The main conclusion was that moving to a completely new interface is not justified, and we try to improve the current one. Another ask from Gordon was to check when the user will be notified if the parameter types are incorrect using the mixin appr

Re: [VOTE] Release 1.17.2, release candidate #1

+1 (non-binding) - Downloaded artifacts - Built Flink from sources - Verified checksums & signatures - Verified pom/NOTICE files - reviewed the web PR On Thu, Nov 23, 2023 at 12:49 PM Leonard Xu wrote: > +1 (binding) > > - verified signatures > - verified hashsums > - checked all dependency art

Re: [VOTE] Release 1.17.2, release candidate #1

+1 (binding) - verified signatures - verified hashsums - checked all dependency artifacts are 1.17.2 - started SQL Client, used MySQL CDC connector to read changelog from database , the result is expected - reviewed the web PR Best, Leonard > 2023年11月22日 下午11:13，Danny Cranmer 写道： > > Thanks

RE: Apicurio Avro format proposal

Hi Ryan, I am reasonable new it this, but here Is my understanding. If we use pure Avro and Flink SQL, when we create a table – the shape of that table is the shape we expect the event to be. This falls down when we evolve the schema, i.e. create new versions of the schema. The new versions need

[jira] [Created] (FLINK-33629) Introduce ArchTest rules to ban Junit 4 for modules already completed the migration

Jiabao Sun created FLINK-33629: -- Summary: Introduce ArchTest rules to ban Junit 4 for modules already completed the migration Key: FLINK-33629 URL: https://issues.apache.org/jira/browse/FLINK-33629 Proje

[jira] [Created] (FLINK-33628) Upgrading Flink's default ZooKeeper version to 3.8.3

Matthias Pohl created FLINK-33628: - Summary: Upgrading Flink's default ZooKeeper version to 3.8.3 Key: FLINK-33628 URL: https://issues.apache.org/jira/browse/FLINK-33628 Project: Flink Issue

Re: [DISCUSS] Towards flink-shaded release 18.0

+1 Thanks for driving this, Sergey. We could also think of removing the ZooKeeper versions 3.5 and 3.6, I guess. 3.6 reached EOL in 2022. Flink 1.17 already switched to 3.7 as the default version. On Thu, Nov 23, 2023 at 9:31 AM Jing Ge wrote: > +1 > > Does it make sense to use 32.1.3-jre Guava

Re: Apicurio Avro format proposal

Pardon me, I forgot to include that I'd seen this before as FLINK-26654. There's a linked JIRA with an open PR that kind of *plugs in* 8-byte ids . I haven't had the chance to check out Apicurio yet, but I'm interested in schema registries in general. All my best, Ryan [1]: https://github.com/ap

Re: Apicurio Avro format proposal

Hello David! In the FLIP, I'd be interested in knowing how the avro-apicurio and avro-confluent formats would differ! Outside of configuration options, are there different features? Would the two schema registry formats have a lot of common base that we could take advantage of? All my best, Rya

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

Btw. even though we've been focusing on GitHub Actions with this FLIP, I'm curious whether somebody has experience with Apache Infra's Jenkins deployment. The discussion I found about Jenkins [1] is quite out-dated (2014). I haven't worked with it myself but could imagine that there are some featur

RE: Apicurio Avro format proposal

Hi Martijn, Ok will do, Kind regards, David. From: Martijn Visser Date: Wednesday, 22 November 2023 at 21:47 To: dev@flink.apache.org Subject: [EXTERNAL] Re: Apicurio Avro format proposal Hi David, Can you create a small FLIP for this? Best regards, Martijn On Wed, Nov 22, 2023 at 6:46

Re: [DISCUSS] REST API behaviour when user main method throws error

Hey David, Thanks for the feedback. > For streaming, this is probably something we could address by book-keeping jobs submitted by the jar and canceling them on exception. I considered two options. 1/ cancel on error as you proposed. This has the downside that it could start processing data temp

[jira] [Created] (FLINK-33627) Bump snappy-java to 1.1.10.4 in flink-statefun

Ryan Skraba created FLINK-33627: --- Summary: Bump snappy-java to 1.1.10.4 in flink-statefun Key: FLINK-33627 URL: https://issues.apache.org/jira/browse/FLINK-33627 Project: Flink Issue Type: Bug

Re: [VOTE] FLIP-386: Support adding custom metrics in Recovery Spans

+1 (binding) Regards, Roman On Wed, Nov 22, 2023 at 12:55 PM Rui Fan <1996fan...@gmail.com> wrote: > +1(binding) > > Thanks for driving this proposal! > > Best, > Rui > > On Wed, Nov 22, 2023 at 7:44 PM Piotr Nowojski > wrote: > > > Hi All, > > > > I'd like to start a vote on the FLIP-386: Su

Re: [DISCUSS] Towards flink-shaded release 18.0

+1 Does it make sense to use 32.1.3-jre Guava? Best regards, Jing On Thu, Nov 23, 2023 at 8:42 AM Yun Tang wrote: > +1, thanks Sergey for driving this work. > > Best > Yun Tang > > From: Leonard Xu > Sent: Thursday, November 23, 2023 9:38 > To: dev > Subject:

[jira] [Created] (FLINK-33626) Wrong style in flink ui

Fang Yong created FLINK-33626: - Summary: Wrong style in flink ui Key: FLINK-33626 URL: https://issues.apache.org/jira/browse/FLINK-33626 Project: Flink Issue Type: Bug Components: Travi

[jira] [Created] (FLINK-33625) FLIP-390: Support System out and err to be redirected to LOG or discarded

Rui Fan created FLINK-33625: --- Summary: FLIP-390: Support System out and err to be redirected to LOG or discarded Key: FLINK-33625 URL: https://issues.apache.org/jira/browse/FLINK-33625 Project: Flink