Re: Flink job fails with org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

Thanks for reporting back. Piotrek > On 20 Feb 2020, at 21:29, John Smith wrote: > > I got this response on Stack: > https://stackoverflow.com/questions/60326869/what-does-kafka-consumer-too-many-open-files-mean/60327741#60327741 > >

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congratulations Jingsong! Well deserved. Best, Hequn On Fri, Feb 21, 2020 at 2:42 PM Yang Wang wrote: > Congratulations！Jingsong. Well deserved. > > > Best, > Yang > > Zhijiang 于2020年2月21日周五 下午1:18写道： > >> Congrats Jingsong! Welcome on board! >> >> Best, >> Zhijiang >> >> -

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congratulations！Jingsong. Well deserved. Best, Yang Zhijiang 于2020年2月21日周五 下午1:18写道： > Congrats Jingsong! Welcome on board! > > Best, > Zhijiang > > -- > From:Zhenghua Gao > Send Time:2020 Feb. 21 (Fri.) 12:49 > To:godfrey he >

Re: How JobManager and TaskManager find each other?

Hi Krzysztof, Xintong has share some information about the configuration. I just want to provide some additional information. >> How JobManager and TaskManager find each other? For non-HA Flink cluster, the taskmanager should have the pre-configured jobmanager address. For HA Flink cluster, the l

Re: Running Flink Cluster on PaaS

Hi KristoffSC, Currently, Flink could run in containerized environment and i think many users and companies deploy their production workloads on Kubernetes. For OpenShift, i think it is just like K8s. You could deploy standalone Flink session/per-job cluster by applying some yamls[1]. Also you cou

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong! Welcome on board! Best, Zhijiang -- From:Zhenghua Gao Send Time:2020 Feb. 21 (Fri.) 12:49 To:godfrey he Cc:dev ; user Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer Congrats Jingsong！ Best Reg

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong！ *Best Regards,* *Zhenghua Gao* On Fri, Feb 21, 2020 at 11:59 AM godfrey he wrote: > Congrats Jingsong! Well deserved. > > Best, > godfrey > > Jeff Zhang 于2020年2月21日周五 上午11:49写道： > >> Congratulations！Jingsong. You deserve it >> >> wenlong.lwl 于2020年2月21日周五 上午11:43写道： >> >>

【checkpoint】Flink 1.9 , checkpoint is declined for exception with message 'Pending record count must be zero at this point'

Hi all , may someone help me !! tks. The full exception as follows. > 2020-02-21 08:32:15,738 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Discarding > checkpoint 941 of job 0e16cf38a0bff313544e1f31d078f75b. > org.apache.flink.runtime.checkpoint.CheckpointException: C

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong! Well deserved. Best, godfrey Jeff Zhang 于2020年2月21日周五 上午11:49写道： > Congratulations！Jingsong. You deserve it > > wenlong.lwl 于2020年2月21日周五 上午11:43写道： > >> Congrats Jingsong! >> >> On Fri, 21 Feb 2020 at 11:41, Dian Fu wrote: >> >> > Congrats Jingsong! >> > >> > > 在 2020年2月21

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congratulations！Jingsong. You deserve it wenlong.lwl 于2020年2月21日周五 上午11:43写道： > Congrats Jingsong! > > On Fri, 21 Feb 2020 at 11:41, Dian Fu wrote: > > > Congrats Jingsong! > > > > > 在 2020年2月21日，上午11:39，Jark Wu 写道： > > > > > > Congratulations Jingsong! Well deserved. > > > > > > Best, > > > J

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congratulations Jingsong! Well deserved. Best, Xingbo wenlong.lwl 于2020年2月21日周五 上午11:43写道： > Congrats Jingsong! > > On Fri, 21 Feb 2020 at 11:41, Dian Fu wrote: > > > Congrats Jingsong! > > > > > 在 2020年2月21日，上午11:39，Jark Wu 写道： > > > > > > Congratulations Jingsong! Well deserved. > > > > > >

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong! > 在 2020年2月21日，上午11:39，Jark Wu 写道： > > Congratulations Jingsong! Well deserved. > > Best, > Jark > > On Fri, 21 Feb 2020 at 11:32, zoudan wrote: > >> Congratulations! Jingsong >> >> >> Best, >> Dan Zou >>

Re: How JobManager and TaskManager find each other?

Hi Krzysztof, You can use the following configuration options to specify JM/TM addresses and ports. - jobmanager.rpc.address - jobmanager.rpc.port - taskmanager.host - taskmanager.rpc.port The configuration accepts both IP addresses and hostnames. If you have two Flink Clusters, you can specify

Re: How does Flink manage the kafka offset

Hi Benchao, Thanks a lot! Eleanore On Thu, Feb 20, 2020 at 4:30 PM Benchao Li wrote: > Hi Jin, > > See below inline replies: > > My understanding is, upon startup, Flink Job Manager will contact kafka to >> get the offset for each partition for this consume group, and distribute >> the task to

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congratulations Jingsong! Well deserved. Best, Jark On Fri, 21 Feb 2020 at 11:32, zoudan wrote: > Congratulations! Jingsong > > > Best, > Dan Zou >

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congratulations! Jingsong Best, Dan Zou

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong! On Fri, Feb 21, 2020 at 11:05 AM Kurt Young wrote: > Hi everyone, > > I'm very happy to announce that Jingsong Lee accepted the offer of the > Flink PMC to > become a committer of the Flink project. > > Jingsong Lee has been an active community member for more than a year now.

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congratulations！！ Jingsong. Best, Leonard Xu > 在 2020年2月21日，11:18，tison 写道： > > Congrats Jingsong, well deserved! > > Best, > tison. > > > Xintong Song mailto:tonysong...@gmail.com>> > 于2020年2月21日周五 上午11:09写道： > Congratulations, Jingsong. Well deserved~！ > > Thank you~ > Xintong Song > >

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong, well deserved! Best, tison. Xintong Song 于2020年2月21日周五 上午11:09写道： > Congratulations, Jingsong. Well deserved~！ > > > Thank you~ > > Xintong Song > > > > On Fri, Feb 21, 2020 at 11:05 AM Kurt Young wrote: > >> Hi everyone, >> >> I'm very happy to announce that Jingsong Lee a

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congratulations, Jingsong. Well deserved~！ Thank you~ Xintong Song On Fri, Feb 21, 2020 at 11:05 AM Kurt Young wrote: > Hi everyone, > > I'm very happy to announce that Jingsong Lee accepted the offer of the > Flink PMC to > become a committer of the Flink project. > > Jingsong Lee has been

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong, well deserved! Kurt Young 于2020年2月21日周五 上午11:05写道： > Hi everyone, > > I'm very happy to announce that Jingsong Lee accepted the offer of the > Flink PMC to > become a committer of the Flink project. > > Jingsong Lee has been an active community member for more than a year now.

[ANNOUNCE] Jingsong Lee becomes a Flink committer

Hi everyone, I'm very happy to announce that Jingsong Lee accepted the offer of the Flink PMC to become a committer of the Flink project. Jingsong Lee has been an active community member for more than a year now. He is mainly focus on Flink SQL, played an essential role during blink planner mergi

Re: [ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

Hi Chad, Beam Portability module solves the management of the Python runtime environment and the communication between the Python runtime environment Python VM and the Java runtime environment JVM in PyFlink. For details, you can refer to FLIP-58[1] and the Blog[2]. Hope to resolve your doubts. B

Re: How does Flink manage the kafka offset

Hi Jin, See below inline replies: My understanding is, upon startup, Flink Job Manager will contact kafka to > get the offset for each partition for this consume group, and distribute > the task to task managers, and it does not use kafka to manage the consumer > group. Generally, yes. If you a

Running Flink Cluster on PaaS

Hi all, are there any obstacles from running Flink Cluster on PaaS like OpenShift for example? Where for example, task manager could be reasigned to different physical box? Especially when Flink will be in form of Docker Job Cluster. Regards, Krzysztof -- Sent from: http://apache-flink-user-mai

How JobManager and TaskManager find each other?

Hi all, I was wondering how JobManager and TaskManager find each other? Do they use multicast for this? Can it be configure to use domain names instead IP's? What I have to do to have two Flink Clusters in same IP network? How I should start task manager in order to tell him, to connect to cluster

How does Flink manage the kafka offset

Hi there, We are running apache beam application with flink being the runner. We use the KafkaIO connector to read from topics: https://beam.apache.org/releases/javadoc/2.19.0/ and we have the following configuration, which enables auto commit of offset, no checkpointing is enabled, and it is pe

Re: Flink job fails with org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

I got this response on Stack: https://stackoverflow.com/questions/60326869/what-does-kafka-consumer-too-many-open-files-mean/60327741#60327741 On Thu, 20 Feb 2020 at 13:58, John Smith wrote: > Ok I have 9 jobs running over 3 nodes. Most jobs are set to 1 parallelism, > worst case 2. So let's ass

Re: Flink job fails with org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

Ok I have 9 jobs running over 3 nodes. Most jobs are set to 1 parallelism, worst case 2. So let's assume maximum parallelism would be 18. I will try increase the ulimit and hopefully, we wont see it... On Thu, 20 Feb 2020 at 04:56, Piotr Nowojski wrote: > But it could be Kafka’s client issue on

Re: JDBC source running continuously

https://stackoverflow.com/questions/48151881/how-to-run-apache-flink-streaming-job-continuously-on-flink-server On Thu, Feb 20, 2020 at 3:14 AM Chesnay Schepler wrote: > Can you show us where you found the suggestion to use iterate()? > > On 20/02/2020 02:08, Fanbin Bu wrote: > > Hi, > > > > My

Re: [DISCUSS] Drop Savepoint Compatibility with Flink 1.2

Thank you for the feedback. Here is the JIRA issue with some more explanation also about the background and implications: https://jira.apache.org/jira/browse/FLINK-16192 Best, Stephan On Thu, Feb 20, 2020 at 2:26 PM vino yang wrote: > +1 for dropping Savepoint compatibility with Flink 1.2 > >

Re: [ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

Hi, Very cool. I’m curious about the relationship between this feature and Apache Beam. What parts of Beam are used and for what? Does this have any impact on existing Beam users like myself who use the Beam python API on top of Flink? Can someone give me a brief overview or point me at the righ

Re: FlinkKinesisConsumer throws "Unknown datum type org.joda.time.DateTime"

What you could try is bumping the avro dependency to 1.9.2 . I don't whether this will /just work/, but it's worth a shot. On 20/02/2020 16:25, Lian Jiang wrote: Thanks. I see from mvnrepository that even Flink-Avro 1.10 (the latest) uses avro 1.8.2. Does this mean I have to use GenericData in

Re: FlinkKinesisConsumer throws "Unknown datum type org.joda.time.DateTime"

Thanks. I see from mvnrepository that even Flink-Avro 1.10 (the latest) uses avro 1.8.2. Does this mean I have to use GenericData instead of avro POJOs if I use FlinkKinesisConsumer? Sent from my iPhone > On Feb 20, 2020, at 4:34 AM, Chesnay Schepler wrote: >

Re: Writing a POJO Schema Evolution E2E test in Java

Hi Theo, This is indeed a tricky feature to test! On Thu, Feb 20, 2020 at 8:59 PM Theo Diefenthal < theo.diefent...@scoop-software.de> wrote: > Hi, > > We have a pipeline which internally uses Java POJOs and also needs to keep > some events entirely in state for some time. > > From time to time,

Re: [ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

Thanks a lot for the release. Great Work, Jincheng! Also thanks to participants who contribute to this release. Best, Xingbo Till Rohrmann 于2020年2月18日周二 下午11:40写道： > Thanks for updating the 1.9.2 release wrt Flink's Python API Jincheng! > > Cheers, > Till > > On Thu, Feb 13, 2020 at 12:25 PM H

Re: [DISCUSS] Drop Savepoint Compatibility with Flink 1.2

+1 for dropping Savepoint compatibility with Flink 1.2 Flink 1.2 is quite far away from the latest 1.10. Especially after the release of Flink 1.9 and 1.10, the code and architecture have undergone major changes. Currently, I am updating state migration tests for Flink 1.10. I can still see some

Re: [DISCUSS] Drop Savepoint Compatibility with Flink 1.2

+1 for dropping compatibility. I personally think that it is very important for a project to keep a good pace in developing that old legacy stuff must be dropped from time to time. As long as there is an upgrade routine (via going to another flink release) that's fine. Von: "Stephan Ewen"

Writing a POJO Schema Evolution E2E test in Java

Hi, We have a pipeline which internally uses Java POJOs and also needs to keep some events entirely in state f or some time . >From time to time, our POJOs evolve, like attributes are added or removed. Now I wanted to write a E2E test that proves the schema migration works (Having different

Re: AWS Client Builder with default credentials

First things first, we do not intend for users to use anything in the S3 filesystem modules except the filesystems itself, meaning that you're somewhat treading on unsupported ground here. Nevertheless, the S3 modules contain a large variety of AWS-provided CerentialsProvider implementations, t

Re: FlinkKinesisConsumer throws "Unknown datum type org.joda.time.DateTime"

From the stack trace it appears that flink-avro is used, which uses avro 1.8.2 internally by default, for which this appears to be a known issue that was fixed in 1.9.?. Are you sure that avro 1.9.2 is actually being used? On 19/02/2020 23:53, Lian Jiang wrote: Hi, I use a FlinkKinesisConsume

AWS Client Builder with default credentials

I'm using org.apache.flink.fs.s3base.shaded.com.amazonaws.client.builder.AwsClientBuilder to create a S3 client to copy objects and delete object inside a TwoPhaseCommitSinkFunction. Shouldn't be another way to set up configurations without put them hardcoded ? Something like core-site.xml or flin

Re: build error for flink 1.10

MapR has a known problem with their HTTPS repository where certificates are being rejected in some (not well-defined) cases. The only reliable workaround is to work against the HTTP repository. Do also note that the vendor-repos profile no longer exists in 1.10; additional repositories must be

build error for flink 1.10

Hi Flink team, When I tried to build flink 1.10 source code, I always got the following build error: [ERROR] Failed to execute goal on project flink-connector-hive_2.11: Could not resolve dependencies for project org.apache.flink:flink-connector-hive_2.11:jar:1.10.0: Failed to collect dependen

Re: Emit message at start and end of event time session window

I think one "edge" case which is not handled would be that the first event (by event-time) arrives late, then a wrong "started-window" would be reported. Rafi On Thu, Feb 20, 2020 at 12:36 PM Manas Kale wrote: > Is the reason ValueState cannot be use because session windows are always > formed

Re: JDBC source running continuously

Can you show us where you found the suggestion to use iterate()? On 20/02/2020 02:08, Fanbin Bu wrote: Hi, My app creates the source from JDBC inputformat and running some sql and print out. But the source terminates itself after the query is done. Is there anyway to keep the source running?

Re: Side Outputs from RichAsyncFunction

I don't think this is possible. At the very least you should be able to workaround this by having your AsyncFunction return an Either, and having a subsequent ProcessFunction do the side-output business. On 19/02/2020 22:25, KristoffSC wrote: Hi, any thoughts about this one? Regards, Krzysz

Re: Tests in FileUtilsTest while building Flink in local

Is the stacktrace identical in both tests? Did these fail on the command-line or in the IDE? Can you check what directory the java.io.tmpdir points to? On 19/02/2020 20:42, Arujit Pradhan wrote: Hi all, I was trying to build Flink in my local machine and these two unit tests are failing. *

Re: Emit message at start and end of event time session window

Is the reason ValueState cannot be use because session windows are always formed by merging proto-windows of single elements, therefore a state store is needed that can handle merging. ValueState does not provide this functionality, but a ReducingState does? On Thu, Feb 20, 2020 at 4:01 PM Manas K

Re: Emit message at start and end of event time session window

Hi Till, Thanks for your answer! You also answered the next question that I was about to ask "Can we share state between a Trigger and a Window?" Currently the only (convoluted) way to share state between two operators is through the broadcast state pattern, right? Also, in your example, why can't

[DISCUSS] Extend (or maintain) "shell" script support for Windows

Hi Everyone, I started a discussion on the dev mailing list about retiring the windows .bat scripts: https://lists.apache.org/thread.html/r7693d0c06ac5ced9a34597c662bcf37b34ef8e799c32cc0edee373b2%40%3Cdev.flink.apache.org%3E Please have a look at my initial e-mail there and let us know if thi

[DISCUSS] Drop Savepoint Compatibility with Flink 1.2

Hi all! For some cleanup and simplifications, it would be helpful to drop Savepoint compatibility with Flink version 1.2. That version was released almost three years ago. I would expect that no one uses that old version any more in a way that they actively want to upgrade directly to 1.11. Even

Re: Flink job fails with org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

But it could be Kafka’s client issue on the Flink side (as the stack trace is suggesting). You can just try to increase limit of opened files for Flink, or try to identify who is opening all of those files and limit it somehow - if it’s Kafka client indeed, maybe it can be configured to use less

Re: Identifying Flink Operators of the Latency Metric

Hey Morgan, I would query the Monitoring REST API: https://ci.apache.org/projects/flink/flink-docs-master/monitoring/rest_api.html For example: GET http://localhost:8082/jobs/9a6748889bf24987495eead247aeb1ff Returns: 1. {jid: "9a6748889bf24987495eead247aeb1ff", name: "CarTopSpeedWindowingE