Cannot find JDBC Postgres driver when running a job more than once

Hi All, I'm new to Flink and have been testing version 1.14. (My issue also happens in the previous version.) When I run a job once it works fine. However when I submit it again and try to run it I get a 'No suitable driver found' exception. The only way I have found to rerun my job is to restart

Re: RocksDB: Spike in Memory Usage Post Restart

We tried with 1.14.0, unfortunately we still run into the issue. Any thoughts or suggestions? On Mon, Oct 4, 2021 at 9:09 AM Kevin Lam wrote: > Hi Fabian, > > We're using our own image built from the official Flink docker image, so > we should have the code to use jemalloc in the docker entrypoi

asyhcnrouonous io question

Hello, I have a recently ran into an issue with RichAsyncFunction and wanted to get some guidance from the community Please see snippet class AsyncFetchFromHttp extends RichAsyncFunction> {23    private transient AysncHttpClient client;45    @Override6    public void

Re: Flink application mode with no ui , how to start job using k8s ?

Thanks Dawid,       if I am not exposing UI , how I am going to run my job we need to submit jar someway, I do not want my flink image tightly coupled with my jar  On Monday, October 4, 2021, 09:52:31 AM EDT, Dawid Wysakowicz wrote: Hi Dhiru, For the question about auto scaling I'

Re: Error: Timeout of 60000ms expired before the position for partition

Hi, Do you mean that you fail to start Kafka? Or do you get the exception from Flink. Could you please share the full stack trace of the error? Best, Dawid On 02/10/2021 16:58, Dipanjan Mazumder wrote: > Hi, > >   I am getting below error while starting the flink as a standalone > single jvm pr

Re: Exception thrown during batch job execution on YARN even though job succeeded

Hey Ken, Regarding Rufus, I know he might be a bit eager in changing lines ;) If you want to ignore his changes in git blame, please take a look here[1]. For the main issue, do you mind creating a ticket? I hope someone will be able to pick it up. Best, Dawid [1] https://nightlies.apache.org/

Re: Flink application mode with no ui , how to start job using k8s ?

Hi Dhiru, For the question about auto scaling I'd recommend you this[1] blogpost from my colleague. I believe he explains it quite well how to do it. Besides that I am not sure what is your other question. Are you asking how to start the jobmanager without the UI? Can't you just simply not expose

Re: Missing dependency in flink-shaded-zookeeper-35

Thanks! From: Chesnay Schepler Date: Monday, October 4, 2021 at 9:27 AM To: Peter Westermann , user Subject: Re: Missing dependency in flink-shaded-zookeeper-35 Indeed, it looks like the client-server SSL support added in 3.5 is implemented with netty. I will create a ticket. On 04/10/2021 15:

Re: Missing dependency in flink-shaded-zookeeper-35

Indeed, it looks like the client-server SSL support added in 3.5 is implemented with netty. I will create a ticket. On 04/10/2021 15:08, Peter Westermann wrote: I’ve been working on migrating our Flink cluster to zookeeper SSL (this on Flink 1.12) but ran into a missing dependency issue. St

Re: RocksDB: Spike in Memory Usage Post Restart

Hi Fabian, We're using our own image built from the official Flink docker image, so we should have the code to use jemalloc in the docker entrypoint. I'm going to give 1.14 a try and will let you know how it goes. On Mon, Oct 4, 2021 at 8:29 AM Fabian Paul wrote: > Hi Kevin, > > We bumped the

Missing dependency in flink-shaded-zookeeper-35

I’ve been working on migrating our Flink cluster to zookeeper SSL (this on Flink 1.12) but ran into a missing dependency issue. Steps to reproduce: * Replace flink-shaded-zookeeper-3.4.14.jar in the lib folder with flink-shaded-zookeeper-3.5.6.jar from the opt folder * Add zookeeper SSL

Re: In flight records on Flink : Newbie question

Hi Fabian I am currently using the streaming file sink to local disk though potentially this sink could change to be Elastic Search. Declan On Mon, Oct 4, 2021 at 1:16 PM Fabian Paul wrote: > Hi Declan, > > I forgot to ask which sink you are using. I do not think it is generally > applicable t

Re: RocksDB: Spike in Memory Usage Post Restart

Hi Kevin, We bumped the RocksDb version with Flink 1.14 which we thought increases the memory control [1]. In the past we also saw problems with the allocator used of the OS. We switched to use jemalloc within our docker images which has a better memory fragmentation [2]. Are you using the offi

Re: In flight records on Flink : Newbie question

Hi Declan, I forgot to ask which sink you are using. I do not think it is generally applicable that all sinks buffer records and only send them periodically. It depends a lot on the connector and what kind of capabilities the external system you are writing to offers. The amount of buffered da

Re: Flink - dealing with missing events in keyBy window

Hey Yun, thanks for your response. This method can work, but it has one problem I still have to deal with: until the sensor is returning it's initial data, the stream won't process events and the onTimer function won't be called. I need to know if the sensor is starting in 'off' state. I thought o

Kubernetes HA - Reusing storage dir for different clusters

Hello, If I deploy a Flink-Kubernetes application with HA, I need to set high-availability.storageDir. If my application is a batch job that may run multiple times with the same configuration, do I need to manually clean up the storage dir between each execution? Regards, Alexis.