Rolling K8s without Flink disruptions

2023-11-29 Thread Salva Alcántara
Are there any special precautions that need to be taken for undergoing regular K8s maintenance procedures such as migrating/upgrading clusters? For the sake of concreteness, I'm running my jobs via the Flink K8s Operator and I'm finding that when rolling out new nodes and migrating my jobs to them

Re: How to call the open method of JdbcSink?

2023-11-29 Thread Sai Vishnu
Hi Feng, Thank you for your response and the suggestion. I was able to cast the SinkFunction to GenericJdbcSinkFunction which enabled me to override the open and close methods. On proceeding further, I have observed that if the batch encounters BatchUpdateException due to a packet inside the batch

[ANNOUNCE] Apache Flink 1.16.3 released

2023-11-29 Thread Rui Fan
The Apache Flink community is very happy to announce the release of Apache Flink 1.16.3, which is the third bugfix release for the Apache Flink 1.16 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

Re: [EXTERNAL] TaskManagers Crushing

2023-11-29 Thread Ivan Webber via user
Were you ever able to find a workaround for this? I also have transient failures due to org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException. From: Kenan Kılıçtepe Sent: Saturday, August 19, 2023 5:50 PM To: user@flink.apache.org Subject:

Re: Doubts about state and table API

2023-11-29 Thread Matthias Pohl via user
Hi Oscar, could you provide the Java code to illustrate what you were doing? The difference between version A and B might be especially helpful. I assume you already looked into the FAQ about operator IDs [1]? Adding the JM and TM logs might help as well to investigate the issue, as Yu Chen mentio

Re: Java 17 as default

2023-11-29 Thread Matthias Pohl via user
The 1.18 Docker images were pushed on Oct 31. This also included Java 17 images [1]. [1] https://hub.docker.com/_/flink/tags?page=1&name=java17 On Wed, Nov 15, 2023 at 7:56 AM Tauseef Janvekar wrote: > Dear Team, > > I saw the documentation for 1.18 and Java 17 is not supported and the > image

Having Problems as beginners using Flink in our Data Engineering internship

2023-11-29 Thread ALIZOUAOUI
Hey, I hope y'all are doing well, i started a project (4 months internship in Data Engineering) it's a totally new field for me and i have to present 2 months from now, i took with Java (i have no prior experience using it) and i started with Flink documentation, we have to follow the operation

Flink K8s operator ignores taskmanager.host setting

2023-11-29 Thread Salva Alcántara
Hi! I'm deploying a job via the Flink K8s Operator with these settings in the FlinkDeployment resource: ``` spec: flinkConfiguration: taskmanager.host: 0.0.0.0 <-- ignored / not applied ``` When I look into the flink-conf.yaml in the TM the setting is not there. Is there any reason wh

RE: Re: Re: inputQueueSize metric

2023-11-29 Thread Dimitris Banelas via user
Thank you for your previous response Feng. It clears things up a little. Regarding the inputQueueSize metric. What does it express after all? Observing the inputQueueSize and inPoolUsage metrics in Prometheus, I found that when the inPoolUsage is high, the inputQueueSize is low. This observati