flink connector hive support collect metrics

Dear all, I am using the flink-connector-hive to sync records into hive table. However, the sink conector doesn't collect operator running metrics like numRecordsIn. so I couldn't collect these data. Is there a way to support the flink-connector-hive to collect flink metrics? Or is there some m

Re: "Not all slot managed memory is freed"

Thanks, Gabor! We switched back to the hashmap state backend for now. We'll troubleshoot if we go back to rocksdb. Jad On Wed, Nov 6, 2024 at 6:51 AM Gabor Somogyi wrote: > Hi Jad, > > There is no low hanging fruit here if you really want to find this out. > Such case the memory manager tries t

Re: "Not all slot managed memory is freed"

Hi Jad, There is no low hanging fruit here if you really want to find this out. Such case the memory manager tries to allocate and deallocate the total memory which is prepared for. When not all the memory is available then it's not going to be successful and you see the mentioned exception. I wo

Flink BigQuery Connector Public Preview

Greetings, The Google BigQuery team is reaching out to help customers employ the Flink BigQuery connector for building better data streaming and analytics solutions. We recently released a new version (0.4.0), which can be accessed via GitHub

Calling 3rd party JARs from Pyflink

Hello, I want to read data from Kafka topics which have Confluent-encoded Protobuf messages (not plain Protobuf, Confluent adds a "magic byte" and schema ID) with Python Datastream API. I have found that Confluent has a Java class[1] which implements org.apache.kafka.common.serialization.Deseriali

Java 21 support and apache/flink docker images

Hi all, What is the status of Apache Flink Java 21 support and apache/flink images? I am currently using Java 17 in production but would like to update to Java 21 if possible. For the 1.19 release I can see that it was announced that Java 21 has beta support. However, it looks like the associated

Re: [ANNOUNCE] Apache Flink 2.0 Preview released

Clear, thank you Zakelly Le mer. 6 nov. 2024 à 09:16, Zakelly Lan a écrit : > Hi Benoit, > > Please find the result here[1]. > > The Nexmark repo[2] does not officially support the flink 2.0 preview > version. However, we have made a PR[3] for this and once it is merged, we > will offer a guide

Re: [DISCUSS] Is it a bug that the AdaptiveScheduler does not prioritize releasing TaskManagers during downscaling in Application mode?

Thanks Yuepeng for the PR and starting this discussion! And thanks Gyula and Yuanfeng for the input! I also agree to fix this behaviour in the 1.x line. The adaptive scheduler and rescaling API provide powerful capabilities to increase or decrease parallelism. The main benefit I understand of d

Re: [ANNOUNCE] Apache Flink 2.0 Preview released

Hi Benoit, Please find the result here[1]. The Nexmark repo[2] does not officially support the flink 2.0 preview version. However, we have made a PR[3] for this and once it is merged, we will offer a guide to run Nexmark Q20 with disaggregated state management. [1] https://github.com/ververica/

Re: [DISCUSS] Is it a bug that the AdaptiveScheduler does not prioritize releasing TaskManagers during downscaling in Application mode?

> Is it considered an error if the adaptive scheduler fails to release the task manager during scaling? +1 . When we enable adaptive mode and perform scaling operations on tasks, a significant part of the goal is to reduce resource usage for the tasks. However, due to some logic in the adaptive sc