Auditing sink using table api

2023-11-03 Thread Bo
Hello community, I am looking for a way to perform auditing of the various sinks (mostly JdbcDynamicTableSink) using the table api. By "auditing", I mean to log details of every row data coming into the sink, and any anormalies when the sink write to external systems. Does flink have some kin

Re: Inquiry about ActiveResourceManager and StandaloneResourceManager in Flink

2023-11-03 Thread Yu Chen
Hi Steven, As stated in the `StandaloneResourceManager` comments, the manager does not acquire new resources and the user needs to manually start the Taskmanager by themself. While `ActiveResourceManager` achieves requesting or releasing resources on demand(that's what active means) based on some

Re: Flink custom parallel data source

2023-11-03 Thread David Anderson
> As you suggested message broker below then how it is feasible in this case? To my mind, the idea would be to use something like a socket source for Kafka Connect. This would give you a simple, reliable way to get the data stored into a replayable data store. You'd then be able to start, stop, an

How to tell if job is being restarted in log?

2023-11-03 Thread John Smith
Hi I'm getting metaspace issues and I understand that certain libraries like JDBC don't unload properly and we need to put them in the global class path of flink. But technically my jobs should not be restarting, so what can I look for in the logs to see when the restart?

Re: flink-kubernetes-operator cannot handle SPECCHANGE for 100+ FlinkDeployments concurrently

2023-11-03 Thread Nicolas Fraison via user
Hi, We have faced a similar issue with flink kubernetes operator, having multiple operators running as leader at the same time. On our side the issue was on the java-operator-sdk which was not well killing operator that lost leadership ( https://github.com/operator-framework/java-operator-sdk/issu

Re: flink-kubernetes-operator cannot handle SPECCHANGE for 100+ FlinkDeployments concurrently

2023-11-03 Thread Evgeniy Lyutikov
Hello! I constantly get a similar error when operator (working in single instance) receiving deployment statuses Details described in this message https://lists.apache.org/thread/0odcc9pvlpz1x9y2nop9dlmcnp9v1696 I tried changing versions and allocated resources, as well as the number of reconci