Re: Jobmanager trying to be registered for Zombie Job

Hi Peter, thanks for sharing. That doesn't sound right. May you provide the entire jobmanager logs? Best, Matthias On Thu, Apr 21, 2022 at 6:08 PM Peter Schrott wrote: > Hi Flink-Users, > > I am not sure if this does something to my cluster or not. But since > updating to Flink 1.15 (atm rc4) I

Re: [ANNOUNCE] Apache Kyuubi (Incubating) released 1.5.1-incubating

Hi Fu, I think the mail might have been sent to Flink user mail list by mistake? Best, Paul Lam > 2022年4月22日 11:00，Fu Chen 写道： > > Hi all, > > The Apache Kyuubi (Incubating) community is pleased to announce that > Apache Kyuubi (Incubating) 1.5.1-incubating has been released! > > Apache Kyuu

Flink batch mode does not sort by event timestamp

I have a custom flink Source, and I have a SerializableTimestampAssigner that assigns event timestamps to records emitted by the source. The source may emit records out of order because of the nature of the underlying data storage, however with BATCH mode, I expect Flink to sort these records by

RE: DebeziumAvroDeserializationSchema

Wonder if this is a bug or not but if I use AvroRowDeserializationSchema,In PyFlink the error still occure ?py4j.protocol.Py4JError: An error occurred while calling None.org.apache.flink.formats.avro.AvroRowDeserializationSchema. Trace:org.apache.flink.api.python.shaded.py4j.Py4JException: Construc

RE: AvroRowDeserializationSchema

Wonder if this is a bug or not but if I use AvroRowDeserializationSchema,In PyFlink the error still occure ?py4j.protocol.Py4JError: An error occurred while calling None.org.apache.flink.formats.avro.AvroRowDeserializationSchema. Trace:org.apache.flink.api.python.shaded.py4j.Py4JException: Construc

JobManager doesn't bring up new TaskManager during failure recovery

Hi developers! I got a strange bug during failure recovery of Flink. It seems the JobManager doesn't bring up new TaskManager during failure recovery. Some logs and information of the Flink job are pasted below. Can you take a look and give me some guidance? Thank you so much! Flink version: 1

UUID on TableAPI

Hi team,Currently, I want to use savepoints in Flink. However, one of the things that I concern is that is there any way we can set the UUID while using Table API (SQL API) ? If not, does it has any mechanism to know that when we start the Flink again, it will know that it was that UUID ?Best,Quynh

DebeziumAvroDeserializationSchema

Hi team,Currently, I did not see this functions in PyFlink, therefore any suggestion on using this on PyFlink ?Best,Quynh. Sent from Mail for Windows 

[ANNOUNCE] Apache Kyuubi (Incubating) released 1.5.1-incubating

Hi all, The Apache Kyuubi (Incubating) community is pleased to announce that Apache Kyuubi (Incubating) 1.5.1-incubating has been released! Apache Kyuubi (Incubating) is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark and designed

How to dynamically modify the schema information of a table

The following is the Java code @Test public void test(){ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env); DataStream dataStream = env.fromElements("Alice", "Bob", "John"); Schema.Builder b

Re: Huge number of GAX-Threads in Flink

Hi, Which connector are you using? Is the increase in the number of threads gradual or is it related to task failover? It would be helpful if you could provide a thread dump. > 2022年4月22日 上午2:12，SHREEKANT ANKALA 写道： > > Hi, Can somebody please help with this issue? This is really blocking us

Re: Integration Test for Kafka Streaming job

Thank you for information ! From: Farouk Sent: Thursday, April 21, 2022 1:14:00 AM To: Aeden Jameson Cc: Alexey Trenikhun ; Flink User Mail List Subject: Re: Integration Test for Kafka Streaming job Hi I would recommend to use kafka-junit5 from salesforce htt

Exception Handling in ElasticsearchSink

Hi all, I've recently been encountering some issues that I've noticed in the logs of my Flink job that handles writing to an Elasticsearch index. I was hoping to leverage some of the metrics that Flink exposes (or piggyback on them) to update metric counters when I encounter specific kinds of erro

RE: Huge number of GAX-Threads in Flink

Hi, Can somebody please help with this issue? This is really blocking us in Production. If needed, I can provide the thread dump as well. Thanks, Shreekant A. Sent from Mail for Windows From: SHREEKANT ANKALA Sent

Jobmanager trying to be registered for Zombie Job

Hi Flink-Users, I am not sure if this does something to my cluster or not. But since updating to Flink 1.15 (atm rc4) I find the following logs: INFO: Registering job manager ab7db9ff0ebd26b3b89c3e2e56684...@akka.tcp:// fl...@flink-jobmanager-xxx.com:40015/user/rpc/jobmanager_2 for job 5566648d9b

Re: RocksDB efficiency and keyby

Hi Trystan, You can use async-profiler[1] to detect the CPU stack within RocksDB to see what happened, maybe you can try to enable partitioned index & filters[2] if the call stack is occupied by loading index or filter block. [1] https://github.com/jvm-profiling-tools/async-profiler [2] https:

RE: RocksDB's state size discrepancy with what's seen with state processor API

Hello, I enabled some of the RocksDB metrics and I noticed some additional things. After changing the configuration YAML, I restarted the cluster with a savepoint, and I can see that it only used 5.6MB on disk. Consequently, after the job switched to running state, the new checkpoints were also

Re: How to define event time and Watermark for intermediary joined table in FlinkSQL?

Hi, The watermark of the join operator is the minimum of the watermark of the input streams. ``` JoinOperator.watermark = min(left.watermark, right.watermark); ``` I think it's enough for most cases. Could you share more details about the logic in the UDF getEventTimeInNS? I think the better s

Re: Integration Test for Kafka Streaming job

Hi I would recommend to use kafka-junit5 from salesforce https://github.com/salesforce/kafka-junit On top of that, you can use org.apache.flink.runtime.minicluster.TestingMiniCluster Your stack should be complete. Cheers Le jeu. 21 avr. 2022 à 07:10, Aeden Jameson a écrit : > I've had success

RE: Kubernetes killing TaskManager - Flink ignoring taskmanager.memory.process.size

Hi Dan, Assuming from previous mails that you are using RocksDb … this could have to do with the glibc bug [1][2] … I’m never sure in which setting this is already been taken care of … However your situation is very typical with glibc as allocator underneath RocksDb and giving more memory won’t

RE: Restore Job from CheckPoint in IntelliJ IDE - MiniCluster

Hi Kostas, Did you give setting execution.savepoint.path a try? You can set the property on local environment by means of env.configure(...). This work for me ... (didn't try yet on Flink 1.15) Thias [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#exec

Re: Kubernetes killing TaskManager - Flink ignoring taskmanager.memory.process.size

Could you please configure a bigger memory to avoid OOM and use NMTracker[1] to figure out the memory usage categories? [1]. https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html Best, Yang Dan Hill 于2022年4月21日周四 07:42写道： > Hi. > > I upgraded to Flink v1.14.4 an

Re: Problems with PrometheusReporter

Please check the logs for warnings. It could be that a metric registered by a job is throwing exceptions. On 20/04/2022 18:45, Peter Schrott wrote: Hi kuweiha, Just to confirm, you tried with 1.15 - none of the rcs are working for me? This port is definitely free as it was already used on the