flink job : TPS drops from 400 to 30 TPS

Hi , I have a flink real time job which processes user records via topic and also reading data from hbase acting as a look table . If the look table does not contain required metadata then it queries the external db via api . First 1to 2 hours it works fine without issues, later it drops down dras

How do I verify data is written to a JDBC sink?

In my Flink Job, I am using event time to process time-series data. Due to our business requirements, I need to verify that a specific subset of data written to a JDBC sink has been written before I send an activemq message to another component. My job flows like this: 1. Kafka Source 2. Split s

could not stop with a Savepoint.

Today, I kept on receiving a timeout exception when stopping my job with a savepoint. This happened with Flink version 1.12.2 running in EMR. I had to use the deprecated cancel with savepoint feature instead. In fact, stopping with a savepoint, creating a savepoint, and cancelling with a savepoin

Re: Can't access Debezium metadata fields in Kafka table

Hi, Harshvardhan The format debezium-avro-confluent doesn’t support read metadata yet[1], the supported formats including debezium-json, canal-json and maxwell-json, you can try the supported formats. Best, Leonard [1] https://issues.apache.org/jira/browse/FLINK-20454

Re: Write Streaming data to S3 in Parquet files

Hi, Thanks for the response. How can this streaming data be written to S3 for the path to be given? Also I see that the FileSink takes GenericRecord, so how can the DataStream be converted to a GenericRecord? Please bear with me if my questions don't make any sense. On Sun, Sep 26, 2021 at 9:12

Re: 退订

Hi, To unsubscribe emails from Flink dev mail list, send an email to dev-unsubscr...@flink.apache.org To unsubscribe emails from Flink user mail list, send an email to user-unsubscr...@flink.apache.org To unsubscribe emails from Flink user -zh mail list, send an email to user-zh-unsubscr...@flink

Re: Potential bug when assuming roles from AWS EKS when using S3 as RocksDb checkpoint backend?

Hi Thomas, I haven't encountered that before, sorry. I assume you're still using Flink 1.12? The PR I linked to updated the AWS dependencies to the minimum required versions to use this feature, so I'm not sure just setting the credentials provider alone would be sufficient. The PR was currently o