Re: CDC using Query

Thanks. I looked at it. Our primary DB is Oracle and MySql. Flink CDC Connector uses Debezium. I think. So ververica doesn't have a Flink CDC Connector for Oracle ? On Mon, Feb 7, 2022 at 3:03 PM Leonard Xu wrote: > Hello, mohan > > 1. Does flink have any support to track any missed source Jdbc

Illegal reflective access by org.apache.flink.api.java.ClosureCleaner

Hello, what could be the reason for warning like this:WARNING: An illegal reflective access operation has occurredWARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/var/flink/flink-1.13.2/lib/flink-dist_2.12-1.13.2.jar) to field java.lang.String.valueWARNING: Plea

There Is a Delay While Over Aggregation Sending Results

Hey Flink Community, I am using FlinkSQL Over Aggregation to calculate the number of uuid per client ip during the past 1 hour. The flink sql I am using is something like below: SELECT COUNT(DISTINC

Re: [External] : Re: Use TwoPhaseCommitSinkFunction or StatefulSink+TwoPhaseCommittingSink to achieve EXACTLY_ONCE sink?

Hello Yun, Thanks for the quick response. This is really helpful. I have confirmed with Oracle Streaming Service (OSS) that they currently don’t support EXACTLY_ONCE semantic, only AT_LEAST_ONCE semantic works. They suggest to add some deduplicate mechanisms at Sink to mitigate the issue. Ques

Re: question on dataSource.collect() on reading states from a savepoint file

Thanks Bastien. I will check it out. Antonio. On Thu, Feb 10, 2022 at 11:59 AM bastien dine wrote: > I haven't used s3 with Flink, but according to this doc : > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/filesystems/s3/ > You can setup pretty easily s3 and use it

Re: question on dataSource.collect() on reading states from a savepoint file

I haven't used s3 with Flink, but according to this doc : https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/filesystems/s3/ You can setup pretty easily s3 and use it with s3://path/to/your/file with a write sink The page talk about DataStream but it should work with DataSet

Re: question on dataSource.collect() on reading states from a savepoint file

Thanks Bastien. Can you point to an example of using a sink as we are planning to write to S3? Thanks again for your help. Antonio. On Thu, Feb 10, 2022 at 11:49 AM bastien dine wrote: > Hello Antonio, > > .collect() method should be use with caution as it's collecting the > DataSet (multiple

Re: question on dataSource.collect() on reading states from a savepoint file

Hello Antonio, .collect() method should be use with caution as it's collecting the DataSet (multiple partitions on multiple TM) into a List single list on JM (so in memory) Unless you have a lot of RAM, you can not use it this way and you probably should not I recommend you to use a sink to print

question on dataSource.collect() on reading states from a savepoint file

Hi, I am using the stateful processing api to read the states from a savepoint file. It works fine when the state size is small, but when the state size is larger, around 11GB, I am getting an OOM. I think it happens when it is doing a dataSource.collect() to obtain the states. The stackTrace is c

Problem with kafka with key=None using pyhton-kafka module

Hello, I am following the example in Python Walkthrough , I downloaded the zip file with the project skeleton. I'm having a problem when changing the key attribute in t

Re: Issue with Flink UI for Flink 1.14.0

Hi, This issue is impacting all deployments with 2 JobManagers or more (HA mode), because in this case serialization is used (well, depending on the JobManager who is responding, the Leader or a Follower). It prevents: * usage of Flink UI * usage of Flink command "flink.sh list" * usage of

Re: Issue with Flink UI for Flink 1.14.0

Hi, AFAIK there are no plans currently to release 1.14.4. The previous one (1.14.3) was released on Jan 20, so I'd 1.14.4 preparation to start in the next several weeks. Regards, Roman On Tue, Feb 8, 2022 at 7:31 PM Sweta Kalakuntla wrote: > I am facing the same issue, do we know when 1.14.4

Re: JSONKeyValueDeserializationSchema cannot be converted to ObjectNode>

Thanks for sharing the full solution, much appreciated! On Thu, 10 Feb 2022 at 09:07, HG wrote: > The complete solution for the record ( that others can benefit from it). > > KafkaSource source = KafkaSource.builder() > .setProperties(kafkaProps) > .setProperty("ssl.truststore.ty

Re: JSONKeyValueDeserializationSchema cannot be converted to ObjectNode>

The complete solution for the record ( that others can benefit from it). KafkaSource source = KafkaSource.builder() .setProperties(kafkaProps) .setProperty("ssl.truststore.type",trustStoreType) .setProperty("ssl.truststore.password",trustStorePassword) .setProperty(