Re: Async IO operator to write to DB

-generic-asynchronous-base-sink/ > > > On 13 Jun 2023, at 05:29, Karthik Deivasigamani > wrote: > > *CAUTION*: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. &g

Re: Async IO operator to write to DB

a over > using Async IO. The primary use case for Async IO is enrichment, not for > writing to a sync. > > Best regards, > > Martijn > > On Mon, Jun 12, 2023 at 4:10 PM Karthik Deivasigamani > wrote: > >> Thanks Martijn for your response. >> One thing I

Re: Async IO operator to write to DB

va Driver once the migration is done. ~ Karthik On Mon, Jun 12, 2023 at 4:56 PM Martijn Visser wrote: > Hi, > > Why wouldn't you just use the Flink Kafka connector and the Flink > Cassandra connector for your use case? > > Best regards, > > Martijn > > O

Async IO operator to write to DB

Hi, I have a use case where I need to read messages from a Kafka topic, parse it and write it to a database (Cassandra). Since Cassandra supports async APIs I was considering using Async IO operator for my writes. I do not need exactly-once semantics for my use-case. Is it okay to leverage the A

Re: async io operator timeouts

d. A request fails, an > exception is thrown and > the job is restarted. > > A handler would be a nice addition. If you want, you can open a JIRA about > it and if would like > to work on it, feel free to submit a PR. > > Thanks, > Kostas > > > On Oct 6, 2017, at

Re: Checkpoint was declined (tasks not ready)

but only once in a while, it > should not be a problem. > It simply means that this particular checkpoint could not be triggered, > because some sources were not ready yet. > > It should try another checkpoint and then be okay. > > > On Fri, Oct 6, 2017 at 4:53 PM, Karthik Deiv

async io operator timeouts

Hi, Is there a way to catch the timeouts thrown from async io operator? We use async io API to make some high latency HTTP API calls. Currently when the underlying http connection hangs and fails to timeout in the configured time the async timeout kicks in and throws an exception which causes th

Re: Checkpoint was declined (tasks not ready)

We are using Flink 1.3.1 in Standalone mode with a HA job manager setup. ~ Karthik On Fri, Oct 6, 2017 at 8:22 PM, Karthik Deivasigamani wrote: > Hi, > I'm noticing a weird issue with our flink streaming job. We use async > io operator which makes a HTTP call and in certain

Checkpoint was declined (tasks not ready)

Hi, I'm noticing a weird issue with our flink streaming job. We use async io operator which makes a HTTP call and in certain cases when the async task times out, it throws an exception and causing the job to restart. java.lang.Exception: An async function call terminated with an exception. Fai

Re: stream partitioning to avoid network overhead

ess you have measured this, but I'm a little confused > why the intra-Flink network traffic would cause significant problems: It > seems to be the same amount of data that you're querying from the > external webservice, and that connection should be a bottleneck before > the

stream partitioning to avoid network overhead

Hi, I have a use case where we read messages from a Kafka topic and invoke a webservice. The web-service call can take a take couple of seconds and then gives us back on avg 800KB of data. This data is set to another operator which does the parsing and then it gets sent to sink which saves the

flink kafka consumer lag

Hi, I'm using Flink 1.2.1 and FlinkKafkaConsumer09 to read data from Kafka Server (0.10.1.0). My consumer is able to read and everything works fine. But when I use the kafka_consumer_groups.sh command to find the lag metrics it does not return the lag. Upon reading online I found that Flink Kafk