where to capture a failed task's exception

2023-12-26 Thread Akash Dhiman
Hello, I have a requirement where I need to detect failed tasks based on the specific errors and emit a metric only when it doesn't fail based on these specific errors (these include unknown_topic_or_partition, specific cases of ConfigException etc) I know about a similar metric accessible via Pr

KafkaConsumer taking time to determine offset

2024-06-12 Thread Akash Dhiman
Hello, we have a usecase where we use kafkaConsumer in a SourceTask of a source connector to poll messages from an aws msk. if we try to produce data into the source topic immediately after the connector gets into the running state we sometimes notice that kafkaconsumer misses some of the records

Making connector fault tolerant

2024-06-12 Thread Akash Dhiman
Hello, we have a requirement to make kafka connector more fault tolerant for our use, where in we don't want them to fail for some kinds of errors, i.e error where bootstrap broker are missing or if we don't have sufficient permission to read data from topic (we are reading from aws msk). we tried

Re: Making connector fault tolerant

2024-06-14 Thread Akash Dhiman
ed with a process > change? (if this is an ongoing problem then it sounds like there's a > larger issue happening that should be addressed) > > When you say "bootstrap brokers are missing"... are you referring to > someone misconfiguring a connector such that the bo