Source Connector Task in a distributed env

Hello Team, I am developing a custom Source Connector that watches a given directory for any new files. My question is in a Distributed environment, how will the tasks in different nodes handle the file Queue? Referring to this sample

Kafka question on Stream Processing

Hi team, Say we have a client which has pushed a message to a topic. The message has a a simple structure Task - Time of task Send an email - 1530 Now say that this message is consumed by a consumer subscribed to this topic. Since Topic already has a storage, what I intend to do is just read the

Kafka consumer downgrade issue

Hi all, Recently we upgraded our application from the more primitive Java client APIs (kafka.javaapi.consumer.SimpleConsumer, kafka.api.FetchRequest and friends) to the more friendly poll-based org.apache.kafka.clients.consumer.KafkaConsumer using Kafka Java client libraries version 1.1.0. The

Re: Source Connector Task in a distributed env

Venkata, the example you have linked creates a single task config s.t. there is no parallelism -- a single task runs on the cluster, regardless of the number of nodes. In order to introduce parallelism, your SourceConnector needs to group all known files among N partitions and return N task configs

Re: Source Connector Task in a distributed env

Your connector sounds a lot like this one https://github.com/jcustenborder/kafka-connect-spooldir I do not think you can run such a connector in distributed mode though. Typically something like this runs in standalone mode to avoid conflicts. -hans On Wed, Apr 24, 2019 at 1:08 AM Venkata S A

Re: Kafka question on Stream Processing

Hi Gagan, If you want to read a message, you need to poll the message from the broker. The brokers have only very limited notion of message content. They only know that a message has a key, a value, and some metadata, but they are not able to interpret the contents of those message components. The

Re: Source Connector Task in a distributed env

Thank you Ryann & Hans. I will look into it. The spooldir, I explored it too and found that it too suits for standalone as you mentioned. 'Venkata On Wed 24 Apr, 2019, 22:34 Hans Jespersen, wrote: > Your connector sounds a lot like this one > https://github.com/jcustenborder/kafka-connect-spool

RE: kafka consumer metadata expire

Hi, Have you tried setting the METADATA_MAX_AGE_CONFIG (default: 300,000ms) smaller? It seems the consumer won't actually update the metadata info until it's out of date. >-Original Message- >From: Shengnan YU [mailto:ysna...@hotmail.com] >Sent: Wednesday, April 24, 2019 1:43 PM >To: use

Too many commits

Hi all , In my application i am committing every offset to kafka one by one and my max poll size is 30 . I am facing lot of commit failures so is it because of above reasons ? Thanks Yubraj Singh [image: Mailtrack]