Round Robin for several consumers in KAFKA

Hi, I have 5 separate docker images : 1 for kafka broker, 1 zookeeper , 1 producer and 2 consumers. I publish messages to the topic via the producer. Basically, I would likw that the messages will be published in a round robin algorithm, so for that purpose I defined the consumers with the same

Re: Round Robin for several consumers in KAFKA

what is the partition size? you need at least 2 partitions to distribute across two consumers On Wed, Sep 13, 2017 at 1:24 PM, Liel Shraga (lshraga) wrote: > Hi, > > > > I have 5 separate docker images : 1 for kafka broker, 1 zookeeper , 1 > producer and 2 consumers. > > I publish messages to t

RE: Round Robin for several consumers in KAFKA

Hi, I didn’t define the partition size. How can I do it with kafka-clients API? Thanks,   Liel Shraga ENGINEER.SOFTWARE ENGINEERING lshr...@cisco.com Tel: +972 2 588 6394 Cisco Systems, Inc. 32 HaMelacha St., (HaSharon Bldg) P.O.Box 8735, I.Z.Sapir SOUTH NETANYA 42504 Israel cisco.com Think

[ANNOUCE] Apache Kafka 0.11.0.1 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 0.11.0.1. This is a bug fix release that fixes 51 issues in 0.11.0.0. All of the changes in this release can be found in the release notes: *https://archive.apache.org/dist/kafka/0.11.0.1/RELEASE_NOTES.html

Re: Metrics for Kafka Connect

Thanks for the KIP link. Those metrics will be a nice addition, but the piece I'm most interested in is adding hooks in the Connect API for connectors to define metrics specific to their domain, which this KIP explicitly identifies as out of scope. I'm currently deploying Confluent's S3 connector,

Fw: Admin Client : no way to create topic with default partitions and replication factor

Maybe it could of interest for users as well. Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoT Microsoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExp

Re: Kafka Operational Wierdness

Some more detail on this issue: On a hunch I tried restarting my docker-compose stack a few more times. Still same problem, my application using the Kafka Client APIs claims it is talking to Kafka, but the Kafka logs disagree. So I restarted the stack once more. With 'docker-compose up' this

Re: Kafka 11 | Stream Application crashed the brokers

Adding more info:- Hi Guozhang, I was using exactly_once processing here, I can see this in the client logs, however I am not setting transaction id though. application.id = c-7-e6 application.server = bootstrap.servers = [172.29.65.190:9092, 172.29.65.192:9092, 172.29.65.193:9092] buffered.reco

Kafka Connect Sink Connector for multiple JDBC sinks

Hi, I need a little help/suggestion if possible. Does anyone know if it's possible in Kafka to develop a connector that can sink for multiple JDBC urls for the same topic (i.e. table) ? The examples I can see on Confluent talks about one JDBC url (one-to-one sink). Would it be possible to achieve

Re: Kafka Connect Sink Connector for multiple JDBC sinks

This is possible. Once you have records in your put method, its up your logic how you are redirecting it to multiple jdbc connections for insertion. In my use case i have implemented many to many sources and sinks. Regards, Srijith On 13-Sep-2017 10:14 pm, "M. Manna" wrote: Hi, I need a littl

unable to move virgin partitions in 0.11

Hi, having bad luck with hardware, I need to move all topics to new hardware. It turns out that reassigning partitions hangs on virgin (i.e. created but never written to) partitions, namely __consumer-offsets I tried kafkacat to write to it, but it (or the broker?) refuses to write to it. Any i

Re: Kafka 11 | Stream Application crashed the brokers

When exactly_once is turned on the transactional id would be set automatically by the Streams client. What I'd inspect is the healthiness of the brokers since the " *TimeoutException*", if you have metrics on the broker servers regarding request handler thread idleness / request queue length / req

Re: Reliably producing records to remote cluster: what are my options?

In my experience, 7 is the easiest route. Just make sure to run the mirror-maker on the consumer side of the wan, it's order of magnitude faster this way. If you put receive.buffer.bytes=33554432 send.buffer.bytes=33554432 in your consumer config and adjust the remote server.config to socket.recei

Re: [ANNOUCE] Apache Kafka 0.11.0.1 Released

Thanks for driving this Damian! Guozhang On Wed, Sep 13, 2017 at 4:36 AM, Damian Guy wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.11.0.1. This is a bug fix release that fixes 51 issues in 0.11.0.0. > > All of the changes in this release can be fou

Re: [ANNOUCE] Apache Kafka 0.11.0.1 Released

Thanks for driving the release Damian. > On Sep 13, 2017, at 1:18 PM, Guozhang Wang wrote: > > Thanks for driving this Damian! > > > Guozhang > >> On Wed, Sep 13, 2017 at 4:36 AM, Damian Guy wrote: >> >> The Apache Kafka community is pleased to announce the release for Apache >> Kafka 0.11.

performance test using real data - comparing throughput & latency

We are testing kafka’s performance with the real prod data and plan to test things like the below. We would have producers publishing and consumers processing production data on a separate non-prod kafka cluster. * Impact of number of Partitions per Topic on throughput and latency on Produ