Re: Multiple topics strategy

Without knowing much else about your requirements - Yes, your consumers should not be listening to topics they are not interested in. There's nothing wrong with having more topics. > On Jul 10, 2016, at 10:37 AM, Vladi Feigin wrote: > > Hi All, > > Will appreciate if you share your thoughts

Re: Kafka on AWs EMR

Kafka is in no way related to map reduce, this doesn't really make any sense. You could have consumers running in Hadoop listening to Kafka topics - see the Samza project. > On Jul 10, 2016, at 11:16 AM, Snehalata Nagaje > wrote: > > > > Hi All, > > > I wanted to setup kafka on AWS EMR?

Re: Reg. Partition Rebalancing

Of possible interest, we just released a script that we use to pick partition assignments. https://github.com/chartbeat-labs/trepl > On Mar 29, 2016, at 1:33 PM, Todd Palino wrote: > > There’s two things that people usually mean when they talk about > rebalancing. > > One is leader reelectio

Re: Using Kafka for persistence

This sounds like a square peg in a round hole sort of solution. That said, you might want to look at the work being done with kafka-streams to expose a topic as a table. > On Mar 30, 2016, at 3:23 PM, Michael D. Spence wrote: > > > Any advice on using Kafka to store the actual messages? > >

ClientId and groups recommendation

Hi, If I have multiple consumers in a consumer group for load handling for the same application is there any recommendation if the clientId should all be unique for each? It's the same application. Each will have it's own consumer memberId given on the join group so they can always be distingu

Re: ClientId and groups recommendation

e want to > use the ip as the id, so when you get errors you know where to look. > > On Tue, Apr 19, 2016 at 6:51 PM Rick Rineholt wrote: > > > Hi, > > If I have multiple consumers in a consumer group for load handling for > the > > same application is there any re

Avro deserialization

Config is a private class and can’t be instantiated. It also seems that KafkaAvroDecoder does not implement Deserializer, and thus can’t be used in the normal way deserializers are registered. Has anyone gotten this stuff to work? Thanks, Rick signature.asc Description: Message signed with O

Re: Avro deserialization

ter/kafka-clients/specific-avro-consumer/src/main/java/io/confluent/examples/consumer/AvroClicksSessionizer.java> IMHO it’s confusing having the same class names in different packages when most people probably rely on an IDE to manage their imports. Thanks! Rick > On May 30, 2016, at 5:44 AM

Re: Avro deserialization

Thanks! > On May 31, 2016, at 1:00 PM, Michael Noll wrote: > > FYI: I fixed the docs of schema registry (vProps -> props). > > Best, Michael > > > On Tue, May 31, 2016 at 2:05 AM, Rick Mangi wrote: > >> That was exactly the problem, I found

Re: Multiple topics strategy

That depends on the data and your consumers. It’s not necessarily an issue at all. That said, we prefer to have consumers read off a single topic but our data is high frequency and easily hashed across partitions so we have hundreds of consumers reading off some topics. I would try to partition

One Way Kafka (or seperate ports for producers and consumers)

x27;m sure there are other solutions which might be better, so please let me know if there is a way to do this, but one way I was considering is allowing consumers to connect to one port, and producers to connect to another, and doing the IP based access control with firewalls. Thanks, Rick