Kafka as an event store for Event Sourcing

I've been working on an application which uses Event Sourcing, and I'd like to use Kafka as opposed to, say, a SQL database to store events. This would allow me to easily integrate other systems by having them read off the Kafka topics. I do have one concern, though: the consistency of the data ca

Re: Anouncing PyKafka!

Thanks, a question, does it support keyed Producer? I mean how it works when I have multiple partitions? Will it be able to identify partition based of the key which I pass? On 9 June 2015 at 00:54, Keith Bourgoin wrote: > Hi Kafka folks, > > I'm happy to announce the 1.0 release of PyKafka >

Re: Anouncing PyKafka!

Hi Vijay, PyKafka support custom partitioners so that you can control how the producer distributes messages. Looking at it now, I realize the documentation around this is a bit lacking, so I'll add that. In the meantime, here's an example of how to use the hashing partitioner which is included w

Re: Anouncing PyKafka!

Thanks a lot Keith for detailed explanation, that will surely help me. Another question, I can see AsyncProducer is not yet implemented, any idea by when it can be available? Thanks, Vijay On 10 June 2015 at 18:42, Keith Bourgoin wrote: > Hi Vijay, > > PyKafka support custom partitioners so tha

Re: ReplicaFetcherThread Error, Massive Logging, and Leader Flapping

Just for sake of future forums readers, the solution to the "leader flapping" problem I described was to increase the zookeeper session timeout setting (zookeeper.session.timeout.ms). I believe we doubled it (15000ms to 3ms). For the ReplicaFetcherThread and https://issues.apache.org/jira/brow

Kafka 0.8.3 - New Consumer - user implemented partition.assignment.strategies?

Is it possible to have a consumer rebalancing partition assignment strategy that will rebalance with a minimal number of reassignments? Per the "Kafka 0.9 Consumer Rewrite Design" it should be possible to define my own partition assignment strategy: "partition.assignment.strategies - may take a

Re: Anouncing PyKafka!

I would guess within a week or two, but it's hard to say for sure. Emmett Butler, the other main contributor, has been hard at work on it when he has the time. On Wed, Jun 10, 2015 at 12:18 PM Vijay Patil wrote: > Thanks a lot Keith for detailed explanation, that will surely help me. > Another

Re: Kafka 0.8.3 - New Consumer - user implemented partition.assignment.strategies?

Hi Olof, Yes we have plans to allow user defined partitions.assignment strategy to pass in to the consumer coordinator; I am not sure if this feature will not be available in the first release of the new consumer in 0.8.3 though. Currently users still have to choose one from the server-defined str

How to manage the consumer group id?

Hi, How are people specifying/persisting/resetting the consumer group identifier ("group.id") when using the high-level consumer? I understand how it works. I specify some string and all consumers that use that same string will help consume a topic. The partitions will be distributed amongst t

Re: How to manage the consumer group id?

For us, group ID is a configuration parameter of the application. So we store it in configuration files (generally on disk) and maintain it there through our configuration and deployment infrastructure. As you pointed out, hard coding the group ID into the application is not usually a good pattern.

Changing replication factor for an existing topic

Greetings, With Kafka 0.8.2.1, is there a way to either raise or lower the replication factor of an existing topic? Searching through this list, there was mention that the tool has yet to be developed. It doesn't seem possible still looking at the replication tools wiki, nor with yahoo's kafka m

RE: Changing replication factor for an existing topic

The replica list that you specify can be used to increment/decrement the replication factor. http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor Aditya From: Robin Yamaguchi [ro...@tune.com] Sent: Wednesday, June 10, 2015 4:05

Re: Changing replication factor for an existing topic

AirBNB's kafkat tool has a "set-replication-factor" option. I've never tried it myself. https://github.com/airbnb/kafkat -James > On Jun 10, 2015, at 4:20 PM, Aditya Auradkar > wrote: > > The replica list that you specify can be used to increment/decrement the > replication factor. > http:/

Re: How to manage the consumer group id?

> On Jun 10, 2015, at 1:26 PM, Todd Palino wrote: > > For us, group ID is a configuration parameter of the application. So we > store it in configuration files (generally on disk) and maintain it there > through our configuration and deployment infrastructure. As you pointed > out, hard coding t

Increased replication factor. Replication didn't happen!

Ran this: $KAFKA_HOME/bin/kafka-reassign-partitions.sh But Kafka did not actually do the replication. Topic description shows the right numbers, but it just didn't replicate. What's wrong, and how do I trigger the replication to occur?? I'm running 0.8.2.0 thanks

Re: Increased replication factor. Replication didn't happen!

What do the logs show? On Wed, Jun 10, 2015 at 5:07 PM, Dillian Murphey wrote: > Ran this: > > $KAFKA_HOME/bin/kafka-reassign-partitions.sh > > But Kafka did not actually do the replication. Topic description shows the > right numbers, but it just didn't replicate. > > What's wrong, and how do I

expose setConsumerRebalanceListener

Hi, Just wondering why setConsumerRebalanceListener is not exposed in kafka.javaapi.consumer.ConsumerConnector? In the latest trunk code setConsumerRebalanceListener is in kafka.javaapi.consumer.ZookeeperConsumerConnector but not in kafka.javaapi.consumer.ConsumerConnector which makes the method n

Re: Increased replication factor. Replication didn't happen!

Right now, Kafka topics do not support changing replication factor or partition number after creation. The kafka-reassign-partitions.sh tool can only reassign existent partitions. 2015-06-11 9:31 GMT+08:00 Gwen Shapira : > What do the logs show? > > On Wed, Jun 10, 2015 at 5:07 PM, Dillian Murph

Re: offsets.storage=kafka, dual.commit.enabled=false still requires ZK

Looking forward to the 0.8.3 release. BTW, some official management consoles should be better. Non of the ones mentioned on the official website rocks. 2015-06-10 2:38 GMT+08:00 Ewen Cheslack-Postava : > The new consumer implementation, which should be included in 0.8.3, only > needs a bootstrap

offset storage as kafka

Apologies for my novice question: I changed the offset storage to kafka (0.8.2.1) using the consumer property offsets.storage=kafka. I see that a new topic, __consumer_offsets got created. But after this I could still see the consumer group offset changing in zookeeper. The config consumer.proper