Re: running kafka/zookeeper running locally

2015-04-19 Thread Warren Henning
Also consider with whether you could get away with mocking out the Kafka broker, depending on what/how you like to test. On Sun, Apr 19, 2015 at 10:47 PM, sunil kalva wrote: > Hi > Any one tried running zookeeper and kafka locally , which can be useful for > automating the test cases for API bui

Consuming keyed messages with null value

2015-04-29 Thread Warren Henning
I have an application producing Avro-encoded keyed messages (Martin Kleppmann's new Bottled Water project). It encodes a delete as a keyed message with an id as a key and a null payload. I have log compaction turned on. The Avro console consumer correctly displays this as "null" in my terminal, b

Re: Kafka integration with Hadoop

2015-05-11 Thread Warren Henning
You could start by looking at Linkedin's Camus and go from there? On Mon, May 11, 2015 at 8:10 PM, Rajesh Datla wrote: > Hi All, > > How to integrate Kafka with Hadoop ecosystem. > > How to store Kafka messages into HDFS in parquet format > > Regards > Raj >

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

2015-05-18 Thread Warren Henning
Interesting. Thanks for sharing and working on this! Can you comment on how this compares to Confluent's REST proxy? On Sat, May 16, 2015 at 12:01 AM, Marcin Kuthan wrote: > Hi Everyone > > Hermes is an asynchronous message broker built on top of Kafka. It > provides reliable, fault tolerant RE

Re: questions

2015-05-21 Thread Warren Henning
Yes, you can specify the partition count when creating a topic. http://kafka.apache.org/documentation.html#quickstart On Thu, May 21, 2015 at 11:05 PM, ram kumar wrote: > Hi, > > can v specify the partition size of a particular topic >

Architecture for multiple consumers of a given message

2015-05-24 Thread Warren Henning
I'm working on a simple web application where I want the same message of a given topic to be consumed by multiple separate consumers (one that writes to a database, another that writes to a search index like Solr/Elasticsearch). Right now as this is just a small personal project I only have one pa

Re: Is it a good idea to use Apache Kafka for this purpose?

2015-05-26 Thread Warren Henning
There's not much specific detail in what you described, but that sounds like classic pub-sub, which Kafka is well-suited for. On Tue, May 26, 2015 at 8:09 AM, Mert metin wrote: > Hi there! > I have multiple servers. What I want to do is to send user information > across servers. Basically, there