Re: log compaction scaling with ~100m messages

2015-10-08 Thread vipul jhawar
Just want to chime on this question as this does seem a good option to avoid some memory hungry K,V store in case we are ok with some async processing. There are cases where you want a combination of some near realtime and some offline processing of the same index and as kafka topic is much efficie

Re: powered by kafka

2014-11-09 Thread vipul jhawar
Hi Jay Thanks for posting the update. However, i checked the page history and the hyperlink is pointing to the wrong domain. Exponential refers to www.exponential.com. I sent the twitter handle, should have sent the domain. Please correct. Thanks On Sat, Nov 8, 2014 at 3:45 PM, vipul jhawar

powered by kafka

2014-11-08 Thread vipul jhawar
Exponential @exponentialinc is using kafka in production to power the events ingestion pipeline for real time analytics and log feed consumption. Please post on powered by kafka wiki - https://cwiki.apache.org/confluence/display/KAFKA/Powered+By Thanks Vipul http://in.linkedin.com/in/vjhawar/

Re: Announcing Confluent

2014-11-07 Thread vipul jhawar
Best of luck. Will stay tuned to news. On Thu, Nov 6, 2014 at 11:58 PM, Jay Kreps wrote: > Hey all, > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a > company around Kafka called Confluent. We are planning on productizing the > kind of Kafka-based real-time data platfor

Re: Issue with 240 topics per day

2014-08-11 Thread vipul jhawar
Your use case requires messages to pushed out when time comes instead of the order in which they arrived, while kafka may not be best for this as within the Q you want some message batch to be sent out early and some later. There could be another way to solve this with offset management as kafka is

Re: Handling of failure to send message in kafka async producer

2014-07-11 Thread vipul jhawar
Hi Jun We tested the sync, but its impossible to use it with that performance. Tried even by batching messages in one large message, to decrease the acks, but not much improvement. As you mentioned about next release, i saw an update - http://grokbase.com/t/kafka/users/145m2sabrt/async-producer-c

running kafka on netapp

2014-05-16 Thread vipul jhawar
Hi Does anyone have any stats or experience to share with running kafka on netapp based filer / storage. Has anybody faced any issues with it or it did not matter whether the storage is local to the host or a network storage. Thanks