Re: Help is processing huge data through Kafka-storm cluster

2014-06-17 Thread Robert Rodgers
we have been experimenting with Samza which is also worth a look. It's basically a topic-to-topic node on Yarn. On Jun 17, 2014, at 10:44 AM, hsy...@gmail.com wrote: > Hi Shaikh, > > I heard some throughput bottleneck of storm. It cannot really scale up with > kafka. > I recommend you to try

Re: [DISCUSS] Kafka Security Specific Features

2014-06-03 Thread Robert Rodgers
... client specific presented information, signed in some way, listing topic permissions. read, write, list. TLS lends itself to client certificates. On Jun 3, 2014, at 12:57 PM, Joe Stein wrote: > 4) Authorization > > We should have a policy of "404" for data, topics, partitions (etc) if >

Re: Broker rejoin with big replica lag

2014-02-05 Thread Robert Rodgers
this would be great to add to the operational section of the Kafka documentation. On Feb 5, 2014, at 2:18 PM, Andrew Otto wrote: >> - Increasing num.replica.fetchers (defaults is one) > Awesome! I just tried this one, bumped it up to 8 (12 cores on this broker > box). It is now catching up a

Re: Kafka Authenication and Monitoring Tools

2014-01-14 Thread Robert Rodgers
> https://cwiki.apache.org/confluence/display/KAFKA/Security Interesting. I'm curious what people are doing in the interim. We'd been chewing on options for secure communication and mutual authentication like stunnel. Actually imposing discretionary controls on the activity of clients using

Re: Data loss in case of request.required.acks set to -1

2013-12-18 Thread Robert Rodgers
what happens if the physical machine dies or the kernel panics? On Dec 18, 2013, at 9:44 AM, Hanish Bansal wrote: > Yup definitely i would like to try that If controlled.shutdown.enable > property works in case of kill -9. > > I hope that this option will be perfect. > > Thanks for quick resp

Re: Kafka cluster with lots of topics

2013-11-18 Thread Robert Rodgers
If I understand your comment correctly you are trying to use kafka topics as per-endpoint message queues. I may be mistaken, but to me Kafka seems does not really seem like a good match for that. For such a purpose you will eventually want something that is not actually a queue - a means to