Re: Why do I need to specify replication factor when creating a topic?

2017-05-11 Thread Andrew Psaltis
Jeff, I'm not sure if this is an option for you. However, I have been faced with a similar problem before and we handled it by putting all of the information needed to connect and use the Kafka API's in a config file. In our case we were using Typesafe config [1] for lots of configuration in our se

Re: Recording - Storm & Kafka Meetup on April 20th 2017

2017-04-21 Thread Andrew Psaltis
t; > > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Re: Kafka/Hadoop consumers and producers

2013-08-12 Thread Andrew Psaltis
ndrew >please let me know if would like to >pick this up instead. Thanks! > >Kam > > >____________ > From: Jay Kreps >To: Ken Goodhope >Cc: Andrew Psaltis ; >dibyendu.bhattacha...@pearson.com; "camus_...@googlegroups.com"

Re: Kafka/Hadoop consumers and producers

2013-08-09 Thread Andrew Psaltis
Dibyendu, According to the pull request: https://github.com/linkedin/camus/pull/15 it was merged into the camus-kafka-0.8 branch. I have not checked if the code was subsequently removed, however, two at least one the important files from this patch (camus-api/src/main/java/com/linkedin/camus/etl/Re

Re: Kafka/Hadoop consumers and producers

2013-08-08 Thread Andrew Psaltis
Felix, The Camus route is the direction I have headed for allot of the reasons that you described. The only wrinkle is we are still on Kafka 0.7.3 so I am in the process of back porting this patch: https://github.com/linkedin/camus/commit/87917a2aea46da9d21c8f67129f6463af52f7aa8 that is descri

Number of Partitions Per Broker

2013-01-14 Thread Andrew Psaltis
All, I was re-reading this: https://cwiki.apache.org/confluence/display/KAFKA/Operations and noticed that the number of partitions is 1. Is this accurate? In our environment we are currently running 20+ partitions per topic - with two brokers, the gut feel was this would speed up our ability to