Re: Apache Kafka in AWS

2013-05-22 Thread Ken Krugler
d. Obviously the Sun/Oracle JRE isn't bug-free, but it gets a lot more stress testing. So one of my basic guidelines in the ops portion of the Hadoop class I teach is that every server must have exactly the same version of Oracle's JRE. > _______

Re: Apache Kafka in AWS

2013-05-22 Thread Ken Krugler
not the intended recipient, be aware that any > disclosure, copying, distribution or use of the contents of this information > is prohibited. If you have received this electronic transmission in error, > please notify us by e-mail at (postmas...@rapid7.com) immediately. -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

Re: Relationship between Zookeeper and Kafka

2013-05-20 Thread Ken Krugler
ith a 5 machine cluster? Based on the >>> whitepapers published, LinkedIn describes a peak of 170,000 events per >>> second across their cluster. My 20,000 seems so far away from their >>> production figures. >>> >>> What is the relationship, in terms of performance, between ZK and Kafka? >>> Do I need to have a more performant ZK cluster, the same, or does it >>> really >>> not matter in terms of maximizing throughput. >>> >>> Thanks for any suggestions ­ I've been pulling knobs and turning levers >>> on >>> this for several days now. >>> >>> >>> Jason >>> -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

Re: ETL with Kafka

2013-01-07 Thread Ken Krugler
ould have to back-port these to the older Hadoop APIs in order to work with Cascading. Also Cascading sends all data around as the key (value is always NullWritable) whereas the Kafka input/output formats do the opposite. -- Ken > On Jan 7, 2013 1:51 PM, "Ken Krugler" wrote: >

Re: ETL with Kafka

2013-01-07 Thread Ken Krugler
king about a Cascading Tap, right? -- Ken > On Mon, Jan 7, 2013 at 9:57 AM, Ken Krugler > wrote: > >> Hi Guy, >> >> On Jan 6, 2013, at 11:11pm, Guy Doulberg wrote: >> >>> Hi, >>> Thanks David, >>> >>> I am looking for a produ

Re: ETL with Kafka

2013-01-07 Thread Ken Krugler
t; for. Maybe you could describe your use case a bit more? >> >> On Sunday, January 6, 2013, Guy Doulberg wrote: >> >>> Hi >>> >>> I am looking for an ETL tool that can connect to kafka, as a consumer and >>> as a producer, >>> >>