map reduce Outputformat KafkaOutputFormat

2014-10-01 Thread Abraham Jacob
Hi All, After a map reduce computation I would like to send the results out to the a Kafka queue. Is there a bridge available that will let me publish the output to a Kafka queue. Similar to the TextOutputFormat... maybe a KafkaOutputFormat... I found one source https://github.com/kafka-dev/kafk

Clarification about Custom Encoder/Decoder for serialization

2014-10-09 Thread Abraham Jacob
Hi All, I wanted to get some clarification on Kafka's Encoder/Decoder usage. Lets say I want to implement a custom Encoder. public class CustomMessageSerializer implements Encoder { @Override public byte[] toBytes(String arg0) { // serialize the MyCustomObject return serializedCustomObject ; }

Re: Clarification about Custom Encoder/Decoder for serialization

2014-10-09 Thread Abraham Jacob
is instantiated once when the producer is constructed. > > Thanks, > > Jun > > On Thu, Oct 9, 2014 at 6:45 PM, Abraham Jacob > wrote: > > > Hi All, > > > > I wanted to get some clarification on Kafka's Encoder/Decoder usage. > > > > Lets

Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

2014-10-13 Thread Abraham Jacob
Hi All, I have a 8 node Kafka cluster (broker.id - 1..8). On this cluster I have a topic "wordcount", which was 8 partitions with a replication factor of 3. So a describe of topic wordcount # bin/kafka-topics.sh --describe --zookeeper tr-pan-hclstr-08.amers1b.ciscloud:2181/kafka/kafka-clstr-01 --

Re: Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

2014-10-14 Thread Abraham Jacob
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper > ) > > has the correct host/port? > > > > Thanks, > > > > Jun > > > > On Mon, Oct 13, 2014 at 5:35 PM, Abraham Jacob > > wrote: > > > >> Hi All, >

Re: Kafka - NotLeaderForPartitionException / LeaderNotAvailableException

2014-10-15 Thread Abraham Jacob
e, Oct 14, 2014 at 10:35 PM, Abraham Jacob > wrote: > > > Hi Jun, > > > > Thanks for responding... > > > > I am using Kafka 2.9.2-0.8.1.1 > > > > I looked through the controller logs on a couple of nodes and did not > find > > any exceptions