ClassNotFound means the Hadoop job is not able to find the related jar. Have you made sure the related jars are registered in the distributed cache?
On Fri, Nov 8, 2013 at 8:40 AM, Abhi Basu <9000r...@gmail.com> wrote: > Can anyone help me with this issue? I feel like I am very close and am > probably making some silly config error. > > Kafka team, please provide more detailed notes on how to make this > component work. > > Thanks. > > > On Fri, Nov 8, 2013 at 5:23 AM, Abhi Basu <9000r...@gmail.com> wrote: > > > Simplekafkaetljob class, as mentioned in the post. > > > > Thanks > > > > Abhi > > > > From Samsung Galaxy S4 > > On Nov 7, 2013 8:34 PM, "Jun Rao" <jun...@gmail.com> wrote: > > > >> Which class is not found? > >> > >> Thanks, > >> > >> Jun > >> > >> > >> On Thu, Nov 7, 2013 at 11:56 AM, Abhi Basu <9000r...@gmail.com> wrote: > >> > >> > Let me describe my environment. Working on two nodes currently: > >> > 1.Single-node hadoop cluster (will refer as Node1) > >> > 2.Single node Kafka cluster (will refer as Node2) > >> > > >> > Node 2 has 1 broker started with a topic (iot.test.stream) and one > >> command > >> > line producer and one command line consumer to test the kafka install. > >> > Producer can send messages and the Consumer is receiving it. > >> > > >> > Node 1 (hadoop cluster) has kafka hadoop consumer code built. Have > >> edited > >> > the /kafka-0.8/contrib/hadoop-consumer/test/test.properties file with > >> the > >> > following: > >> > > >> > kafka.etl.topic=iot.test.stream > >> > hdfs.default.classpath.dir=/tmp/kafka/lib > >> > hadoop.job.ugi=kafka,hadoop > >> > kafka.server.uri=tcp://idh251-kafka:9095 > >> > input=/tmp/kafka/data > >> > output=/tmp/kafka/output > >> > kafka.request.limit=-1 > >> > ........... > >> > > >> > I have copied the copy-jars.sh to /tmp/kafka/lib (on HDFS) > >> > > >> > Next I run the following on Node 1: > >> > ./run-class.sh kafka.etl.impl.SimpleKafkaETLJob test/test.properties > >> from > >> > the /kafka-0.8/contrib/hadoop-consumer folder and get a > >> > classnotfoundexception for kafka.etl.impl.SimpleKafkaETLJob class. > >> > > >> > What am I missing? I was thinking that running the sh file would allow > >> me > >> > to retrieve messages with the same topic name to HDFS from Node 2 to > >> Node > >> > 1. I just want to do an end to end test to see that messages coming > into > >> > Kafka are being stored in HDFS with the minimal amount of code change > >> > required. > >> > > >> > Thanks, > >> > > >> > Abhi > >> > > >> > -- > >> > Abhi Basu > >> > > >> > > > > > -- > Abhi Basu >
