This is wierd, and looks like an Ambari class that ended up in the classpath and that somehow we are trying to load?
Perhaps Sriharsha or one of the HDP dudes can help. Does it happen without the Connector too? It looks like it has to do with Kafka broker metrics in general: https://issues.apache.org/jira/browse/AMBARI-9185 Gwen On Fri, Aug 5, 2016 at 2:16 PM, Banias H <banias4sp...@gmail.com> wrote: > Thanks Gwen. I went with Confluent 2.0 as it has Kakfa 0.9 that matches > with that in HDP 2.4. I installed confluent-kafka-connect-hdfs and > confluent-common and softlinked a couple jar into kafka libs/. > > I was able to start Kafka Connect but kafka.out was showing the following > error: > > [2016-08-05 20:57:01,187] ERROR Exception emitting metrics > (org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter) > org.apache.hadoop.metrics2.sink.timeline.UnableToConnectException: > java.net.ConnectException: Connection refused > at > org.apache.hadoop.metrics2.sink.timeline.AbstractTimelineMetricsSink.emitMetrics(AbstractTimelineMetricsSink.java:87) > at > org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter.access$200(KafkaTimelineMetricsReporter.java:58) > at > org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter$TimelineScheduledReporter.report(KafkaTimelineMetricsReporter.java:253) > at > org.apache.hadoop.metrics2.sink.kafka.ScheduledReporter.report(ScheduledReporter.java:185) > at > org.apache.hadoop.metrics2.sink.kafka.ScheduledReporter$1.run(ScheduledReporter.java:137) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Is there any configuration I should also look into? I started with > > ./connect-standalone.sh ../config/connect-standalone.properties > /etc/kafka-connect-hdfs/quickstart-hdfs.properties > > And here is my quickstart-hdfs.properties: > > name=hdfs-sink > connector.class=io.confluent.connect.hdfs.HdfsSinkConnector > tasks.max=1 > topics=hdfs > hdfs.url=hdfs://sandbox.hortonworks.com:8020 > flush.size=3 > > Thanks, > -B > > On Fri, Aug 5, 2016 at 3:31 PM, Gwen Shapira <g...@confluent.io> wrote: > >> The installation instructions from Confluent will still work for you :) >> >> If you are using deb/rpm packages, basically add the repositories as >> explained here: >> http://docs.confluent.io/3.0.0/installation.html#rpm-packages-via-yum >> >> and then: >> sudo yum install confluent-kafka-connect-hdfs >> or >> sudo apt-get install confluent-kafka-connect-hdfs >> >> This will put the connector config in /etc/kafka-connect-hdfs and the >> connector jars in /usr/share/java/ >> >> You may need to move the jar so it is on the classpath for connect >> (I'm not sure what's the default kafka classpath for HDP). >> >> BTW. We (Confluent) are testing the HDFS connector with HDP (we >> basically install Confluent platform on one machine and HDP on another >> and use Connect to move data) - so this setup should work :) >> >> Gwen >> >> >> On Fri, Aug 5, 2016 at 8:47 AM, Banias H <banias4sp...@gmail.com> wrote: >> > Hi, >> > >> > We are using Hortonworks HDP 2.4 with Apache Kafla 0.9 and we have an >> > in-house solution to pull messages from Kafka to HDFS. I would like to >> try >> > using kakfa-connector-hdfs to push messages to HDFS. As far as I concern, >> > Apache Kafka 0.9 doesn't come with kafka-connector-hdfs. What is a solid >> > way to run kafka-connector-hdfs in HDP? I won't be able to install >> > Confluent platform there though... I would appreciate any pointers. >> Thanks. >> > >> > -B >> >> >> >> -- >> Gwen Shapira >> Product Manager | Confluent >> 650.450.2760 | @gwenshap >> Follow us: Twitter | blog >> -- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
