Avro source and sink

2015-09-15 Thread Buntu Dev
Currently I have a single flume agent that converts apache logs into Avro and writes to HDFS sink. I'm looking for ways to create tiered topology and want to have the Avro records available to other flume agents. I used Kafka channel/sink to write these Avro records but was running into this error

Re: Avro source and sink

2015-09-15 Thread Gonzalo Herreros
I'm not sure if I understand your topology and what you mean exactly by "used Kafka channel/sink", it would help if you send the configuration. My best guess about the error is that you are pointing the kafka source to a topic that is used by a channel and not by a kafka sink Regards, Gonzalo O

Re: Avro source and sink

2015-09-15 Thread Buntu Dev
Thanks Gonzalo. You are correct about the topology in that I'm using Kafka channel as a source. Based on this thread , I was under the impression that Kafka sink is redundant. Heres the topology: Agent#1: spooldir source -> morphlines (transforms to av