I want to store avro files into hdfs using flume, could anybody suggest me the configuration? I am trying with the below configuration, but it throws "flume.EventDeliveryException: Could not find schema for event" error, please help me
first I copied the schema file to /etc/flume.conf/schemas(created this directory explicitly)/avro.avsc #defined source as spooldir, channel as memory, sink as hdfs, agent.sources = s1 agent.channels = c1 agent.sinks = r1 agent.sources.s1.type = spooldir agent.sources.s1.spoolDir = <dir path> agent.sources.s1.deserializer = avro agent.sources.s1.deletePolicy = immediate agent.sources.s1.channels = c1 agent.sources.s1.interceptors = attach-schema agent.sources.s1.interceptors.attach-schema = static agent.sources.s1.interceptors.attach-schema.key = flume.avro.schema.url agent.sources.s1.interceptors.attach-schema.value = flle:/etc/flume-ng/schemas/avro.avsc agent.channels.c1.type = memory agent.channels.c1.capacity = 10000000 agent.channels.c1.transactionCapacity = 1000 agent.sinks.s1.type = hdfs agent.sinks.s1.hdfs.path = /flume/ agent.sinks.s1.hdfs.fileType = DataStream agent.sinks.s1.channel = c1 agent.sinks.s1.hdfs.batchSize = 100 agent.sinks.s1.hdfs.serializer = org.apache.flume.sink.hdfs.AvroEventSerializer$Builder -- Thanks, Kishore.
