yes sharninder, I tried with the below configuration, but it throws "flume.EventDeliveryException: Could not find schema for event" error, please help me
first I copied the schema file to /etc/flume.conf/schemas( created this directory explicitly)/avro.avsc #defined source as spooldir, channel as memory, sink as hdfs, agent.sources = s1 agent.channels = c1 agent.sinks = r1 agent.sources.s1.type = spooldir agent.sources.s1.spoolDir = <dir path> agent.sources.s1.deserializer = avro agent.sources.s1.deletePolicy = immediate agent.sources.s1.channels = c1 agent.sources.s1.interceptors = attach-schema agent.sources.s1.interceptors.attach-schema = static agent.sources.s1.interceptors.attach-schema.key = flume.avro.schema.url agent.sources.s1.interceptors.attach-schema.value = flle:/etc/flume-ng/schemas/avro.avsc agent.channels.c1.type = memory agent.channels.c1.capacity = 10000000 agent.channels.c1.transactionCapacity = 1000 agent.sinks.s1.type = hdfs agent.sinks.s1.hdfs.path = /flume/ agent.sinks.s1.hdfs.fileType = DataStream agent.sinks.s1.channel = c1 agent.sinks.s1.hdfs.batchSize = 100 agent.sinks.s1.hdfs.serializer = org.apache.flume.sink.hdfs. AvroEventSerializer$Builder On Wed, Jun 25, 2014 at 2:07 PM, Sharninder <sharnin...@gmail.com> wrote: > Did you try using the spooldir source with an hdfs sink? What problems did > you face? > > -- > Sharninder > > > > On Wed, Jun 25, 2014 at 12:15 PM, kishore alajangi < > alajangikish...@gmail.com> wrote: > >> Hi Flume Experts, >> >> Could anybody help me to store avro files located in my localfilesystem >> into hdfs using flume, frequently new files will be generated so "spooldir" >> work for it as source, >> >> -- >> Thanks, >> Kishore. >> > > -- Thanks, Kishore.
