Flume-kafka problem

HI, In my project i used flume and kafkaSink, but with a strange phenomeno, the flume didnot work after an hour, that mean the files in flume spooldir cannot be process normally，it seems like something wrong but with no errors,the flume-ng server log has nothing except the info: （Sourcing

Flume/Avro schema manipulation

I am looking for some good documentation that would explain how I can pull logs from a file and apply a certain schema to that log file format. I have gone threw the exercise of sending a file using a Avro source/sink with the flume_ng avro_client to a flume_ng agent but my understanding is tha

Re: Flume/Avro schema manipulation

Hi Ed, maybe after read logs from your file, you can apply org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder interceptor and then apply to Avro morphline .

Hadoop 2x Compatability

Hi All, I'm pretty new to Flume so forgive the newbish question, but I've been working with Hadoop 2x for a little while. I'm trying to configure flume (1.5.0) with a HDFS sink however the agent won't start citing the following error: 29 Aug 2014 13:40:13,435 ERROR [conf-file-poller-0] (org.apac

RE: Hadoop 2x Compatability

I have setup flume 1.5..and use hdfssink on hadoop 2. Its in my local laptop. Flume agent has started and saved data into hdfs using the sink fine and is running without any problem. Let me know if you want to see jars in lib folder of my flume installation. -Original Message- From: "C

Re: Hadoop 2x Compatability

Yes please that would be very helpful. Are you using the tar release from the flume website? On Fri, Aug 29, 2014 at 3:54 PM, Sandeep khurana wrote: > I have setup flume 1.5..and use hdfssink on hadoop 2. Its in my local > laptop. Flume agent has started and saved data into hdfs using the sink

Re: Flume/Avro schema manipulation

From initial reading of what it does, it look promising. So, I could ultimately leverage something like the readLine and toAvro commands Thanks, Ed On Aug 29, 2014, at 8:25 AM, Gaspar Muñoz wrote: > Hi Ed, > > maybe after read logs from your file, you can apply > org.apache.flume.sink.so

Re: Hadoop 2x Compatability

How are you using Flume? If you downloaded the Flume binary tarball, that is built against hadoop 1.1 and will not work against hadoop2. You can however, download and build flume against hadoop 2.x, using either the hadoop-2 profile (mvn clean package -Dhadoop.profile=2) or hbase-98 profile (mv