Flume scaling best practices question

Hi! I'm using this flume-snmp-source plugin [1] to query a managed host with the following logic: agent.sources.source1.type = org.apache.flume.source.SNMPQuerySource agent.sources.source1.host = 23.23.52.11 agent.sources.source1.port = 161 agent.sources.source1.delay = 30 agent.sources.source1.o

Use Flume to Stream Real Time Video into HDFS

Hi, Is there any sample or tutorial about streaming Real Time Videos into HDFS? I am trying to use net cat but it might not be a good choice (I am new to flume). Regards Arthur (below is my test configuration) ### A single-node Flume configuration ### Name the components on this agent a3.sour

Re: Use Flume to Stream Real Time Video into HDFS

You may try writing a custom source which can use a gstreamer pipeline (which you will build) and send data to channel etc . On Tue, Sep 2, 2014 at 1:51 PM, arthur.hk.c...@gmail.com < arthur.hk.c...@gmail.com> wrote: > Hi, > > Is there any sample or tutorial about streaming Real Time Videos int

Avro source and sink

Does anyone know of any good documentation that talks about the protocol/negotiation used between an Avro source and sink? Thanks, Ed

Re: Avro source and sink

Ed, Did you take a look at the javadoc in the source? Basically the source uses netty as a server and the sink is just an rpc client. If you read over the doc which is in the two links below and take a look at the developer guide and still have questions just ask away and someone will help to answ

performances tuning...

Hi there, I'm a completely newbie of Flume, so I probably made a mistake in my configuration but I cannot point it out. I want to achieve transfer maximum performances. My flume machine has 16GB RAM and 8 Cores I'm using a very simple Flume architecture: Source -> Memory Channel -> Sink Source is o