Re: Flume to stream logs live

2012-12-16 Thread Bertrand Dechoux
ient, please delete and contact the sender immediately. Please > > consider the environment before printing this e-mail. AVIS : le > > présent courriel et toute pièce jointe qui l'accompagne sont > > confidentiels, protégés par le droit d'auteur et peuvent être couverts > > par le secret professionnel. Toute utilisation, copie ou divulgation > > non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de > ce courriel, supprimez-le et contactez immédiatement l'expéditeur. > > Veuillez penser à l'environnement avant d'imprimer le présent courriel > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/ > -- Bertrand Dechoux

High level technical overview of output bucketing for flume (old-gen) ?

2013-01-04 Thread Bertrand Dechoux
Hi, I am using flume (old gen) as an extension to an existant syslog system and would like to use the timestamp of the syslog message as the timestamp of the flume event. I guess the timestamp is used for the '*Fine grained escape sequences date and times*' but I don't have a clear understanding o

Re: High level technical overview of output bucketing for flume (old-gen) ?

2013-01-04 Thread Bertrand Dechoux
imestamp on a syslog event will be delivered. So you have the > availability to automatically create directories for year, month, day, hour > or something like that. > > Best, > Alex > > On Jan 4, 2013, at 3:22 PM, Bertrand Dechoux wrote: > > > Hi, > > > > I

Re: flume to HDFS log event write

2013-01-09 Thread Bertrand Dechoux
cation running in debugmode > > ** ** > > ** ** > > -- > The contents of this e-mail and any attachment(s) may contain confidential > or privileged information for the intended recipient(s). Unintended > recipients are prohibited from taking action on the basis of information in > this e-mail and using or disseminating the information, and must notify the > sender and delete it from their system. L&T Infotech will not accept > responsibility or liability for the accuracy or completeness of, or the > presence of any virus or disabling code in this e-mail" > -- Bertrand Dechoux

Re: Analysis of Data

2013-02-07 Thread Bertrand Dechoux
>From by point of view, while you could do it, flume is not really meant to do it. It's not a CEP engine. But it can be extended. So it really depends on the complexity of your analysis. Bertrand On Thu, Feb 7, 2013 at 11:24 AM, Surindhar wrote: > Thanks Nitin, > > On Thu, Feb 7, 2013 at 3:45 P

Re: Sinking data to a Node.js server/listener

2013-02-17 Thread Bertrand Dechoux
've been looking at creating a custom sink to do this; I've not written > one before, much less written anything in Java, so that'd be new to me. > Any pointers? > > Erik > > -- Bertrand Dechoux

Re: Sinking data to a Node.js server/listener

2013-02-21 Thread Bertrand Dechoux
listening on? > > Again, many thanks for the assistance! > > Erik > > > > On Sun, Feb 17, 2013 at 5:43 PM, Bertrand Dechoux wrote: > >> Should it be assumed that structured data are transferred using flume and >> that's why Avro is mentioned? One easy soluti

Short guide to custom configuration provider?

2013-03-29 Thread Bertrand Dechoux
Hi, Flume NG has a pluggable configuration system called the configuration provider. I cloned the source repository and I indeed found it but I am a bit lost. Is there some kind of short guide explaining how to write a custom configuration provider? I would like to start with something simple : u

Re: Short guide to custom configuration provider?

2013-03-29 Thread Bertrand Dechoux
me/node/PropertiesFileConfigurationProvider.java > > and then'd you have to customize the Application class to "plugin" your > own implementation: > > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java

Re: Short guide to custom configuration provider?

2013-03-29 Thread Bertrand Dechoux
as a third party jar and then selecting it with the help of the command line. Bertrand On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux wrote: > Thanks for the hints! > > The part that you quoted is in fact from the flume wiki and I forgot to > quote it myself. But the page is quit

Re: Short guide to custom configuration provider?

2013-04-02 Thread Bertrand Dechoux
just fall > back to the current mechanism. If you submit a patch, I'd be glad to review > it :-) > > > Thanks > Hari > > -- > Hari Shreedharan > > On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote: > > It seems quite easier than dealing with

Re: Motivation behind the name

2014-03-22 Thread Bertrand Dechoux
And that's the logo. Bertrand On Sat, Mar 22, 2014 at 9:08 AM, Muhammad Asif Abbasi wrote: > Many thanks Steve. > > > On Fri, Mar 21, 2014 at 8:26 PM, Steve Katz wrote: > >> http://en.m.wikipedia.org/wiki/Flume >> >> Its for moving 'log' files. Log, as in wood and log as in data in a file. >>

Re: Realtime sensor's tcpip data to hadoop

2014-05-16 Thread Bertrand Dechoux
Flume is indeed something you should look into. 'Log files' is a simplification. Flume really handles events and yes logs are a common kind of event but not the only one. Regards Bertrand On Wed, May 7, 2014 at 4:48 AM, Alex Lee wrote: > Sensors' may send tcpip data to server. Each sensor may

Re: Skippin those gost darn 0 byte diles

2014-07-22 Thread Bertrand Dechoux
s the same operation : metadata change only). Bertrand Dechoux On Wed, Jul 23, 2014 at 12:16 AM, Edward Capriolo wrote: > Here is the stack trace... > > Caused by: java.io.EOFException > at java.io.DataInputStream.readByte(DataInputStream.java:267) > at org.apach

Syslog-ng, Flume (old gen) and metadata

2012-10-29 Thread Bertrand Dechoux
Hi, I would like to deepen my understanding of syslog-ng and flume integration. I had initial three questions : 1) Does flume timestamp is the same as the received syslog-ng event timestamp? 2) Are the syslog-ng metadata kept by flume as extra metadata (> syslog-ng OSE 3.0)? 3) Are the extra meta