Re: How do I search the past posts for a topic?

Hi, Or if you prefer slightly prettier URLs: http://search-hadoop.com/Flume And if you start your query with "flume", you automatically limit your query to the Flume project, like this: http://www.search-hadoop.com/?q=flume+mike Otis Performance Monitoring for Solr / ElasticSearch / HBase

Using Flume *only* for tailing?

Hi, I'm wondering if Flume (its ExecSource?) can be used separately from the rest of Flume? Specifically, I need to tail a log file from a Java app and then do something with each new line. I just want my Java app to get each new line from tailing and process it in some custom way. In other wor

Re: Using Flume *only* for tailing?

c sources => Flume => one text >file in a local filesystem? > >- Alex > >On Jun 25, 2013, at 6:11 PM, Otis Gospodnetic >wrote: > >> Hi, >> >> I'm wondering if Flume (its ExecSource?) can be used separately from >> the rest of Flume? >

Elasticsearch sink that uses HTTP API

Hi, Is there an Elasticsearch sink that uses ES' HTTP API? If not, may I create a JIRA issue for it? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/

Re: Elasticsearch sink that uses HTTP API

reedharan wrote: Hi Otis, > > >We have only the one elastic search sink. Does that sink not fit your use-case >in some way? Feel free to create a jira.  > > > >Thanks, >Hari > > >On Monday, October 28, 2013 at 9:20 AM, Otis Gospodnetic wrote: >Hi, >> >

RELP support?

Hi, I was looking at Rsyslog and RELP the other day and wondering whether a RELP Source would make sense for Flume? Some links: http://www.rsyslog.com/doc/omrelp.html http://en.wikipedia.org/wiki/Reliable_Event_Logging_Protocol http://search-hadoop.com/?q=relp&fc_project=Flume Thanks, Otis -- Pe

Re: RELP support

Hi, I was looking at Rsyslog and RELP the other day and wondering whether a > RELP Source would make sense for Flume? > > Some links: > http://www.rsyslog.com/doc/omrelp.html > http://en.wikipedia.org/wiki/Reliable_Event_Logging_Protocol > http://search-hadoop.com/?q=relp&fc_project=Flume > > Than

Better sink/source/interceptor lists?

Hi, Do others think it would be valuable to Flume to: 1) more clearly list all available sources, sinks, interceptors Compare: http://flume.apache.org/FlumeUserGuide.html to: http://logstash.net/docs/1.3.2/ (scroll down a bit) I find the Logstash "table" of inputs, codecs, filters, and outputs M

Re: Better sink/source/interceptor lists?

d Floor | Pittsburgh, PA 15212 > Google Voice: 412-256-8556 | www.rdx.com > > > On Wed, Jan 8, 2014 at 2:03 PM, Brock Noland wrote: > >> +1 >> >> >> On Wed, Jan 8, 2014 at 12:56 PM, Otis Gospodnetic < >> otis.gospodne...@gmail.com> wrote: >&

Re: Better sink/source/interceptor lists?

te: > I think in the past we have prefered docs over wiki. I agree with that > sentiment. However, I don't see why we have to have all of our docs in a > single pageand generally I think your suggestions are good. > > > On Wed, Jan 8, 2014 at 3:00 PM, Otis Gospodnetic

Does File Channel write first?

Hi, Does Flume's File Channel write to disk right away? Or only after it attempts to send data to the Source? (e.g. if sending fails) I think it's the former because Channel knows nothing about Source/sending AFAIK, but am hoping for the latter. :) Thanks, Otis -- Performance Monitoring * Log A

Re: Does File Channel write first?

y commits we actually fsync to disk. You can see more details of > the design here: > https://blogs.apache.org/flume/entry/apache_flume_filechannel > > > Hari > > > On Wed, Feb 26, 2014 at 10:18 AM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > >>

Blocking Source if Channel is full?

Hi, Is it possible to configure Flume to block on reading from the Sink when the Channel is full? For example, if the source is a file being tailed, and the sink is HDFS, and writing to HDFS is not working for whatever reason, then the Channel will fill up right? Once the Channel is full, wha

Re: JMX source

Hi, It's already there... http://search-hadoop.com/?q=jmx&fc_project=Flume Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Mar 19, 2014 at 7:54 AM, Luce Sylvain wrote: > Hi, > > Simple question : is there a JMX sourc

Re: Flume ElasticSearch Sink not indexing JSON data

Hi, The ES Sink has been modified quite a bit in the trunk. Would it make sense to give that a try? Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Mon, Apr 7, 2014 at 12:08 PM, Deepak Subhramanian < deepak.subhraman...@g

Re: Import files from a directory on remote machine

Hi Jeff, On Thu, Apr 17, 2014 at 1:11 PM, Jeff Lord wrote: > Using the exec source with a tail -f is not considered a production > solution. > It mainly exists for testing purposes. > This statement surprised me. Is that the general consensus among Flume developers or users or at Cloudera? Is

Re: Import files from a directory on remote machine

e > completely clear - there is absolutely zero guarantee of event delivery > when using this source. For stronger reliability guarantees, consider the > Spooling Directory Source or direct integration with Flume via the SDK." > > -Jeff > > > On Wed, Apr 23, 2014 at 6:48

Re: Flume Jambalaya - A Flume Plugin with Multiple Components

What Wolfgang said :) I'd be interested in hearing how the File Source is different from or better than Exec Source with tail -F or https://issues.apache.org/jira/browse/FLUME-2344 - do you know? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Suppor

Re: Flume Jambalaya - A Flume Plugin with Multiple Components

Had a quick look. Added my observations about FileSource stuff in https://issues.apache.org/jira/browse/FLUME-2344 Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Fri, May 2, 2014 at 8:21 PM, Otis Gospodnetic w

Re: Some dependencies can not be found in 1.5.0 release

Hi, Maybe you can provide some more information about errors you are seeing when you do and somebody may be able to help. Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Mon, May 26, 2014 at 7:23 AM, César Monteserín

Is Memory Channel data lost on process stop?

Hi, What happens with events buffered in the Memory channel (say via the Spillable version - https://issues.apache.org/jira/browse/FLUME-1227 or via its Durable cousin - https://issues.apache.org/jira/browse/FLUME-896 ) when the process is stopped? Say the process is stopped gently and appropriat

Re: Is Memory Channel data lost on process stop?

;> >> Related, and you probably know this already, the only signals that invoke >> the JVM shutdown hooks (clean shutdown) are HUP and TERM. (kill -HUP ; >> kill -TERM ). >> On May 28, 2014 3:37 PM, "Otis Gospodnetic" >> wrote: >> >>> Hi,

Re: AWS S3 flume source

+1 for seeing S3Source, starting with a JIRA issue. But being able to dynamically add/remove S3 buckets from which to pull data seems important. Any suggestions for how to approach that? Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://semat

Re: AWS S3 flume source

> the same source? > I think the above addresses this question. > Thanks, > Natty > Thanks! Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ > > > On Thu, Jul 31, 2014 at 4:58 PM, Otis Gospodnetic < &

Configuring Flume via ZooKeeper?

Hi, In a recent thread ( http://search-hadoop.com/m/DEeB4aM4aA&subj=Re+AWS+S3+flume+source ) there was talk of configuring Flume dynamically. One option suggested was to just change the config file stored on the FS because Flume will automatically reload it when it's changed. But there was also

Re: Configuring Flume via ZooKeeper?

* Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, Aug 2, 2014 at 12:07 AM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi, > > In a recent thread ( > http://search-hadoop.com/m/DEeB4aM4aA&subj=Re+AWS+S3+flume+source ) there >

Re: AWS S3 flume source

;> with similar assumptions to how the SpoolingDirectorySource works, >>>>>>>> where >>>>>>>> each line is a separate event. Does that seem reasonable? >>>>>>>> >>>>>>>> Thanks, >>&g

Re: AWS S3 flume source

It doesn't seem to be part of FLUME-2437 >>> issue. Am I right? >>> >>> > Would you need to be able to pull files from multiple S3 >>> directories with the same source? >>> >>> I think we don't need to track multi

Re: AWS S3 flume source

ny >>>> estimate how long it will take? >>>> >>>> I think the biggest challenge here is to have dynamic >>>> configuration of Flume. It doesn't seem to be part of FLUME-2437 >>>> issue. Am I right? >>>> >>>>

Re: flume on windows ?

Hi Roshan, I haven't tried Flume on Windows yet. but are you saying that one currently cannot run the latest Flume release under Windows (until those patches are committed)? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * h

Re: What Source/Sink would you want next?

Hi Santiago, Very nice. +1 for SNMP traps :) Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Tue, Sep 23, 2014 at 9:14 AM, Santiago Mola wrote: > Hi all, > > I'm developer at Stratio, where I'm part of the d

Poll results: Flume popularity as log shipper

Hi, A few weeks ago we ran a poll to find out which log shippers people use the most. We published the results, so I thought I'd share that here because Flume is in there, of course: http://blog.sematext.com/2014/10/06/top-5-most-popular-log-shippers/ Otis -- Monitoring * Alerting * Anomaly Det

Tailing Avro or Thrift files?

Hi, Is there anything in Flume that can tail binary files, like those to which something is writing Avro or Thrift data? If not, we have a need for that at Sematext and could contribute it. Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Managem

Re: Flume HTTP sink for binary data

Hi, Shouldn't something like that exist in Flume? Jean-François, is this something you could contribute? Maybe we can use yours or work on it if you can put up a patch? If not, I think we can take one of the existing implementations you mentioned and try to improve that and contribute that.

Re: Flume HTTP sink for binary data

nt - just look at any other sink > (like hbase sink or rolling file sink) and follow the same pattern. > > > Thanks, > Hari > > > > On Mon, Oct 27, 2014 at 9:29 AM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > Hi, > > > > Shouldn

Re: Flume HTTP sink for binary data

existing already ? > > > > Best regards > > > > *From:* Otis Gospodnetic [mailto:otis.gospodne...@gmail.com] > *Sent:* lundi 27 octobre 2014 17:28 > *To:* user@flume.apache.org > *Subject:* Re: Flume HTTP sink for binary data > > > > Hi, > > > &

Re: Dynamic sink configuration

As far as I know, Flume is configured only through the config and I *think* Flume automatically reloads the config when it detects the config changed - at least that is what Ashish wrote here: http://search-hadoop.com/m/DEeB4w0kHg1 Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Lo

Flume SDK info?

Hi, I recently noticed somebody (Ashish, I believe) mentioning Flume SDK. Never heard of it before. Googled, but really only found this: https://flume.apache.org/FlumeDeveloperGuide.html#client-sdk Is there any other info/documentation available about this anywhere? What's the main use case for

Flume for JavaScript / NodeJS ?

Hi, Perhaps a bit of a strange question here, but does anyone know of anything Flume-like for NodeJS? Specifically, I'd love to find/reuse something that acts like Flume's Spillover Channel (combines MemoryChannel and FileChannel), but in JavaScript land, so I can use it from NodeJS. Any pointer

Re: Flume restart

I have to join this thread for obvious reasons ;) Bojan - you'll get better help if you share your configs. If you are monitoring Flume with something, sharing various metrics/charts will help, too. You could also run Flume under a Java profiler and see what's eating your heap. Otis -- Mon

Re: Flume for JavaScript / NodeJS ?

ns, so you'll need to check > out their maturity. > > On Fri, Dec 5, 2014 at 5:08 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > >> Hi, >> >> Perhaps a bit of a strange question here, but does anyone know of >> anything Flume-like for NodeJS?

Re: Flume - compressed files

In short, no. You can uncompress the ZIP + parse XML and the feed individual pieces of data to Flume, but not the whole ZIP. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Wed, Feb 4, 2015 at 11:48 PM, Mahade

Re: Memory channel implemented by CircularFifoQueue (purging oldest event if full)

Hi Herman, Maybe this will help: http://search-hadoop.com/m/DEeB4lfnL2/otis+oldest&subj=Re+Asynchronous+SpillableMemoryChannel IMHO it would be nice to add this to the existing Spillable Memory Channel. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elastic

Re: [ANNOUNCE] New Flume Committer - Ashish Paliwal

Congratulations, well deserved! (from Airbus 320 somewhere above Poland) > On May 8, 2015, at 19:42, Hari Shreedharan wrote: > > On behalf of the Apache Flume PMC, I am excited to welcome Ashish Paliwal as > a committer on the Apache Flume project. Ashish has actively contributed > several

Message routing in pre-NG?

Hello, I was wondering if there is any sort of message routing available in pre-NG version of Flume? Concretely, I have a stream of messages belonging to different users and I want to route some messages to one Flume Collector (e.g. based on user ID) and other messages to the other Flume Coll

Re: Flume for multi KB or MB docs?

e for free. Otis  Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm  > > From: Mike Percy >To: user@flume.apache.org; Otis Gospodnetic >Sent: Monday, October 15, 2012 6:15 PM >Subject: Re: Flume for multi KB

Re: ANN: Flume on search-hadoop.com

ch-hadoop.com/Flume > > >Thanks for setting this up! I agree it would be great to get those search >results on the Flume web site. > > >Regards >Mike > > >On Mon, Oct 15, 2012 at 8:20 PM, Otis Gospodnetic >wrote: > >Hi folks, >> >> >>We