Avro 2 Avro transfer on Flume 1.3 screamingly slow

Hello, I am trying to get a decent throughput rate of transfer for an Avro sink to Avro Source transfer, but the best I've been able to get is about 300 records a minute. This thread

Re: Event breaking in flume

For the company I'm with, we looked at using the spooling directory source for multi-line and binary content, and we decided it was best to create our own source that handled our data streams. Our data consisted of 100 MB tar files generated by TeaLeaf and deposited in a directory every 10 seconds.

Re: best way to make all hdfs records in one file under a folder?

How is your data partitioned, by date? On Monday, January 20, 2014, Chen Wang wrote: > Guys, > I have flume setup to flow partitioned data to hdfs, each partition has > its own file folder. Is there a way to specify all the data under one > partition to be in one file? > I am currently using > M

Comprehensive list of command-line attributes

Can someone point to the documentation for all the command-line attributes one can invoke with flume-ng? For example, -Dflume.root.logger=INFO,console is given in the user guide as an example, but it would be nice to have an attribute reference somewhere. All the best, Chris p.s. I am using Flu

Re: Ordering of messages in flume-ng

Interesting question. I can't answer it, but I would like to know what strategies others have pursued if they have had a need to order their data after it gets to the end of the Flume pipeline. - C. On Tue, Feb 11, 2014 at 11:52 PM, Chris Schneider < ch...@christopher-schneider.com> wrote: > I

Re: Multiple flume agent on single machine

What if your sinks have to write out to destinations that have different users and different levels of authorization? How is security managed by Flume? On Mar 6, 2014 8:56 AM, "Simeon Zaharici" wrote: > Hello > > You do not need to run multiple flume agents, although this is possible. > You could

JMS Source on 1.3.0?

I know that the JMS source is new in version 1.4, but has anyone tried running this particular component on Flume 1.3.x? I can"t upgrade to 1.4 because of vendor lock-in. And I am condidering rolling my own JMS source for 1.3 if the source for 1.4 won't work. Any thoughts or recommendations?

Re: JMS Source on 1.3.0?

IBM Big Insights. On Mar 13, 2014 4:47 PM, "Brock Noland" wrote: > I think it should work to run the 1.4 JMS source in 1.3. > > Out of curiosity, which vendor isn't shipping 1.4? > > Brock > > > On Thu, Mar 13, 2014 at 4:43 PM, Christopher Shannon < &

Re: Multiple flume agent on single machine

This looks to be just what I need. Thanx, Jeff - Chris On Mar 13, 2014 4:52 PM, "Jeff Lord" wrote: > You can setup flume to use hdfs.proxyUser > > > https://cwiki.apache.org/confluence/display/FLUME/Flume+1.x+Secure+HDFS+Setup > > > On Thu, Mar 13, 2014

Re: JMS Source on 1.3.0?

#x27; and 'tools' directory from > your Flume 1.4 directory into your $BIGINSIGHTS_HOME/flume/ ( or > /opt/ibm/biginsights/flume) > 5) Test flume functionality by executing the following command in the > 'bin' directory ' ./flume-ng version' > > If ev

Re: JMS Source on 1.3.0?

; Partner.( Disclaimer: I work for IBM) > > > On 13 March 2014 23:19, Christopher Shannon wrote: > >> Kris, >> >> How does running Flume 1.4 affect your service contract with IBM? >> On Mar 13, 2014 5:02 PM, "Kris Ogirri" wrote: >> >>> H

Re: what will gracefully shut down flume?

I have also experienced this. A SIGHUP or a SIGTERM will gracefully shut it down. So look for anything in your system throwing those. Pretty much any other signal will kill it outright. On Friday, March 21, 2014, lulynn_2008 wrote: > Hi All, > > After flume agent is started at 1:10 and it shut i

Re: fileChannel always reached it's capacity

You can add more sinks to the channel to work in parallel until the channel stays drained. Just make sure you assign a unique prefix to each sink's file name. On Mar 21, 2014 4:03 AM, "Bean Edwards" wrote: > I use hdfsSink and fileChannel, it seems hdfsSink consume too slow.here > is my configura

Re: what will gracefully shut down flume?

ssions? > > > > At 2014-03-21 19:46:13,"Christopher Shannon" > > > wrote: > > I have also experienced this. A SIGHUP or a SIGTERM will gracefully shut > it down. So look for anything in your system throwing those. Pretty much > any other signal will kill

Caught IOException writing to HDFSWriter . . . File exists

Flume version 1.3.0. Upgrade not possible because of vendor bundling. I'm having trouble with this nasty exception, which is filling up our logs bigtime. So I'm soliciting recommendations. I shall be obliged - Chris Shannon Exception: 12 Mar 2014 10:36:42,689 WARN [hdfs-k11-call-runner-13] (or

Re: preserve syslog header in hdfs sink

Regex Extractor Intetcrptor can get info from the event body and add to event headers. On Mar 28, 2014 2:28 PM, "Ryan Suarez" wrote: > RTFM indicates I need the following sink properties: > > --- > hadoop-t1.sinks.hdfs1.serializer = org.apache.flume.serialization. > HeaderAndBodyTextEventSerializ

Re: Flume Configuration & topology approach

On Apr 7, 2014 9:35 AM, "Jeff Lord" wrote: > > > > > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal wrote: >> >> Hi Jeff, >> >> Yes, I am using the memory channel, and that's because I want it to be more reliable and not miss any events/messages. >> As I've read in flume documentation that the m

Re: Flume Configuration & topology approach

guarantee delivery of events please use a file channel. > https://blogs.apache.org/flume/entry/apache_flume_filechannel > > > On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon wrote: >> >> >> On Apr 7, 2014 9:35 AM, "Jeff Lord" wrote: >> &g

Re: Flume Configuration & topology approach

Got it. Thanx. On Apr 7, 2014 12:39 PM, "Jeff Lord" wrote: > No not at all. Flume's transactional model guarantees delivery between > hops. > https://blogs.apache.org/flume/entry/flume_ng_architecture > > > On Mon, Apr 7, 2014 at 10:16 AM, Christopher Shannon

Re: flume and hadoop append

Not sure what you are trying to do, but the HDFS sink appends. It's just that you have to determine what your roll-over strategy will be. Instead of every few minutes, you can set the hdfs.rollInterval=0 (disables) and set the hdfs.rollSize to however large you want your files before you roll over

Re: Is Memory Channel data lost on process stop?

Good question; I look forward to someone cogently answering it. Related, and you probably know this already, the only signals that invoke the JVM shutdown hooks (clean shutdown) are HUP and TERM. (kill -HUP ; kill -TERM ). On May 28, 2014 3:37 PM, "Otis Gospodnetic" wrote: > Hi, > > What happens

Performance on Widows vs *NIX

For development and testing, I sometimes have to run multiple agents on the same server / workstation. Recently I had to load test a second agent in a Windows environment with a near identical configuration (except for paths and more memory allocated to the JVM), but the agent in the second JVM soo

Re: Kill flume job

Charles, To kill the Flume job AND invoke the shutdown hooks for a clean shutdown, use kill -HUP or kill -TERM. Any other signal will kill the JVM without invoking the shutdown hooks. On Fri, Aug 1, 2014 at 1:14 PM, Charles Robertson < charles.robert...@gmail.com> wrote: > Hi all, > > I'm playi

Re: Kill flume job

Also, flume will notice that you changed the configuration file and will reload itself with the changes. On Fri, Aug 1, 2014 at 1:16 PM, Christopher Shannon wrote: > Charles, > > To kill the Flume job AND invoke the shutdown hooks for a clean shutdown, > use kill -HUP or kill -TER

Re: Performance on Widows vs *NIX

e, Jul 29, 2014 at 10:15 AM, Christopher Shannon < > cshannon...@gmail.com> wrote: > >> For development and testing, I sometimes have to run multiple agents on >> the same server / workstation. Recently I had to load test a second agent >> in a Windows environment wi

Re: Performance on Widows vs *NIX

me distro ? Hard to say without >> additional info. anything interesting in the logs when it crashes ? >> >> >> On Tue, Jul 29, 2014 at 10:15 AM, Christopher Shannon < >> cshannon...@gmail.com> wrote: >> >>> For development and testing, I sometimes

Re: Performance on Widows vs *NIX

goes away. > -roshan > > > On Sat, Aug 2, 2014 at 6:19 AM, Christopher Shannon > wrote: > >> I do want to add that the Windows agents in our configuration were >> upstream from the agent using the HDFS sink, and the upstream agents would >> not recover gracefully w

Re: Performance on Widows vs *NIX

id not happen (since the direct buffers are > actually cleaned up only on full gc). > > > On Mon, Aug 4, 2014 at 2:57 PM, Christopher Shannon > wrote: > >> The one we are using is the bzip2 codec. That is something we could test. >> >> >> On Monday, August 4, 201

Re: Running an agent

flume-ng blah blah & runs in backround On Aug 8, 2014 8:19 AM, "Charles Robertson" wrote: > Hi all, > > I'm currently running my agent by ssh'ing in to the box and executing the > flume-ng agent command. > > This seems to have two effects (but please correct me if I have this > wrong): > 1. It s