Re: FileChannel Replays consistently take a long time

I've tried out an earlier version of it that helped a lot. Hari was making changes to not require a user to change their memory settings. We try to stay on stable release versions of Flume in prod, so we'll look to upgrade then. On Thu, Aug 28, 2014 at 10:47 PM, Gautham Varada wrote: > Thanks

Re: FileChannel Replays consistently take a long time

Thanks Gary. Have you tried this patch ? Sent from my iPhone > On 29-Aug-2014, at 7:47 am, Gary Malouf wrote: > > Check out https://issues.apache.org/jira/browse/FLUME-2450 > > >> On Thu, Aug 28, 2014 at 10:01 PM, Gautham Varada wrote: >> I’ve the same issue but in my case the data on the fi

Re: FileChannel Replays consistently take a long time

Check out https://issues.apache.org/jira/browse/FLUME-2450 On Thu, Aug 28, 2014 at 10:01 PM, Gautham Varada wrote: > I’ve the same issue but in my case the data on the file channel is > 50g > and flume is just spinning the CPU and ports haven’t opened in the last 8 > hrs.

FileChannel Replays consistently take a long time

I’ve the same issue but in my case the data on the file channel is > 50g and flume is just spinning the CPU and ports haven’t opened in the last 8 hrs.

Re: FileChannel Replays consistently take a long time

t; (org.apache.flume.channel.file.FlumeEventQueue.:114) - QueueSet > > population inserting 22678800 took 1874849 > > > > > -- Forwarded message -- > From: Hari Shreedharan > Date: Wed, Aug 20, 2014 at 12:42 AM > Subject: Re: FileChannel Replays consistently take

Fwd: FileChannel Replays consistently take a long time

[lifecycleSupervisor-1-1] (org.apache.flume.channel.file.FlumeEventQueue.:114) - QueueSet population inserting 22678800 took 1874849 -- Forwarded message -- From: Hari Shreedharan Date: Wed, Aug 20, 2014 at 12:42 AM Subject: Re: FileChannel Replays consistently take a long

Re: FileChannel Replays consistently take a long time

Are you running on EBS or ephemeral storage? I have seen IO being slow on AWS when EBS with provisioned IO is not used. This might be what you are seeing. Also what do you see as checkpoint size when the channel starts up? On Tue, Aug 19, 2014 at 9:18 PM, Gary Malouf wrote: > To be clear, we h

Re: FileChannel Replays consistently take a long time

To be clear, we have this load handled across 3 EC2 instances running Flume so each individually we are asking to handle 3.3k (5k). With 16GB of data in the channel, I would have expected the replay to be faster. On Wed, Aug 20, 2014 at 12:12 AM, Gary Malouf wrote: > Our capacity setting is: >

Re: FileChannel Replays consistently take a long time

Our capacity setting is: agent-1.channels.trdbuy-bid-req-ch1.capacity = 1 Our current channel size can not be accessed because it still is in this odd 'replay' mode. There's not logs, but the cpu is cranking on the flume node and the avro source ports have not yet opened. The pattern w

Re: FileChannel Replays consistently take a long time

How large is your channel (and how long does it take to replay?) Gary Malouf wrote: For the record, we are using Flume 1.4.0 packaged with CDH5.0.2 On Tue, Aug 19, 2014 at 11:55 PM, Gary Malouf mailto:malouf.g...@gmail.com>> wrote: We are repeatedly running into cases where the replays of fr

FileChannel Replays consistently take a long time

We are repeatedly running into cases where the replays of from a file channel going to HDFS take an eternity. I've read this thread , but I just am not convinced

Re: FileChannel Replays consistently take a long time

For the record, we are using Flume 1.4.0 packaged with CDH5.0.2 On Tue, Aug 19, 2014 at 11:55 PM, Gary Malouf wrote: > We are repeatedly running into cases where the replays of from a file > channel going to HDFS take an eternity. > > I've read this thread >