Hi ,
I am new to flume and using Apache Flume 1.5.0. Quick setup explanation
here.
Source:exec , tail –F command for a logfile.
Channel: tried with both Memory & file channel
Sink: HDFS
When flume starts , processing events happens properly and its moved to
hdfs without any issues.
But after
still trying to understanding why the flume stops processing
events after sometime.
Can someone please help me out here.
Thanks,
saravana
On 11 July 2014 17:49, SaravanaKumar TR wrote:
> Hi ,
>
> I am new to flume and using Apache Flume 1.5.0. Quick setup explanation
> here.
>
Hi,
My issue looks little similar to your.My exec source stops collecting data
after some time.
But I like to know how you could say flume is running but ,your custom
source is not running.Let me know how you identify that.
Because in running process "ps" i could see a single process i.e. flume
working
> where as when I run through shell then it is working fine. Is your case is
> also similar.
>
> How are you starting flume, either by java program or by shell?
>
>
>
> On Mon, Jul 14, 2014 at 2:07 PM, SaravanaKumar TR
> wrote:
>
>> Hi,
>>
>> My i
k_rollInterval=300
> sink_rollCount=0
> sink_rollSize=0
> sink_batchSize=100
> channel_transactionCapacity=1000
>
> And the file which you are tailing it should add new data continuously.
>
>
>
> On Mon, Jul 14, 2014 at 4:48 PM, SaravanaKumar TR
> wrote:
>
>>
does your sink configuration look like?
>
> Thanks,
> Natty
>
>
> On Fri, Jul 11, 2014 at 11:05 PM, SaravanaKumar TR > wrote:
>
>> Assuming each line in the logfile is considered as a event for flume ,
>>
>> 1.Do we have any maximum size of event define
log/flume-ng/* | grep "Exception"`,
> does it bring anything back?
>
> Thanks!
> Natty
>
>
> On Tue, Jul 15, 2014 at 2:55 AM, SaravanaKumar TR
> wrote:
>
>> Hi Natty,
>>
>> This is my entire config file.
>>
>> # Name the components
o your HDFS sink, it might be possible for me to
> reproduce this issue. Would it be possible to get some sample data from you?
>
> Thanks,
> Natty
>
>
> On Tue, Jul 15, 2014 at 10:26 AM, SaravanaKumar TR > wrote:
>
>> Hi Natty ,
>>
>> Just to understand , at prese
I am asking point 1 , because in some cases I could see a line in logfile
around 2 MB.So i need to know what mamimum event size.How to measure it?
On 16 July 2014 10:18, SaravanaKumar TR wrote:
> Hi Natty,
>
> Please help me to get the answers for the below queries.
>
> 1,
Hi Natty,
While looking further , i could see memory channal stops if a line comes
with greater than 2 MB.Let me know which parameter helps us to define max
event size of about 3 MB.
On 16 July 2014 12:46, SaravanaKumar TR wrote:
> I am asking point 1 , because in some cases I could se
is too small, this might be a cause of
> errors.
>
> Does anything get written to the log when you try to pass in an event of
> this size?
>
> Thanks,
> Natty
>
>
> On Wed, Jul 16, 2014 at 1:46 AM, SaravanaKumar TR
> wrote:
>
>> Hi Natty,
>>
>> W
of the current command,
> you can run:
>
> "/cv/jvendor/bin/java -Xmx1g -Dflume.root.logger=DEBUG,LOGFILE.."
>
>
> On Wed, Jul 16, 2014 at 3:03 AM, SaravanaKumar TR
> wrote:
>
>> I guess i am using defaulk values , from running flume i could see these
tion, actual memory usage would
> be higher.
>
>
> On Thu, Jul 17, 2014 at 11:21 AM, SaravanaKumar TR > wrote:
>
>> Okay thanks , So for 128 GB , I will allocate 1 GB as a heap memory for
>> flume agent.
>>
>> But I am surprised why there was no error regi
yes , sorry I missed to update as 1 GB.
But for out of memory error ,do we get notified in flume logs? I haven't
see any exception till now.
On 17 July 2014 11:55, SaravanaKumar TR wrote:
> Thanks Ashish , So I wil go ahead and update the flume-env,sh file with
>
> JAVA_
ath use this -XX:HeapDumpPath=./java_pid.hprof
>
>
> On Thu, Jul 17, 2014 at 12:09 PM, SaravanaKumar TR > wrote:
>
>> yes , sorry I missed to update as 1 GB.
>>
>> But for out of memory error ,do we get notified in flume logs? I haven't
>> see any exception ti
this will work well.
On 17 July 2014 12:24, SaravanaKumar TR wrote:
> thanks its really helpful.
> I guess the default heap dump path is /tmp ?
>
>
> On 17 July 2014 12:11, Ashish wrote:
>
>> Nope, a heap dump shall be generated. Please see more options at
>> h
Hi,
I am planning to use spooling directory to move logfiles in hdfs sink.
I like to know how flume identifies the file we are moving to spool
directory is complete file or partial & its move still in progress.
if suppose a file is of large size and we started moving it to spooler
directory , ho
anks,
saravana
On Thu, Jul 17, 2014 at 8:21 PM, SaravanaKumar TR
wrote:
> Thanks Natty & Ashish.
>
> I have restarted flume agent with below config.Will monitor it for couple
> of days whether it stops randomly.
>
> JAVA_OPTS="-Xms1g -Xmx1g -Dcom
e re-read
> which will create some duplicate events.
>
>
> https://github.com/apache/flume/blob/flume-1.5/flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java#L474
>
>
> On Tue, Jul 22, 2014 at 6:15 AM, SaravanaKumar TR
> wrote:
>
>&g
its log file and stop processing.
>
> To avoid the above issues, it may be useful to add a unique identifier
> (such as a timestamp) to log file names when they are moved into the
> spooling directory."
>
>
> On Wed, Jul 23, 2014 at 10:17 AM, SaravanaKumar TR > wrote:
Business Park, Hook, Hampshire RG27 9UP
> D 01256 75 3362
> I welcome VSRE emails. Learn more at http://vsre.info/
>
>
> --
> *From:* SaravanaKumar TR [mailto:saran0081...@gmail.com]
> *Sent:* 23 July 2014 06:38
> *To:* user@flume.apache.org
&
55, line=388 (Interpreted frame)
- sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=360
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)
On Thu, Jul 17, 2014 at 8:21 PM, SaravanaKumar TR
wrote:
> Thanks Natty & Ashish.
>
> I have
(either enable GC logging or use visualvm)
> VisualVM would give complete idea of what's happening inside the running
> JVM
>
>
> On Tue, Jul 29, 2014 at 11:53 AM, SaravanaKumar TR > wrote:
>
>> Out of curiosity to find the exact issue , i started flume with 20 MB
&g
eStamps -Xloggc:
let me know if i have any other option for getting better idea why jvm
stops in mid.
On Wed, Jul 30, 2014 at 9:36 PM, SaravanaKumar TR
wrote:
> It looks like JvisualVM is a GUI tool.
>
> can you please suggest me some CLI tool to get more info about JVM process.
&
Hi,
I am using Apache flume 1.5.0.Quick setup explanation here.
Source:exec , tail –F command for a logfile.
Channel: file channel
Sink: HDFS
Use case:to move real time data from logfile to HDFS.
It appears like exec is not a reliable source , as we may data loss if
channel/source is down.
about active
> log file being appended.
>
> Anyway, spooling directory source is a way to go, as it will leave log
> files in place, just renamed.
>
> Regards,
> Ahmed
>
>
> On Mon, Oct 27, 2014 at 10:21 AM, SaravanaKumar TR > wrote:
>
>> Hi,
>
ation (Tomcat or
> similar), then you can create multiple output files via log4j.properties
> configuration without application itself knowing anything about it.
>
> Regards,
> Ahmed
>
>
> On Mon, Oct 27, 2014 at 10:56 AM, SaravanaKumar TR > wrote:
>
>> Ahmed,
>
dropped.
>
>
> Regards,
> Ahmed
>
> On Mon, Oct 27, 2014 at 11:13 AM, SaravanaKumar TR > wrote:
>
>> Thanks for comments Ahmed.
>>
>> So from your comments , I consider that flume doesn't have any reliable
>> source option for use case provided
be taken as a trade off.
>
>
> On Mon, Oct 27, 2014 at 12:09 PM, SaravanaKumar TR > wrote:
>
>> Yes I understand the concerns with this use case.
>>
>> If so we need to configure failover in this scenario , can we have it
>> like channel level ,sink channel.
>>
&
sten.
>
> Even so, some sources/producers, even they use pull approach, doesn't have
> to know how to return to the last read event. It's up to implementation.
>
> Regards,
> Ahmed
>
>
> On Mon, Oct 27, 2014 at 12:48 PM, SaravanaKumar TR > wrote:
>
>> y
Ahmed,
Thanks for your details comments.
Final point, in which cases these logging solution will be considered as a
perfect system without any tradeoffs,
On Mon, Oct 27, 2014 at 6:47 PM, Ahmed Vila wrote:
> Exactly up to the point.
>
>
>
>
> On Mon, Oct 27, 2014 at 1:57 PM
ly send all logs directly to the flume.
> Not sure how back-offs are handled, but that's the level at which
> applications should communicate.
>
> On the other hand, directory spool is mature to it's finest details,
> supported by any application, altered easily... so that
no way for Flume to distinguish which file it
> already processed and which not.
>
> On Thu, Nov 13, 2014 at 4:54 AM, SaravanaKumar TR
> wrote:
>
>> Hi Ahmed,
>>
>> I have a query with flume spool directory option.
>>
>> Is that possible to ignor
Ahmed,
Can you please let me know how we configure logrotate.conf to move logs to
flume spool directory.
Because just having rotate directly in flume directory ends up with error I
mentioned.
Thanks,
On Fri, Nov 14, 2014 at 12:54 PM, SaravanaKumar TR
wrote:
> yes got it.I think we don
34 matches
Mail list logo
