yes , sorry I missed to update as 1 GB. But for out of memory error ,do we get notified in flume logs? I haven't see any exception till now.
On 17 July 2014 11:55, SaravanaKumar TR <saran0081...@gmail.com> wrote: > Thanks Ashish , So I wil go ahead and update the flume-env,sh file with > > JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote > -XX:-HeapDumpOnOutOfMemoryError" > > > On 17 July 2014 11:39, Ashish <paliwalash...@gmail.com> wrote: > >> Add -XX:-HeapDumpOnOutOfMemoryError parameter as well, if your process >> is OOME, would generate a Heap dump. Allocate Heap based on the number of >> events you need to keep in channel. Try with 1 GB, but calculate according >> the Channel size as (average event size * number of events), plus object >> over heads. >> >> Please note, this is just a rough calculation, actual memory usage would >> be higher. >> >> >> On Thu, Jul 17, 2014 at 11:21 AM, SaravanaKumar TR < >> saran0081...@gmail.com> wrote: >> >>> Okay thanks , So for 128 GB , I will allocate 1 GB as a heap memory for >>> flume agent. >>> >>> But I am surprised why there was no error registered for this memory >>> issues in log file (flume.log). >>> >>> Do i need to check in any other logs? >>> >>> >>> On 16 July 2014 21:55, Jonathan Natkins <na...@streamsets.com> wrote: >>> >>>> That's definitely your problem. 20MB is way too low for this. Depending >>>> on the other processes you're running with your system, the amount of >>>> memory you'll need will vary, but I'd recommend at least 1GB. You should >>>> define it exactly where it's defined right now, so instead of the current >>>> command, you can run: >>>> >>>> "/cv/jvendor/bin/java -Xmx1g -Dflume.root.logger=DEBUG,LOGFILE......" >>>> >>>> >>>> On Wed, Jul 16, 2014 at 3:03 AM, SaravanaKumar TR < >>>> saran0081...@gmail.com> wrote: >>>> >>>>> I guess i am using defaulk values , from running flume i could see >>>>> these lines "/cv/jvendor/bin/java -Xmx20m >>>>> -Dflume.root.logger=DEBUG,LOGFILE......" >>>>> >>>>> so i guess it takes 20 mb as agent flume memory. >>>>> My RAM is 128 GB.So please suggest how much can i assign as heap >>>>> memory and where to define it. >>>>> >>>>> >>>>> On 16 July 2014 15:05, Jonathan Natkins <na...@streamsets.com> wrote: >>>>> >>>>>> Hey Saravana, >>>>>> >>>>>> I'm attempting to reproduce this, but do you happen to know what the >>>>>> Java heap size is for your Flume agent? This information leads me to >>>>>> believe that you don't have enough memory allocated to the agent, which >>>>>> you >>>>>> may need to do with the -Xmx parameter when you start up your agent. That >>>>>> aside, you can set the byteCapacity parameter on the memory channel to >>>>>> specify how much memory it is allowed to use. It should default to 80% of >>>>>> the Java heap size, but if your heap is too small, this might be a cause >>>>>> of >>>>>> errors. >>>>>> >>>>>> Does anything get written to the log when you try to pass in an event >>>>>> of this size? >>>>>> >>>>>> Thanks, >>>>>> Natty >>>>>> >>>>>> >>>>>> On Wed, Jul 16, 2014 at 1:46 AM, SaravanaKumar TR < >>>>>> saran0081...@gmail.com> wrote: >>>>>> >>>>>>> Hi Natty, >>>>>>> >>>>>>> While looking further , i could see memory channal stops if a line >>>>>>> comes with greater than 2 MB.Let me know which parameter helps us to >>>>>>> define >>>>>>> max event size of about 3 MB. >>>>>>> >>>>>>> >>>>>>> On 16 July 2014 12:46, SaravanaKumar TR <saran0081...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I am asking point 1 , because in some cases I could see a line in >>>>>>>> logfile around 2 MB.So i need to know what mamimum event size.How to >>>>>>>> measure it? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 16 July 2014 10:18, SaravanaKumar TR <saran0081...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Natty, >>>>>>>>> >>>>>>>>> Please help me to get the answers for the below queries. >>>>>>>>> >>>>>>>>> 1,In case of exec source , (tail -F <logfile>) , is that each line >>>>>>>>> in file is considered to be a single event ? >>>>>>>>> If suppose a line is considered to be a event , what is that >>>>>>>>> maximum size of event supported by flume?I mean maximum characters in >>>>>>>>> a >>>>>>>>> line supported? >>>>>>>>> 2.When event stop processing , I am not seeing "tail -F" command >>>>>>>>> running in the background. >>>>>>>>> I have used option like "a1.sources.r1.restart = true >>>>>>>>> a1.sources.r1.logStdErr = true".. >>>>>>>>> Does these config will not send any errors to flume.log if any >>>>>>>>> issues in tail? >>>>>>>>> Will this config doesnt try to restart the "tail -F" if its not >>>>>>>>> running in the background. >>>>>>>>> >>>>>>>>> 3.Does flume supports all formats of data in logfile or it has any >>>>>>>>> predefined data formats.. >>>>>>>>> >>>>>>>>> Please help me with these to understand better.. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 16 July 2014 00:56, Jonathan Natkins <na...@streamsets.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Saravana, >>>>>>>>>> >>>>>>>>>> Everything here looks pretty sane. Do you have a record of the >>>>>>>>>> events that came in leading up to the agent stopping collection? If >>>>>>>>>> you can >>>>>>>>>> provide the last file created by the agent, and ideally whatever >>>>>>>>>> events had >>>>>>>>>> come in, but not been written out to your HDFS sink, it might be >>>>>>>>>> possible >>>>>>>>>> for me to reproduce this issue. Would it be possible to get some >>>>>>>>>> sample >>>>>>>>>> data from you? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Natty >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jul 15, 2014 at 10:26 AM, SaravanaKumar TR < >>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Natty , >>>>>>>>>>> >>>>>>>>>>> Just to understand , at present my settings is as >>>>>>>>>>> "flume.root.logger=INFO,LOGFILE" >>>>>>>>>>> in log4j.properties , do you want me to change it to >>>>>>>>>>> "flume.root.logger=DEBUG,LOGFILE" and restart the agent. >>>>>>>>>>> >>>>>>>>>>> But when I start agent , I am already starting with below >>>>>>>>>>> command.I guess i am using DEBUG already but not in config file , >>>>>>>>>>> while >>>>>>>>>>> starting agent. >>>>>>>>>>> >>>>>>>>>>> ../bin/flume-ng agent -c /d0/flume/conf -f >>>>>>>>>>> /d0/flume/conf/flume-conf.properties -n a1 >>>>>>>>>>> -Dflume.root.logger=DEBUG,LOGFILE >>>>>>>>>>> >>>>>>>>>>> If I do some changes in config "flume-conf.properties" or >>>>>>>>>>> restart the agent , it works again and starts collecting the data. >>>>>>>>>>> >>>>>>>>>>> currently all my logs move to flume.log , I dont see any >>>>>>>>>>> exception . >>>>>>>>>>> >>>>>>>>>>> cat flume.log | grep "Exception" doesnt show any. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 15 July 2014 22:24, Jonathan Natkins <na...@streamsets.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Saravana, >>>>>>>>>>>> >>>>>>>>>>>> Our best bet on figuring out what's going on here may be to >>>>>>>>>>>> turn on the debug logging. What I would recommend is stopping your >>>>>>>>>>>> agents, >>>>>>>>>>>> and modifying the log4j properties to turn on DEBUG logging for >>>>>>>>>>>> the root >>>>>>>>>>>> logger, and then restart the agents. Once the agent stops >>>>>>>>>>>> producing new >>>>>>>>>>>> events, send out the logs and I'll be happy to take a look over >>>>>>>>>>>> them. >>>>>>>>>>>> >>>>>>>>>>>> Does the system begin working again if you restart the agents? >>>>>>>>>>>> Have you noticed any other events correlated with the agent >>>>>>>>>>>> stopping >>>>>>>>>>>> collecting events? Maybe a spike in events or something like that? >>>>>>>>>>>> And for >>>>>>>>>>>> my own peace of mind, if you run `cat /var/log/flume-ng/* | grep >>>>>>>>>>>> "Exception"`, does it bring anything back? >>>>>>>>>>>> >>>>>>>>>>>> Thanks! >>>>>>>>>>>> Natty >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 15, 2014 at 2:55 AM, SaravanaKumar TR < >>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Natty, >>>>>>>>>>>>> >>>>>>>>>>>>> This is my entire config file. >>>>>>>>>>>>> >>>>>>>>>>>>> # Name the components on this agent >>>>>>>>>>>>> a1.sources = r1 >>>>>>>>>>>>> a1.sinks = k1 >>>>>>>>>>>>> a1.channels = c1 >>>>>>>>>>>>> >>>>>>>>>>>>> # Describe/configure the source >>>>>>>>>>>>> a1.sources.r1.type = exec >>>>>>>>>>>>> a1.sources.r1.command = tail -F /data/logs/test_log >>>>>>>>>>>>> a1.sources.r1.restart = true >>>>>>>>>>>>> a1.sources.r1.logStdErr = true >>>>>>>>>>>>> >>>>>>>>>>>>> #a1.sources.r1.batchSize = 2 >>>>>>>>>>>>> >>>>>>>>>>>>> a1.sources.r1.interceptors = i1 >>>>>>>>>>>>> a1.sources.r1.interceptors.i1.type = regex_filter >>>>>>>>>>>>> a1.sources.r1.interceptors.i1.regex = resuming normal >>>>>>>>>>>>> operations|Received|Response >>>>>>>>>>>>> >>>>>>>>>>>>> #a1.sources.r1.interceptors = i2 >>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.type = timestamp >>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.preserveExisting = true >>>>>>>>>>>>> >>>>>>>>>>>>> # Describe the sink >>>>>>>>>>>>> a1.sinks.k1.type = hdfs >>>>>>>>>>>>> a1.sinks.k1.hdfs.path = hdfs:// >>>>>>>>>>>>> testing.sck.com:9000/running/test.sck/date=%Y-%m-%d >>>>>>>>>>>>> a1.sinks.k1.hdfs.writeFormat = Text >>>>>>>>>>>>> a1.sinks.k1.hdfs.fileType = DataStream >>>>>>>>>>>>> a1.sinks.k1.hdfs.filePrefix = events- >>>>>>>>>>>>> a1.sinks.k1.hdfs.rollInterval = 600 >>>>>>>>>>>>> ##need to run hive query randomly to check teh long running >>>>>>>>>>>>> process , so we need to commit events in hdfs files regularly >>>>>>>>>>>>> a1.sinks.k1.hdfs.rollCount = 0 >>>>>>>>>>>>> a1.sinks.k1.hdfs.batchSize = 10 >>>>>>>>>>>>> a1.sinks.k1.hdfs.rollSize = 0 >>>>>>>>>>>>> a1.sinks.k1.hdfs.useLocalTimeStamp = true >>>>>>>>>>>>> >>>>>>>>>>>>> # Use a channel which buffers events in memory >>>>>>>>>>>>> a1.channels.c1.type = memory >>>>>>>>>>>>> a1.channels.c1.capacity = 10000 >>>>>>>>>>>>> a1.channels.c1.transactionCapacity = 10000 >>>>>>>>>>>>> >>>>>>>>>>>>> # Bind the source and sink to the channel >>>>>>>>>>>>> a1.sources.r1.channels = c1 >>>>>>>>>>>>> a1.sinks.k1.channel = c1 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 14 July 2014 22:54, Jonathan Natkins <na...@streamsets.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Saravana, >>>>>>>>>>>>>> >>>>>>>>>>>>>> What does your sink configuration look like? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Natty >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Jul 11, 2014 at 11:05 PM, SaravanaKumar TR < >>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Assuming each line in the logfile is considered as a event >>>>>>>>>>>>>>> for flume , >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1.Do we have any maximum size of event defined for >>>>>>>>>>>>>>> memory/file channel.like any maximum no of characters in a line. >>>>>>>>>>>>>>> 2.Does flume supports all formats of data to be processed as >>>>>>>>>>>>>>> events or do we have any limitation. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am just still trying to understanding why the flume stops >>>>>>>>>>>>>>> processing events after sometime. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can someone please help me out here. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> saravana >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 11 July 2014 17:49, SaravanaKumar TR < >>>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi , >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am new to flume and using Apache Flume 1.5.0. Quick >>>>>>>>>>>>>>>> setup explanation here. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Source:exec , tail –F command for a logfile. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Channel: tried with both Memory & file channel >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sink: HDFS >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> When flume starts , processing events happens properly and >>>>>>>>>>>>>>>> its moved to hdfs without any issues. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> But after sometime flume suddenly stops sending events to >>>>>>>>>>>>>>>> HDFS. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am not seeing any errors in logfile flume.log as >>>>>>>>>>>>>>>> well.Please let me know if I am missing any configuration here. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Below is the channel configuration defined and I left the >>>>>>>>>>>>>>>> remaining to be default values. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> a1.channels.c1.type = FILE >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> a1.channels.c1.transactionCapacity = 100000 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> a1.channels.c1.capacity = 10000000 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Saravana >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> > >
