Ashish, I was continuously collecting the details of heap memory "jstat -gc ".During flume stops processing the event processing i don't see any peak in heap usage.just 66% heap ratio utilization.
Now I have enabled GC logging with -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:<log_file> let me know if i have any other option for getting better idea why jvm stops in mid. On Wed, Jul 30, 2014 at 9:36 PM, SaravanaKumar TR <saran0081...@gmail.com> wrote: > It looks like JvisualVM is a GUI tool. > > can you please suggest me some CLI tool to get more info about JVM process. > It looks JVM was hanged because jstack <pid> doesn't show any ..only -F > option could get some output. > > > On Wed, Jul 30, 2014 at 7:45 PM, Ashish <paliwalash...@gmail.com> wrote: > >> Watch for GC activity (either enable GC logging or use visualvm) >> VisualVM would give complete idea of what's happening inside the running >> JVM >> >> >> On Tue, Jul 29, 2014 at 11:53 AM, SaravanaKumar TR < >> saran0081...@gmail.com> wrote: >> >>> Out of curiosity to find the exact issue , i started flume with 20 MB >>> memory (default) with the option as /cv/jvendor/bin/java -Xms20m -Xmx20m >>> -Dcom.sun.management.jmxremote -XX:-HeapDumpOnOutOfMemoryError >>> -XX:HeapDumpPath=./java_pid<pid>.hprof >>> >>> Again memory channel stops processing the event after a while , but heap >>> dump is not generated here.Is the arguments i provided is corrrect ? >>> >>> Tried getting the jstack of running flume agent and seems the process >>> got hung."tail -F <logfile> process is no more running. >>> >>> From jstack -F <pid> , >>> >>> Deadlock Detection: >>> >>> No deadlocks found. >>> >>> Thread 14788: (state = BLOCKED) >>> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; >>> information may be imprecise) >>> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) >>> @bci=14, line=186 (Compiled frame) >>> - >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() >>> @bci=42, line=2043 (Compiled frame) >>> - >>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() >>> @bci=98, line=1085 (Compiled frame) >>> - >>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() >>> @bci=1, line=807 (Interpreted frame) >>> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 >>> (Interpreted frame) >>> - >>> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) >>> @bci=26, line=1130 (Interpreted frame) >>> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 >>> (Interpreted frame) >>> - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame) >>> >>> >>> Thread 14785: (state = IN_NATIVE) >>> - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 >>> (Interpreted frame) >>> - java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, >>> line=398 (Interpreted frame) >>> - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 >>> (Interpreted frame) >>> - java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame) >>> - sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() >>> @bci=1, line=52 (Interpreted frame) >>> - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() >>> @bci=55, line=388 (Interpreted frame) >>> - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=360 >>> (Interpreted frame) >>> - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame) >>> >>> >>> >>> On Thu, Jul 17, 2014 at 8:21 PM, SaravanaKumar TR < >>> saran0081...@gmail.com> wrote: >>> >>>> Thanks Natty & Ashish. >>>> >>>> I have restarted flume agent with below config.Will monitor it for >>>> couple of days whether it stops randomly. >>>> >>>> JAVA_OPTS="-Xms1g -Xmx1g -Dcom.sun.management.jmxremote >>>> -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./java_pid<pid>.hprof" >>>> >>>> thanks for all again.Hope this will work well. >>>> >>>> >>>> On 17 July 2014 12:24, SaravanaKumar TR <saran0081...@gmail.com> wrote: >>>> >>>>> thanks its really helpful. >>>>> I guess the default heap dump path is /tmp ? >>>>> >>>>> >>>>> On 17 July 2014 12:11, Ashish <paliwalash...@gmail.com> wrote: >>>>> >>>>>> Nope, a heap dump shall be generated. Please see more options at >>>>>> http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html >>>>>> >>>>>> to specify path use this -XX:HeapDumpPath=./java_pid<pid>.hprof >>>>>> >>>>>> >>>>>> On Thu, Jul 17, 2014 at 12:09 PM, SaravanaKumar TR < >>>>>> saran0081...@gmail.com> wrote: >>>>>> >>>>>>> yes , sorry I missed to update as 1 GB. >>>>>>> >>>>>>> But for out of memory error ,do we get notified in flume logs? I >>>>>>> haven't see any exception till now. >>>>>>> >>>>>>> >>>>>>> On 17 July 2014 11:55, SaravanaKumar TR <saran0081...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks Ashish , So I wil go ahead and update the flume-env,sh file >>>>>>>> with >>>>>>>> >>>>>>>> JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote >>>>>>>> -XX:-HeapDumpOnOutOfMemoryError" >>>>>>>> >>>>>>>> >>>>>>>> On 17 July 2014 11:39, Ashish <paliwalash...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Add -XX:-HeapDumpOnOutOfMemoryError parameter as well, if your >>>>>>>>> process is OOME, would generate a Heap dump. Allocate Heap based on >>>>>>>>> the >>>>>>>>> number of events you need to keep in channel. Try with 1 GB, but >>>>>>>>> calculate >>>>>>>>> according the Channel size as (average event size * number of >>>>>>>>> events), plus >>>>>>>>> object over heads. >>>>>>>>> >>>>>>>>> Please note, this is just a rough calculation, actual memory usage >>>>>>>>> would be higher. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jul 17, 2014 at 11:21 AM, SaravanaKumar TR < >>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Okay thanks , So for 128 GB , I will allocate 1 GB as a heap >>>>>>>>>> memory for flume agent. >>>>>>>>>> >>>>>>>>>> But I am surprised why there was no error registered for this >>>>>>>>>> memory issues in log file (flume.log). >>>>>>>>>> >>>>>>>>>> Do i need to check in any other logs? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 16 July 2014 21:55, Jonathan Natkins <na...@streamsets.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> That's definitely your problem. 20MB is way too low for this. >>>>>>>>>>> Depending on the other processes you're running with your system, >>>>>>>>>>> the >>>>>>>>>>> amount of memory you'll need will vary, but I'd recommend at least >>>>>>>>>>> 1GB. You >>>>>>>>>>> should define it exactly where it's defined right now, so instead >>>>>>>>>>> of the >>>>>>>>>>> current command, you can run: >>>>>>>>>>> >>>>>>>>>>> "/cv/jvendor/bin/java -Xmx1g >>>>>>>>>>> -Dflume.root.logger=DEBUG,LOGFILE......" >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 16, 2014 at 3:03 AM, SaravanaKumar TR < >>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I guess i am using defaulk values , from running flume i could >>>>>>>>>>>> see these lines "/cv/jvendor/bin/java -Xmx20m >>>>>>>>>>>> -Dflume.root.logger=DEBUG,LOGFILE......" >>>>>>>>>>>> >>>>>>>>>>>> so i guess it takes 20 mb as agent flume memory. >>>>>>>>>>>> My RAM is 128 GB.So please suggest how much can i assign as >>>>>>>>>>>> heap memory and where to define it. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 16 July 2014 15:05, Jonathan Natkins <na...@streamsets.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hey Saravana, >>>>>>>>>>>>> >>>>>>>>>>>>> I'm attempting to reproduce this, but do you happen to know >>>>>>>>>>>>> what the Java heap size is for your Flume agent? This information >>>>>>>>>>>>> leads me >>>>>>>>>>>>> to believe that you don't have enough memory allocated to the >>>>>>>>>>>>> agent, which >>>>>>>>>>>>> you may need to do with the -Xmx parameter when you start up your >>>>>>>>>>>>> agent. >>>>>>>>>>>>> That aside, you can set the byteCapacity parameter on the memory >>>>>>>>>>>>> channel to >>>>>>>>>>>>> specify how much memory it is allowed to use. It should default >>>>>>>>>>>>> to 80% of >>>>>>>>>>>>> the Java heap size, but if your heap is too small, this might be >>>>>>>>>>>>> a cause of >>>>>>>>>>>>> errors. >>>>>>>>>>>>> >>>>>>>>>>>>> Does anything get written to the log when you try to pass in >>>>>>>>>>>>> an event of this size? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Natty >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jul 16, 2014 at 1:46 AM, SaravanaKumar TR < >>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Natty, >>>>>>>>>>>>>> >>>>>>>>>>>>>> While looking further , i could see memory channal stops if a >>>>>>>>>>>>>> line comes with greater than 2 MB.Let me know which parameter >>>>>>>>>>>>>> helps us to >>>>>>>>>>>>>> define max event size of about 3 MB. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 16 July 2014 12:46, SaravanaKumar TR < >>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am asking point 1 , because in some cases I could see a >>>>>>>>>>>>>>> line in logfile around 2 MB.So i need to know what mamimum >>>>>>>>>>>>>>> event size.How >>>>>>>>>>>>>>> to measure it? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 16 July 2014 10:18, SaravanaKumar TR < >>>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Natty, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please help me to get the answers for the below queries. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1,In case of exec source , (tail -F <logfile>) , is that >>>>>>>>>>>>>>>> each line in file is considered to be a single event ? >>>>>>>>>>>>>>>> If suppose a line is considered to be a event , what is >>>>>>>>>>>>>>>> that maximum size of event supported by flume?I mean maximum >>>>>>>>>>>>>>>> characters in >>>>>>>>>>>>>>>> a line supported? >>>>>>>>>>>>>>>> 2.When event stop processing , I am not seeing "tail -F" >>>>>>>>>>>>>>>> command running in the background. >>>>>>>>>>>>>>>> I have used option like "a1.sources.r1.restart = true >>>>>>>>>>>>>>>> a1.sources.r1.logStdErr = true".. >>>>>>>>>>>>>>>> Does these config will not send any errors to flume.log if >>>>>>>>>>>>>>>> any issues in tail? >>>>>>>>>>>>>>>> Will this config doesnt try to restart the "tail -F" if its >>>>>>>>>>>>>>>> not running in the background. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3.Does flume supports all formats of data in logfile or it >>>>>>>>>>>>>>>> has any predefined data formats.. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please help me with these to understand better.. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 16 July 2014 00:56, Jonathan Natkins < >>>>>>>>>>>>>>>> na...@streamsets.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Saravana, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Everything here looks pretty sane. Do you have a record of >>>>>>>>>>>>>>>>> the events that came in leading up to the agent stopping >>>>>>>>>>>>>>>>> collection? If you >>>>>>>>>>>>>>>>> can provide the last file created by the agent, and ideally >>>>>>>>>>>>>>>>> whatever events >>>>>>>>>>>>>>>>> had come in, but not been written out to your HDFS sink, it >>>>>>>>>>>>>>>>> might be >>>>>>>>>>>>>>>>> possible for me to reproduce this issue. Would it be possible >>>>>>>>>>>>>>>>> to get some >>>>>>>>>>>>>>>>> sample data from you? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Natty >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Jul 15, 2014 at 10:26 AM, SaravanaKumar TR < >>>>>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Natty , >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Just to understand , at present my settings is as >>>>>>>>>>>>>>>>>> "flume.root.logger=INFO,LOGFILE" >>>>>>>>>>>>>>>>>> in log4j.properties , do you want me to change it to >>>>>>>>>>>>>>>>>> "flume.root.logger=DEBUG,LOGFILE" and restart the agent. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> But when I start agent , I am already starting with below >>>>>>>>>>>>>>>>>> command.I guess i am using DEBUG already but not in config >>>>>>>>>>>>>>>>>> file , while >>>>>>>>>>>>>>>>>> starting agent. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ../bin/flume-ng agent -c /d0/flume/conf -f >>>>>>>>>>>>>>>>>> /d0/flume/conf/flume-conf.properties -n a1 >>>>>>>>>>>>>>>>>> -Dflume.root.logger=DEBUG,LOGFILE >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If I do some changes in config "flume-conf.properties" or >>>>>>>>>>>>>>>>>> restart the agent , it works again and starts collecting the >>>>>>>>>>>>>>>>>> data. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> currently all my logs move to flume.log , I dont see any >>>>>>>>>>>>>>>>>> exception . >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> cat flume.log | grep "Exception" doesnt show any. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 15 July 2014 22:24, Jonathan Natkins < >>>>>>>>>>>>>>>>>> na...@streamsets.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Saravana, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Our best bet on figuring out what's going on here may be >>>>>>>>>>>>>>>>>>> to turn on the debug logging. What I would recommend is >>>>>>>>>>>>>>>>>>> stopping your >>>>>>>>>>>>>>>>>>> agents, and modifying the log4j properties to turn on DEBUG >>>>>>>>>>>>>>>>>>> logging for the >>>>>>>>>>>>>>>>>>> root logger, and then restart the agents. Once the agent >>>>>>>>>>>>>>>>>>> stops producing >>>>>>>>>>>>>>>>>>> new events, send out the logs and I'll be happy to take a >>>>>>>>>>>>>>>>>>> look over them. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Does the system begin working again if you restart the >>>>>>>>>>>>>>>>>>> agents? Have you noticed any other events correlated with >>>>>>>>>>>>>>>>>>> the agent >>>>>>>>>>>>>>>>>>> stopping collecting events? Maybe a spike in events or >>>>>>>>>>>>>>>>>>> something like that? >>>>>>>>>>>>>>>>>>> And for my own peace of mind, if you run `cat >>>>>>>>>>>>>>>>>>> /var/log/flume-ng/* | grep >>>>>>>>>>>>>>>>>>> "Exception"`, does it bring anything back? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>>>> Natty >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, Jul 15, 2014 at 2:55 AM, SaravanaKumar TR < >>>>>>>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Natty, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> This is my entire config file. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> # Name the components on this agent >>>>>>>>>>>>>>>>>>>> a1.sources = r1 >>>>>>>>>>>>>>>>>>>> a1.sinks = k1 >>>>>>>>>>>>>>>>>>>> a1.channels = c1 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> # Describe/configure the source >>>>>>>>>>>>>>>>>>>> a1.sources.r1.type = exec >>>>>>>>>>>>>>>>>>>> a1.sources.r1.command = tail -F /data/logs/test_log >>>>>>>>>>>>>>>>>>>> a1.sources.r1.restart = true >>>>>>>>>>>>>>>>>>>> a1.sources.r1.logStdErr = true >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> #a1.sources.r1.batchSize = 2 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> a1.sources.r1.interceptors = i1 >>>>>>>>>>>>>>>>>>>> a1.sources.r1.interceptors.i1.type = regex_filter >>>>>>>>>>>>>>>>>>>> a1.sources.r1.interceptors.i1.regex = resuming normal >>>>>>>>>>>>>>>>>>>> operations|Received|Response >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> #a1.sources.r1.interceptors = i2 >>>>>>>>>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.type = timestamp >>>>>>>>>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.preserveExisting = true >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> # Describe the sink >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.type = hdfs >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.path = hdfs:// >>>>>>>>>>>>>>>>>>>> testing.sck.com:9000/running/test.sck/date=%Y-%m-%d >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.writeFormat = Text >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.fileType = DataStream >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.filePrefix = events- >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollInterval = 600 >>>>>>>>>>>>>>>>>>>> ##need to run hive query randomly to check teh long >>>>>>>>>>>>>>>>>>>> running process , so we need to commit events in hdfs >>>>>>>>>>>>>>>>>>>> files regularly >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollCount = 0 >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.batchSize = 10 >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollSize = 0 >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.useLocalTimeStamp = true >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> # Use a channel which buffers events in memory >>>>>>>>>>>>>>>>>>>> a1.channels.c1.type = memory >>>>>>>>>>>>>>>>>>>> a1.channels.c1.capacity = 10000 >>>>>>>>>>>>>>>>>>>> a1.channels.c1.transactionCapacity = 10000 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> # Bind the source and sink to the channel >>>>>>>>>>>>>>>>>>>> a1.sources.r1.channels = c1 >>>>>>>>>>>>>>>>>>>> a1.sinks.k1.channel = c1 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 14 July 2014 22:54, Jonathan Natkins < >>>>>>>>>>>>>>>>>>>> na...@streamsets.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Saravana, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> What does your sink configuration look like? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>> Natty >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Fri, Jul 11, 2014 at 11:05 PM, SaravanaKumar TR < >>>>>>>>>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Assuming each line in the logfile is considered as a >>>>>>>>>>>>>>>>>>>>>> event for flume , >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1.Do we have any maximum size of event defined for >>>>>>>>>>>>>>>>>>>>>> memory/file channel.like any maximum no of characters in >>>>>>>>>>>>>>>>>>>>>> a line. >>>>>>>>>>>>>>>>>>>>>> 2.Does flume supports all formats of data to be >>>>>>>>>>>>>>>>>>>>>> processed as events or do we have any limitation. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I am just still trying to understanding why the flume >>>>>>>>>>>>>>>>>>>>>> stops processing events after sometime. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Can someone please help me out here. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>> saravana >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 11 July 2014 17:49, SaravanaKumar TR < >>>>>>>>>>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi , >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I am new to flume and using Apache Flume 1.5.0. >>>>>>>>>>>>>>>>>>>>>>> Quick setup explanation here. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Source:exec , tail –F command for a logfile. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Channel: tried with both Memory & file channel >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Sink: HDFS >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> When flume starts , processing events happens >>>>>>>>>>>>>>>>>>>>>>> properly and its moved to hdfs without any issues. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> But after sometime flume suddenly stops sending >>>>>>>>>>>>>>>>>>>>>>> events to HDFS. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I am not seeing any errors in logfile flume.log as >>>>>>>>>>>>>>>>>>>>>>> well.Please let me know if I am missing any >>>>>>>>>>>>>>>>>>>>>>> configuration here. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Below is the channel configuration defined and I >>>>>>>>>>>>>>>>>>>>>>> left the remaining to be default values. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> a1.channels.c1.type = FILE >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> a1.channels.c1.transactionCapacity = 100000 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> a1.channels.c1.capacity = 10000000 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> Saravana >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> thanks >>>>>>>>> ashish >>>>>>>>> >>>>>>>>> Blog: http://www.ashishpaliwal.com/blog >>>>>>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> thanks >>>>>> ashish >>>>>> >>>>>> Blog: http://www.ashishpaliwal.com/blog >>>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal >>>>>> >>>>> >>>>> >>>> >>> >> >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> > >
