Hi Natty, Thank you so much for letting me to know about jstack command. Now the problem got solved. In flume custom source thread execution is stopping at line "System.out.println() or e.printStackTrace()" the statements which prints on console. After knowing this thing I removed all the console print statements it worked then. I don't know what really happening behind print statements. Here is the dump of the thread where it stops execution.
============================================================================= Thread 11911: (state = IN_NATIVE) - java.io.FileOutputStream.writeBytes(byte[], int, int) @bci=0 (Interpreted frame) - java.io.FileOutputStream.write(byte[], int, int) @bci=4, line=282 (Interpreted frame) - java.io.BufferedOutputStream.flushBuffer() @bci=20, line=65 (Interpreted frame) - java.io.BufferedOutputStream.flush() @bci=1, line=123 (Interpreted frame) - java.io.PrintStream.write(byte[], int, int) @bci=30, line=432 (Compiled frame) - sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=202 (Compiled frame) - sun.nio.cs.StreamEncoder.implFlushBuffer() @bci=11, line=272 (Compiled frame) - sun.nio.cs.StreamEncoder.flushBuffer() @bci=15, line=85 (Interpreted frame) - java.io.OutputStreamWriter.flushBuffer() @bci=4, line=168 (Interpreted frame) - java.io.PrintStream.write(java.lang.String) @bci=27, line=477 (Interpreted frame) - java.io.PrintStream.print(java.lang.String) @bci=9, line=619 (Interpreted frame) - java.io.PrintStream.println(java.lang.String) @bci=6, line=756 (Interpreted frame) - org.kavayah.bigdata.flume.source.YoutubeHydrateFlumeSource2.writeComments(java.lang.String) @bci=1311, line=750 (Interpreted frame) - org.kavayah.bigdata.flume.source.YoutubeHydrateFlumeSource2$3.run() @bci=871, line=227 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) =================================================================== Upto some time there is no problem with print statements but after sometime getting error at these print statements. Once again thanks a lot Natty. Thanks, Venkatesh. On Tue, Jul 15, 2014 at 10:57 PM, Jonathan Natkins <na...@streamsets.com> wrote: > Hi Venkatesh, > > Does it reliably stop processing events after about 7 minutes, or does it > happen randomly, and just quickly? Does the program immediately start up > the Flume agent? > > Have you looked at a thread dump from the program, at all? You can use > `jstack -F <pid>` to produce a stacktrace of all the threads in the > program, which might have some useful information. Can you post a thread > dump of the agent process that is spawned from your program? > > Thanks, > Natty > > > On Tue, Jul 15, 2014 at 8:44 AM, Venkatesh B <venkateshkava...@gmail.com> > wrote: > >> Hello, >> >> I have written flume custom source and in custom source in start() method >> I'm initiating a thread. This thread "data_thread" will collect data from >> web and writes to channel processor. >> >> My data collection thread is some thing like thread starts collecting >> data from web and when data is over then sleeps for some time and then >> again fetches data from web, this process is continuous. Here the control >> will be inside the thread "data_thread" all the time, since I have used >> while(true) inside thread logic. >> >> Here thread "data_thread" is collecting data upto 7mins and then it stops >> collecting data. >> I implemented logging and there by I came to know that thread >> "data_thread" state is "BLOCKED". And not returning for ever. >> In my custom source I tried in both ways by synchronizing all methods and >> also by non synchronizing all methods but no use. >> >> This blocking occurs when I start flume agent from java program, but when >> I start flume agent from shell it is working fine data is collected >> continuously for days without any error. >> >> Any idea on why thread is going to BLOCKED state?? >> >> And also I want to know about flume custom source environment >> >> Thanking you >> Venkatesh >> >> >> >
