Hey Sarvana I think you seem to be at very basic level of start. Just try with the following configuration,
channel_capacity=1000 sink_rollInterval=300 sink_rollCount=0 sink_rollSize=0 sink_batchSize=100 channel_transactionCapacity=1000 And the file which you are tailing it should add new data continuously. On Mon, Jul 14, 2014 at 4:48 PM, SaravanaKumar TR <saran0081...@gmail.com> wrote: > No i am using flume provided source exec , just to tail -F <logfile> > > I start flume as below, > > /bin/flume-ng agent -c /d0/flume/conf -f > /d0/flume/conf/flume-conf.properties -n a1 > -Dflume.root.logger=DEBUG,LOGFILE & > > I get logs in flume.log file.I could see flume agent running from ps. > > I use file/memory channel..Agents data for sometime but stops after that , > but it doesnt happen in certain interval as you mentioned 6 hrs , it > happens randomly. > > I think channel might gets full.NOt sure how to verfiy the same.Any idea? > > Please tell me how to enable logging to identify when source stops. > > > On 14 July 2014 16:04, Venkatesh B <venkateshkava...@gmail.com> wrote: > >> Hi sarvana, >> >> My flume agent is surely running, when I see in running processes ps -ef >> it is showing flume agent process is running.But my custom source is not >> running. >> I have implemented logging, if flume custom source is running then it >> will write to log files. I have written many log statements I know where it >> is stopping, but not know why it is stopping. >> >> My custom source works fine as when I run from shell directly but not >> from java program, then custom source is running for many hours. >> And also flume is working fine I have written other custom sources before >> and they are running for days continuously when I start from java program. >> Here I'm not understanding what is happening internally, when I start to >> run flume through java program then only my custom source stops working >> where as when I run through shell then it is working fine. Is your case is >> also similar. >> >> How are you starting flume, either by java program or by shell? >> >> >> >> On Mon, Jul 14, 2014 at 2:07 PM, SaravanaKumar TR <saran0081...@gmail.com >> > wrote: >> >>> Hi, >>> >>> My issue looks little similar to your.My exec source stops collecting >>> data after some time. >>> >>> But I like to know how you could say flume is running but ,your custom >>> source is not running.Let me know how you identify that. >>> >>> Because in running process "ps" i could see a single process i.e. flume >>> agent running.How we identity if source is running or not. >>> >>> Thanks, >>> Saravana >>> >>> >>> On 14 July 2014 13:48, Venkatesh B <venkateshkava...@gmail.com> wrote: >>> >>>> Hello, >>>> >>>> I'm running flume in Cent os 6.3 version. >>>> Apache flume version 1.3 >>>> >>>> I had written many FLUME CUSTOM SOURCES, >>>> All sources are working fine, except two sources. >>>> >>>> When I start flume i.e., flume-ng ...... command from shell it is >>>> working fine, flume is collecting data continuously for 6hours I tested it >>>> for many times about 10 times. >>>> >>>> But when I start flume from java program, i.e., In one ".sh" file I >>>> will write "flume-ng --conf path to conf folder -f flume configuration file >>>> path -n agent" command, And from java program I will run this .sh file by >>>> calling Runtime.getRuntime().exec(.sh file path).waitFor(). >>>> >>>> >>>> In flume custom source I have written logic for collecting data from >>>> web and then I will write that data to channel processor. Here sink is >>>> hdfs. >>>> >>>> Here the PROBLEM is flume is collecting data about 7 minutes and then >>>> the not collecting when I start flume from java program. >>>> >>>> When I search the running processes it is showing that flume is running >>>> but my custom sources is not running. >>>> I implemented logging for the flume custom source by logging what I >>>> came to know is that flume custom source is not running at the same time no >>>> error is caught. >>>> >>>> Here flume custom source is stopping without any intimation of error. >>>> >>>> What is the difference between running flume directly from shell and >>>> through java program? >>>> Why it is stopping did anyone face this problem? >>>> >>>> What actually my flume custom source code looks is >>>> in start() method I'm creating a thread and this thread will collect >>>> data. >>>> >>>> >>>> >>>> Thanking you.. >>>> >>>> >>>> >>> >> >
