Thanks venkatesh .I already have this config in place.Flume is writing data in hdfs till some point.
After some random point , it stops writing , I dont see any *.tmp file created in hdfs but still flume agent is running. I am not sure why i stops writing data..but still logfile produces data continuously. Is that any case where channel gets full or cannot push event to hdfs ? On 14 July 2014 18:34, Venkatesh B <venkateshkava...@gmail.com> wrote: > Hey Sarvana I think you seem to be at very basic level of start. > > Just try with the following configuration, > > channel_capacity=1000 > sink_rollInterval=300 > sink_rollCount=0 > sink_rollSize=0 > sink_batchSize=100 > channel_transactionCapacity=1000 > > And the file which you are tailing it should add new data continuously. > > > > On Mon, Jul 14, 2014 at 4:48 PM, SaravanaKumar TR <saran0081...@gmail.com> > wrote: > >> No i am using flume provided source exec , just to tail -F <logfile> >> >> I start flume as below, >> >> /bin/flume-ng agent -c /d0/flume/conf -f >> /d0/flume/conf/flume-conf.properties -n a1 >> -Dflume.root.logger=DEBUG,LOGFILE & >> >> I get logs in flume.log file.I could see flume agent running from ps. >> >> I use file/memory channel..Agents data for sometime but stops after that >> , but it doesnt happen in certain interval as you mentioned 6 hrs , it >> happens randomly. >> >> I think channel might gets full.NOt sure how to verfiy the same.Any idea? >> >> Please tell me how to enable logging to identify when source stops. >> >> >> On 14 July 2014 16:04, Venkatesh B <venkateshkava...@gmail.com> wrote: >> >>> Hi sarvana, >>> >>> My flume agent is surely running, when I see in running processes ps -ef >>> it is showing flume agent process is running.But my custom source is not >>> running. >>> I have implemented logging, if flume custom source is running then it >>> will write to log files. I have written many log statements I know where it >>> is stopping, but not know why it is stopping. >>> >>> My custom source works fine as when I run from shell directly but not >>> from java program, then custom source is running for many hours. >>> And also flume is working fine I have written other custom sources >>> before and they are running for days continuously when I start from java >>> program. >>> Here I'm not understanding what is happening internally, when I start to >>> run flume through java program then only my custom source stops working >>> where as when I run through shell then it is working fine. Is your case is >>> also similar. >>> >>> How are you starting flume, either by java program or by shell? >>> >>> >>> >>> On Mon, Jul 14, 2014 at 2:07 PM, SaravanaKumar TR < >>> saran0081...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> My issue looks little similar to your.My exec source stops collecting >>>> data after some time. >>>> >>>> But I like to know how you could say flume is running but ,your custom >>>> source is not running.Let me know how you identify that. >>>> >>>> Because in running process "ps" i could see a single process i.e. flume >>>> agent running.How we identity if source is running or not. >>>> >>>> Thanks, >>>> Saravana >>>> >>>> >>>> On 14 July 2014 13:48, Venkatesh B <venkateshkava...@gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> I'm running flume in Cent os 6.3 version. >>>>> Apache flume version 1.3 >>>>> >>>>> I had written many FLUME CUSTOM SOURCES, >>>>> All sources are working fine, except two sources. >>>>> >>>>> When I start flume i.e., flume-ng ...... command from shell it is >>>>> working fine, flume is collecting data continuously for 6hours I tested it >>>>> for many times about 10 times. >>>>> >>>>> But when I start flume from java program, i.e., In one ".sh" file I >>>>> will write "flume-ng --conf path to conf folder -f flume configuration >>>>> file >>>>> path -n agent" command, And from java program I will run this .sh file by >>>>> calling Runtime.getRuntime().exec(.sh file path).waitFor(). >>>>> >>>>> >>>>> In flume custom source I have written logic for collecting data from >>>>> web and then I will write that data to channel processor. Here sink is >>>>> hdfs. >>>>> >>>>> Here the PROBLEM is flume is collecting data about 7 minutes and then >>>>> the not collecting when I start flume from java program. >>>>> >>>>> When I search the running processes it is showing that flume is >>>>> running but my custom sources is not running. >>>>> I implemented logging for the flume custom source by logging what I >>>>> came to know is that flume custom source is not running at the same time >>>>> no >>>>> error is caught. >>>>> >>>>> Here flume custom source is stopping without any intimation of error. >>>>> >>>>> What is the difference between running flume directly from shell and >>>>> through java program? >>>>> Why it is stopping did anyone face this problem? >>>>> >>>>> What actually my flume custom source code looks is >>>>> in start() method I'm creating a thread and this thread will collect >>>>> data. >>>>> >>>>> >>>>> >>>>> Thanking you.. >>>>> >>>>> >>>>> >>>> >>> >> >
