Re: how flume identifies a file transfer is complete or not

2014-07-25 Thread Jonathan Natkins
Hi Anand, What you're doing is a slightly odd way to use Flume. With the exec source, Flume will execute that command, and consume the output as events. Often the exec source is used to tail -F a file, which allows you to pipe more data to the file and ingest additional events. By using cat, Flume

Re: how flume identifies a file transfer is complete or not

2014-07-25 Thread Jayant Shekhar
Hi Anand, +1 to Natty. Also, if you are looking to move files into HDFS, check out Spooling Directory Source. https://flume.apache.org/FlumeUserGuide.html Thanks, Jayant On Fri, Jul 25, 2014 at 10:52 AM, Jonathan Natkins wrote: > Hi Anand, > > What you're doing is a slightly odd way to use

Re: how flume identifies a file transfer is complete or not

2014-07-25 Thread Anandkumar Lakshmanan
Hi Natty, Thanks for the Reply. So far I am verifying the transfer is complete or not by checking the file in the destination or as you mentioned only. Thanks Anand. On 07/25/2014 11:22 PM, Jonathan Natkins wrote: Hi Anand, What you're doing is a slightly odd way to use Flume. With the ex