Yes they should work together. Please send the updated conf and log file.
-- Brock Noland Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Tuesday, September 18, 2012 at 5:49 AM, prabhu k wrote: > I have tried both ways, but still not working.... > can you please confirm flume 1.2.0 support hadoop 1.0.3 version? > > Thanks, > Prabhu. > On Tue, Sep 18, 2012 at 3:32 PM, Nitin Pawar <nitinpawar...@gmail.com > (mailto:nitinpawar...@gmail.com)> wrote: > > can you write something in file continuously after you start flume-ng > > > > if you do tail -f it will start getting only new entries > > or you can just change the command in the config file from tail -f to > > tail so each time it bring default last 10 lines from the the file > > > > ~nitin > > > > On Tue, Sep 18, 2012 at 2:51 PM, prabhu k <prabhu.fl...@gmail.com > > (mailto:prabhu.fl...@gmail.com)> wrote: > > > Hi Nitin, > > > > > > While executing flume-ng, i have updated the flume_test.txt file,still > > > unable to do HDFS sink. > > > > > > Thanks, > > > Prabhu. > > > > > > On Tue, Sep 18, 2012 at 2:35 PM, Nitin Pawar <nitinpawar...@gmail.com > > > (mailto:nitinpawar...@gmail.com)> > > > wrote: > > >> > > >> Hi Prabhu, > > >> > > >> are you sure there is continuous text being written to your file > > >> flume_test.txt. > > >> > > >> if nothing is written to that file, flume will not write anything into > > >> hdfs. > > >> > > >> On Tue, Sep 18, 2012 at 2:31 PM, prabhu k <prabhu.fl...@gmail.com > > >> (mailto:prabhu.fl...@gmail.com)> wrote: > > >> > Hi Brock, > > >> > > > >> > Thanks for the reply. > > >> > > > >> > As per your suggestion, i have modified,but still same issue. > > >> > > > >> > My hadoop version is : 1.0.3 & Flume version is 1.2.0. Please let us > > >> > know is > > >> > there any incompatible version? > > >> > > > >> > On Mon, Sep 17, 2012 at 8:01 PM, Brock Noland <br...@cloudera.com > > >> > (mailto:br...@cloudera.com)> > > >> > wrote: > > >> >> > > >> >> Hi, > > >> >> > > >> >> I believe, this line: > > >> >> agent1.sinks.HDFS.hdfs.type = hdfs > > >> >> > > >> >> should be: > > >> >> agent1.sinks.HDFS.type = hdfs > > >> >> > > >> >> Brock > > >> >> > > >> >> On Mon, Sep 17, 2012 at 5:17 AM, prabhu k <prabhu.fl...@gmail.com > > >> >> (mailto:prabhu.fl...@gmail.com)> > > >> >> wrote: > > >> >> > Hi Users, > > >> >> > > > >> >> > I have followed the below link for sample text file to HDFS sink > > >> >> > using > > >> >> > tail > > >> >> > source. > > >> >> > > > >> >> > > > >> >> > > > >> >> > http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html#more > > >> >> > > > >> >> > I have executed flume-ng like as below command. it seems got stuck. > > >> >> > and > > >> >> > attached flume.conf file. > > >> >> > > > >> >> > #bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf > > >> >> > > > >> >> > > > >> >> > flume.conf > > >> >> > ========== > > >> >> > agent1.sources = tail > > >> >> > agent1.channels = MemoryChannel-2 > > >> >> > agent1.sinks = HDFS > > >> >> > > > >> >> > agent1.sources.tail.type = exec > > >> >> > agent1.sources.tail.command = tail -F > > >> >> > > > >> >> > > > >> >> > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt > > >> >> > agent1.sources.tail.channels = MemoryChannel-2 > > >> >> > > > >> >> > agent1.sources.tail.interceptors = hostint > > >> >> > agent1.sources.tail.interceptors.hostint.type = > > >> >> > org.apache.flume.interceptor.HostInterceptor$Builder > > >> >> > agent1.sources.tail.interceptors.hostint.preserveExisting = true > > >> >> > agent1.sources.tail.interceptors.hostint.useIP = false > > >> >> > > > >> >> > agent1.sinks.HDFS.hdfs.channel = MemoryChannel-2 > > >> >> > agent1.sinks.HDFS.hdfs.type = hdfs > > >> >> > agent1.sinks.HDFS.hdfs.path = hdfs://<hostname>:54310/user > > >> >> > > > >> >> > agent1.sinks.HDFS.hdfs.fileType = dataStream > > >> >> > agent1.sinks.HDFS.hdfs.writeFormat = text > > >> >> > agent1.channels.MemoryChannel-2.type = memory > > >> >> > > > >> >> > > > >> >> > > > >> >> > flume.log > > >> >> > ========== > > >> >> > 12/09/17 15:40:05 INFO lifecycle.LifecycleSupervisor: Starting > > >> >> > lifecycle > > >> >> > supervisor 1 > > >> >> > 12/09/17 15:40:05 INFO node.FlumeNode: Flume node starting - agent1 > > >> >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Node > > >> >> > manager > > >> >> > starting > > >> >> > 12/09/17 15:40:05 INFO > > >> >> > properties.PropertiesFileConfigurationProvider: > > >> >> > Configuration provider starting > > >> >> > 12/09/17 15:40:05 INFO lifecycle.LifecycleSupervisor: Starting > > >> >> > lifecycle > > >> >> > supervisor 10 > > >> >> > 12/09/17 15:40:05 INFO > > >> >> > properties.PropertiesFileConfigurationProvider: > > >> >> > Reloading configuration file:conf/flume.conf > > >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS > > >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS > > >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS > > >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS > > >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS > > >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Added sinks: HDFS > > >> >> > Agent: > > >> >> > agent1 > > >> >> > 12/09/17 15:40:05 WARN conf.FlumeConfiguration: Configuration empty > > >> >> > for: > > >> >> > HDFS.Removed. > > >> >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Post-validation > > >> >> > flume > > >> >> > configuration contains configuration for agents: [agent1] > > >> >> > 12/09/17 15:40:05 INFO > > >> >> > properties.PropertiesFileConfigurationProvider: > > >> >> > Creating channels > > >> >> > 12/09/17 15:40:05 INFO > > >> >> > properties.PropertiesFileConfigurationProvider: > > >> >> > created channel MemoryChannel-2 > > >> >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: > > >> >> > Starting > > >> >> > new > > >> >> > configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: { > > >> >> > source:org.apache.flume.source.ExecSource@c24c0 }} sinkRunners:{} > > >> >> > > > >> >> > > > >> >> > channels:{MemoryChannel-2=org.apache.flume.channel.MemoryChannel@140c281} > > >> >> > } > > >> >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: > > >> >> > Starting > > >> >> > Channel MemoryChannel-2 > > >> >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: > > >> >> > Starting > > >> >> > Source tail > > >> >> > 12/09/17 15:40:05 INFO source.ExecSource: Exec source starting with > > >> >> > command:tail -F > > >> >> > > > >> >> > > > >> >> > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt > > >> >> > > > >> >> > Please suggest and help me on this issue. > > >> >> > > >> >> > > >> >> > > >> >> -- > > >> >> Apache MRUnit - Unit testing MapReduce - > > >> >> http://incubator.apache.org/mrunit/ > > >> > > > >> > > > >> > > >> > > >> > > >> -- > > >> Nitin Pawar > > > > > > > > > > > > > > -- > > Nitin Pawar >
