yes, i could not able to see data in HDFS, and i have appended new data,but still I'm unable to see the data in HDFS /user directory.
Thanks, Prabhu On Thu, Sep 20, 2012 at 8:18 PM, Brock Noland <br...@cloudera.com> wrote: > It actually looks like it's working. Are you sure no data is showing > up in hdfs and that new data is being appended to the file you are > tailing? > > On Thu, Sep 20, 2012 at 3:48 AM, prabhu k <prabhu.fl...@gmail.com> wrote: > > Hi Brock, > > > > I have changed, as per your suggestion,but still the issue same, script > > seems stuck, pasted flume.log file. > > > > flume.log > > ========== > > 12/09/20 14:07:40 INFO lifecycle.LifecycleSupervisor: Starting lifecycle > > supervisor 1 > > 12/09/20 14:07:40 INFO node.FlumeNode: Flume node starting - agent1 > > 12/09/20 14:07:40 INFO nodemanager.DefaultLogicalNodeManager: Node > manager > > starting > > 12/09/20 14:07:40 INFO properties.PropertiesFileConfigurationProvider: > > Configuration provider starting > > 12/09/20 14:07:40 INFO lifecycle.LifecycleSupervisor: Starting lifecycle > > supervisor 10 > > 12/09/20 14:07:40 INFO properties.PropertiesFileConfigurationProvider: > > Reloading configuration file:conf/flume.conf > > 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Processing:HDFS > > 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Processing:HDFS > > 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Processing:HDFS > > 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Processing:HDFS > > 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Processing:HDFS > > 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: > > agent1 > > 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Post-validation flume > > configuration contains configuration for agents: [agent1] > > 12/09/20 14:07:40 INFO properties.PropertiesFileConfigurationProvider: > > Creating channels > > 12/09/20 14:07:40 INFO properties.PropertiesFileConfigurationProvider: > > created channel MemoryChannel-2 > > 12/09/20 14:07:40 INFO sink.DefaultSinkFactory: Creating instance of sink > > HDFS typehdfs > > > > > > On Mon, Sep 17, 2012 at 8:01 PM, Brock Noland <br...@cloudera.com> > wrote: > >> > >> Hi, > >> > >> I believe, this line: > >> agent1.sinks.HDFS.hdfs.type = hdfs > >> > >> should be: > >> agent1.sinks.HDFS.type = hdfs > >> > >> Brock > >> > >> On Mon, Sep 17, 2012 at 5:17 AM, prabhu k <prabhu.fl...@gmail.com> > wrote: > >> > Hi Users, > >> > > >> > I have followed the below link for sample text file to HDFS sink using > >> > tail > >> > source. > >> > > >> > > >> > > http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html#more > >> > > >> > I have executed flume-ng like as below command. it seems got stuck. > and > >> > attached flume.conf file. > >> > > >> > #bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf > >> > > >> > > >> > flume.conf > >> > ========== > >> > agent1.sources = tail > >> > agent1.channels = MemoryChannel-2 > >> > agent1.sinks = HDFS > >> > > >> > agent1.sources.tail.type = exec > >> > agent1.sources.tail.command = tail -F > >> > > >> > > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt > >> > agent1.sources.tail.channels = MemoryChannel-2 > >> > > >> > agent1.sources.tail.interceptors = hostint > >> > agent1.sources.tail.interceptors.hostint.type = > >> > org.apache.flume.interceptor.HostInterceptor$Builder > >> > agent1.sources.tail.interceptors.hostint.preserveExisting = true > >> > agent1.sources.tail.interceptors.hostint.useIP = false > >> > > >> > agent1.sinks.HDFS.hdfs.channel = MemoryChannel-2 > >> > agent1.sinks.HDFS.hdfs.type = hdfs > >> > agent1.sinks.HDFS.hdfs.path = hdfs://<hostname>:54310/user > >> > > >> > agent1.sinks.HDFS.hdfs.fileType = dataStream > >> > agent1.sinks.HDFS.hdfs.writeFormat = text > >> > agent1.channels.MemoryChannel-2.type = memory > >> > > >> > > >> > > >> > flume.log > >> > ========== > >> > 12/09/17 15:40:05 INFO lifecycle.LifecycleSupervisor: Starting > lifecycle > >> > supervisor 1 > >> > 12/09/17 15:40:05 INFO node.FlumeNode: Flume node starting - agent1 > >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Node > >> > manager > >> > starting > >> > 12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider: > >> > Configuration provider starting > >> > 12/09/17 15:40:05 INFO lifecycle.LifecycleSupervisor: Starting > lifecycle > >> > supervisor 10 > >> > 12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider: > >> > Reloading configuration file:conf/flume.conf > >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS > >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS > >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS > >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS > >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS > >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Added sinks: HDFS > Agent: > >> > agent1 > >> > 12/09/17 15:40:05 WARN conf.FlumeConfiguration: Configuration empty > for: > >> > HDFS.Removed. > >> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Post-validation flume > >> > configuration contains configuration for agents: [agent1] > >> > 12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider: > >> > Creating channels > >> > 12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider: > >> > created channel MemoryChannel-2 > >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Starting > >> > new > >> > configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: { > >> > source:org.apache.flume.source.ExecSource@c24c0 }} sinkRunners:{} > >> > > >> > > channels:{MemoryChannel-2=org.apache.flume.channel.MemoryChannel@140c281} > } > >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Starting > >> > Channel MemoryChannel-2 > >> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Starting > >> > Source tail > >> > 12/09/17 15:40:05 INFO source.ExecSource: Exec source starting with > >> > command:tail -F > >> > > >> > > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt > >> > > >> > Please suggest and help me on this issue. > >> > >> > >> > >> -- > >> Apache MRUnit - Unit testing MapReduce - > >> http://incubator.apache.org/mrunit/ > > > > > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/ >
