Hi,
Flume writes to HDFS(we use Cloudera 4.1.2 release and Flume 1.3.1) using the
HDFS nameservice which points to 2 namenodes (one of which is active and the
other is standby). When the HDFS service is restarted, the namenode which comes
up first becomes active. If the active namenode was swapped as a result of the
HDFS restart, then, we see the below error:
* Do we need to ensure that Flume is shutdown prior to an HDFS restart?
* The Hadoop documentation mentioned that using the nameservice as the
HDFS file destination ensures that the Hadoop client would look at both the
namenodes and determine the currently active namenode, then, perform
writes/reads from the currently active namenode. Is this not true with the HDFS
sink ?
* What is the general practice around what needs to be done with Flume
when the HDFS service parameters are changed and then restarted?
25 Feb 2013 08:26:59,836 WARN [hdfs-hdfs-sink1-call-runner-5]
(org.apache.flume.sink.hdfs.BucketWriter.append:384) - Caught IOException
while closing file (hdfs://nameservice1/*/event.1361494307973.tmp). Exception
follows.
java.net.ConnectException: Call From flume* to namenode-v01-00b.*:8020 failed
on connection exception: java.net.ConnectException: Connection refused; For
more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)
at org.apache.hadoop.ipc.Client.call(Client.java:1164)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy11.getAdditionalDatanode(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:312)
at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy12.getAdditionalDatanode(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:846)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:958)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:755)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:424)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:523)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:488)
at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:476)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:570)
at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:220)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1213)
at org.apache.hadoop.ipc.Client.call(Client.java:1140)
... 13 more
