Yep. The disk space calls require an NFS call for each write, and that slows
things down a lot.
--
Hari Shreedharan
On Tuesday, December 18, 2012 at 8:43 AM, Brock Noland wrote:
We'd need those thread dumps to help confirm but I bet that FLUME-1609
results in a NFS call on each operation on the channel.
If that is true, that would explain why it works well on local disk.
Brock
On Tue, Dec 18, 2012 at 10:17 AM, Brock Noland <br...@cloudera.com> wrote:
Hi,
Hmm, yes in general performance is not going to be great over NFS, but
there haven't been any FC changes that stick out here.
Could you take 10 thread dumps of the agent running the file channel
and 10 thread dumps of the agent sending data to the agent with the
file channel? (You can address them to myself directly since the list
won't take attachements.)
Are there any patterns, like it works for 40 seconds then times out
and then works for 39 seconds, etc?
Brock
On Tue, Dec 18, 2012 at 10:07 AM, Rakos, Rudolf
<rudolf.ra...@morganstanley.com> wrote:
Hi,
We’ve run into a strange problem regarding NFS and File Channel performance
while evaluating the new version of Flume.
We had no issues with the previous version (1.2.0).
Our configuration looks like this:
· Node1:
(Avro RPC Clients ->) Avro Source and Custom Sources -> File Channel -> Avro
Sink (-> Node 2)
· Node2:
(Node1s ->) Avro Source -> File Channel -> Custom Sink
Both the checkpoint and the data directories of the File Channels are on NFS
shares. We use the same share for checkpoint and data directories, but
different shares for each Node. Unfortunately it is not an option for us to
use local directories.
The events are about 1KB large, and the batch sizes are the following:
· Avro RPC Clients: 1000
· Custom Sources: 2000
· Avro Sink: 5000
· Custom Sink: 10000
We are experiencing very slow File Channel performance compared to the
previous version, and high amount of timeouts (almost always) in the Avro
RPC Clients and the Avro Sink.
Something like this:
· 2012-12-18 15:43:31,828
[SinkRunner-PollingRunner-ExceptionCatchingSinkProcessor] WARN
org.apache.flume.sink.AvroSink - Failed to send event batch
org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: ***,
port: *** }: Failed to send batch
at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:236)
~[flume-ng-sdk-1.3.0.jar:1.3.0]
***
at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
[flume-ng-core-1.3.0.jar:1.3.0]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31]
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: ***, port: *** }: Handshake timed out after 20000ms
at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:280)
~[flume-ng-sdk-1.3.0.jar:1.3.0]
at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:224)
~[flume-ng-sdk-1.3.0.jar:1.3.0]
... 5 common frames omitted
Caused by: java.util.concurrent.TimeoutException: null
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
~[na:1.6.0_31]
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
~[na:1.6.0_31]
at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:278)
~[flume-ng-sdk-1.3.0.jar:1.3.0]
... 6 common frames omitted
(I had to remove some details, sorry for that.)
We managed to narrow down the root cause of the issue to the File Channel,
because:
· Everything works fine if we switch to the Memory Channel or to the
Old File Channel (1.2.0).
· Everything works fine if we use local directories.
We’ve tested this on multiple different PCs (both Windows and Linux).
I spent the day debugging and profiling, but I could not find anything worth
mentioning (nothing with excessive CPU usage, no threads are waiting too
much, etc…). The only problem is that File Channel takes and puts take way
more time than with the previous version.
Could someone please try the File Channel on an NFS share?
Does anyone have similar issues?
Thank you for your help.
Regards,
Rudolf
Rudolf Rakos
Morgan Stanley | ISG Technology
Lechner Odon fasor 8 | Floor 06
Budapest, 1095
Phone: +36 1 881-4011
rudolf.ra...@morganstanley.com
Be carbon conscious. Please consider our environment before printing this
email.
________________________________
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions
or views contained herein are not intended to be, and do not constitute,
advice within the meaning of Section 975 of the Dodd-Frank Wall Street
Reform and Consumer Protection Act. If you have received this communication
in error, please destroy all electronic and paper copies and notify the
sender immediately. Mistransmission is not intended to waive confidentiality
or privilege. Morgan Stanley reserves the right, to the extent permitted
under applicable law, to monitor electronic communications. This message is
subject to terms available at the following link:
http://www.morganstanley.com/disclaimers If you cannot access these links,
please notify us by reply message and we will send the contents to you. By
messaging with Morgan Stanley you consent to the foregoing.
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
