Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/2753#issuecomment-59475022
@rxin Thanks for the preliminary pass. I've updated the PR to include
UploadBlocks and to make communication bidirectionally possible (though this
functionality is not used yet, I anticipate its eventual use for "real" upload
when efficiency is required).
I have also completed significant testing on a cluster of r3.2xlarge and
r3.8xlarge, including correctness testing as well as raw perf testing. We can
max out the 10GigE on the r3.8xlarge using low CPU utilization. In a separate
test between containers on the same machine, I was able to see ~7 GB/s with
roughly 10% total CPU utilization (mainly split between interrupt servicing and
sys). It's not clear what the bottleneck was at that speed, but 7GB/s should be
sufficient for most real networks at present and so I did not investigate
further.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
