0.20.2 is too old. The current stable release of hadoop is 1.2.1.
Anyway, for 0.20.2, you probably need -Djava.net.preferIPv4Stack=true as
well, which is already set for Hadoop 1+. BTW, you'll need to set all these
in mapred.child.java.opts as well.
On Thu, Oct 10, 2013 at 5:12 AM, Yasin Celik
Hi Luke,
I actually tested hadoop RPC using SDP. But it did not work for me.
I used hadoop-core 0.20.2
Following is the error I received:
java.io.IOException: Address family not supported by protocol
at sun.net.sdp.SdpSupport.
convert0(Native Method)
at sun.net.sdp.SdpSupport.convertSocke
On 10 October 2013 09:00, Luke Lu wrote:
> Hadoop works with Java 7, which has builtin SDP support. You'll need to
> pass SDP related properties (-Dcom.sun.sdp.conf=your_sdp.conf) to hadoop
> JVMs via env var HADOOP_OPTS.
>
> One caveat of SDP is the 64KB message size (performance cliff beyond 64
Hadoop works with Java 7, which has builtin SDP support. You'll need to
pass SDP related properties (-Dcom.sun.sdp.conf=your_sdp.conf) to hadoop
JVMs via env var HADOOP_OPTS.
One caveat of SDP is the 64KB message size (performance cliff beyond 64KB),
which _should_ work with default hadoop config,
I cannot believe i made the same mistake twice, of reply-to to mailing list. I
need to get off the outlook client.
once again, sorry folks. this was supposed to go to steve's personal email.
- milind
Sent from my iPhone
> On Oct 9, 2013, at 8:47, Suresh Srinivas wrote:
>
> Milind, please sto
Milind, please stop this. The topic here is not what Steve's employers
wants to sell or recommend. Please stick to the technical issue. This is
the second time this week where a thread unnecessarily goes beyond
technical issues. If you have an axe to grind, please keep it off this
forum. It is gett
No, Steve, I meant exactly what I wrote. One day, when you are here, Let's meet
in the same chinese restaurant, and I will give you the numbers on performance
and cost, and let us do the division of these two numbers.
Let's talk about how the nn latency becomes a bottleneck for rest of the
clus
oops, didn't finish a sentence:
On 9 October 2013 09:45, Steve Loughran wrote:
>
>
>
> IB should be most interesting at the app level -for apps > classic MR.
> That's giraph, streaming work, Tez. I'd like to see some numbers there. As
> the oracle
>
As the oracle exadata appliance has IB, ther
On 9 October 2013 01:57, Milind Bhandarkar wrote:
> Yes, we have. It works very well, but it is considered too niche by folks
> who insist on buying the least capable hardware for their test clusters,
> and therefore, recommend such underpowered clusters to customers as well.
>
surely you meant
Have you checked out Mellanox's UDA ?
There have been a lot of efforts from RoCE providers to get Hadoop
optimized implementations with modern networking hardware. Even RedHat has
announced HPN (High Performance Networking) interfaces to allow modern
networks. It might take a few years to get ther
Hello
How do you get hadoop worked with infiniband? Do you use SDP?
I only need to use RPC of hadoop for now. Any help on how to get it worked?
Thanks
Yasin
On Tue, Oct 8, 2013 at 8:57 PM, Milind Bhandarkar wrote:
> Yes, we have. It works very well, but it is considered too niche by folks
> w
Yes, we have. It works very well, but it is considered too niche by folks who
insist on buying the least capable hardware for their test clusters, and
therefore, recommend such underpowered clusters to customers as well.
- milind
> On Oct 8, 2013, at 16:49, Yasin Celik wrote:
>
> Hello,
>
>
Hello,
I looked around but I think it seems there is no Sockets Direct
Protocol(SDP) support in Hadoop RPC.
I basically want to take advantage of Infiniband when transferring data.
Is there any known support for this? Has anybody tried this before?
Best
-
Yasin Celik
13 matches
Mail list logo
