On 06/10/2015 10:52, Steve Loughran wrote:
HADOOP-11127, "Improve versioning and compatibility support in native library for downstream hadoop-common users." says "we need to do better here", which is probably some way of packaging native libs.
From that JIRA:
Colin Patrick McCabe added a comment - 18/Apr/15 00:48 I was thinking we: 1. Add the Hadoop release version to libhadoop.so. It's very, very simple and solves a lot of problems here. 2. Remove libhadoop.so and libhdfs.so from the release tarball, since they are CPU and OS-specific and the tarballs are not 3. Schedule some follow-on work to include the native libraries inside jars, as Chris suggested. This will take longer but ultimately be the best solution.
And:
I just spotted one: HADOOP-10027. A field was removed from the Java layer, which still could get referenced by an older version of the native layer. A backwards-compatible version of that patch would preserve the old fields in the Java layer.
I've been thinking about this and I really don't think the strategy of trying to shim old methods and fields back in to Hadoop is the correct one. The current Java-JNI interactions have been developed in an ad-hoc manner with no formal API definition and are explicitly Not-An-Interface and as a result no consideration has been given to cross-version stability. A compatibility shim approach is neither sustainable nor maintainable even on a single platform, and will severely compromise efforts to get Hadoop native components working on other platforms.
The approach suggested in HADOOP-11127 seems a much better way forward, in particular #2 (versioned libhadoop). As pointed out in the JIRA, #1 (freeze libahdoop forever) is an obvious non-starter, and #3 (distribute libahadoop inside the JAR) is also a non-starter as it will not work cross-platform.
I'm happy to work on HADOOP-10027 and make that a prerequisite for fixing the Solaris DomainSocket issues discussed in this thread. I believe it's not practical to provide a fix for DomainSocket on Solaris with a 'No JNI signature changes' restriction.
Does that sound acceptable? If so I can draft up a proposal for native library version and platform naming, library search locations etc.
Thanks, -- Alan Burlison --
