Hi Arthur! Our Snappy build instructions are currently out of date and I'm working on updating them[1]. In short, I don't think there are any special build steps for using snappy.
I'm still working out what needs to be included in our instructions for local and cluster testing. If you use the test for compression options, locally things will fail because the native hadoop libs won't be present: bin/hbase org.apache.hadoop.hbase.util.CompressionTest file://*/tmp/snappy-test* snappy (for comparison, replace "snappy" with "gz" and you will get a warning about not having native libraries, but the test will succeed.) I believe JM's suggestion is for you to copy the Hadoop native libraries into the local HBase lib/native directory, which would allow the local test to pass. If you are running in a deployed Hadoop cluster, I would expect the necessary libraries to already be available to HBase. [1]: https://issues.apache.org/jira/browse/HBASE-6189 -Sean On Tue, Aug 26, 2014 at 8:30 AM, arthur.hk.c...@gmail.com < arthur.hk.c...@gmail.com> wrote: > Hi JM > > Below are my commands, tried two cases under same source code folder: > a) compile with snappy parameters(failed), > b) compile without snappy parameters (successful). > > Regards > Arthur > > wget http://mirrors.devlib.org/apache/hbase/stable/hbase-0.98.4-src.tar.gz > tar -vxf hbase-0.98.4-src.tar.gz > mv hbase-0.98.4 hbase-0.98.4-src_snappy > cd hbase-0.98.4-src_snappy > nano dev-support/generate-hadoopX-poms.sh > (change hbase_home=“/usr/local/hadoop/hbase-0.98.4-src_snappy”) > > > bash -x ./dev-support/generate-hadoopX-poms.sh 0.98.4 0.98.4-hadoop2 > a) with snappy parameters > mvn -f pom.xml.hadoop2 install -DskipTests assembly:single > -Prelease,hadoop-snappy -Dhadoop-snappy.version=0.0.1-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > [INFO] Building HBase - Server 0.98.4-hadoop2 > [INFO] > ------------------------------------------------------------------------ > [WARNING] The POM for org.apache.hadoop:hadoop-snappy:jar:0.0.1-SNAPSHOT > is missing, no dependency information available > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] HBase ............................................. SUCCESS [8.192s] > [INFO] HBase - Common .................................... SUCCESS [5.638s] > [INFO] HBase - Protocol .................................. SUCCESS [1.535s] > [INFO] HBase - Client .................................... SUCCESS [1.206s] > [INFO] HBase - Hadoop Compatibility ...................... SUCCESS [0.193s] > [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [0.798s] > [INFO] HBase - Prefix Tree ............................... SUCCESS [0.438s] > [INFO] HBase - Server .................................... FAILURE [0.234s] > [INFO] HBase - Testing Util .............................. SKIPPED > [INFO] HBase - Thrift .................................... SKIPPED > [INFO] HBase - Shell ..................................... SKIPPED > [INFO] HBase - Integration Tests ......................... SKIPPED > [INFO] HBase - Examples .................................. SKIPPED > [INFO] HBase - Assembly .................................. SKIPPED > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 19.474s > [INFO] Finished at: Tue Aug 26 21:21:13 HKT 2014 > [INFO] Final Memory: 51M/1100M > [INFO] > ------------------------------------------------------------------------ > [ERROR] Failed to execute goal on project hbase-server: Could not resolve > dependencies for project org.apache.hbase:hbase-server:jar:0.98.4-hadoop2: > Failure to find org.apache.hadoop:hadoop-snappy:jar:0.0.1-SNAPSHOT in > http://maven.oschina.net/content/groups/public/ was cached in the local > repository, resolution will not be reattempted until the update interval of > nexus-osc has elapsed or updates are forced -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn <goals> -rf :hbase-server > > > > > b) try again, without snappy parameters > mvn -f pom.xml.hadoop2 install -DskipTests assembly:single -Prelease > [INFO] Building tar: > /edh/hadoop_all_sources/hbase-0.98.4-src_snappy/hbase-assembly/target/hbase-0.98.4-hadoop2-bin.tar.gz > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] HBase ............................................. SUCCESS [3.290s] > [INFO] HBase - Common .................................... SUCCESS [3.119s] > [INFO] HBase - Protocol .................................. SUCCESS [0.972s] > [INFO] HBase - Client .................................... SUCCESS [0.920s] > [INFO] HBase - Hadoop Compatibility ...................... SUCCESS [0.167s] > [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [0.504s] > [INFO] HBase - Prefix Tree ............................... SUCCESS [0.382s] > [INFO] HBase - Server .................................... SUCCESS [4.790s] > [INFO] HBase - Testing Util .............................. SUCCESS [0.598s] > [INFO] HBase - Thrift .................................... SUCCESS [1.536s] > [INFO] HBase - Shell ..................................... SUCCESS [0.369s] > [INFO] HBase - Integration Tests ......................... SUCCESS [0.443s] > [INFO] HBase - Examples .................................. SUCCESS [0.459s] > [INFO] HBase - Assembly .................................. SUCCESS > [13.240s] > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 31.408s > [INFO] Finished at: Tue Aug 26 21:22:50 HKT 2014 > [INFO] Final Memory: 57M/1627M > [INFO] > ------------------------------------------------------------------------ > > > > > > On 26 Aug, 2014, at 8:52 pm, Jean-Marc Spaggiari <jean-m...@spaggiari.org> > wrote: > > > Hi Arthur, > > > > How have you extracted HBase source and what command do you run to > build? I > > will do the same here locally so I can provide you the exact step to > > complete. > > > > JM > > > > > > 2014-08-26 8:42 GMT-04:00 arthur.hk.c...@gmail.com < > arthur.hk.c...@gmail.com > >> : > > > >> Hi JM > >> > >> Not too sure what you mean, do you mean I should create a new folder in > my > >> HBASE_SRC named lib/native/Linux-x86 and copy these files to this folder > >> then try to compile it again? > >> > >> Regards > >> ARthur > >> > >> > >> On 26 Aug, 2014, at 8:17 pm, Jean-Marc Spaggiari < > jean-m...@spaggiari.org> > >> wrote: > >> > >>> Hi Arthur, > >>> > >>> Almost done! You now need to copy them on the HBase folder. > >>> > >>> hbase@hbasetest1:~/hbase-0.98.2-hadoop2/lib$ tree | grep -v .jar | > grep > >> -v > >>> .rb > >>> . > >>> ├── native > >>> │ └── Linux-x86 > >>> │ ├── libsnappy.a > >>> │ ├── libsnappy.la > >>> │ ├── libsnappy.so > >>> │ ├── libsnappy.so.1 > >>> │ └── libsnappy.so.1.2.0 > >>> > >>> I don't have any hadoop-snappy lib in my hbase folder and it works very > >>> well with Snappy for me... > >>> > >>> JM > >>> > >>> 2014-08-26 8:09 GMT-04:00 arthur.hk.c...@gmail.com < > >> arthur.hk.c...@gmail.com > >>>> : > >>> > >>>> Hi JM, > >>>> > >>>> Below are my steps to install snappy lib, do I miss something? > >>>> > >>>> Regards > >>>> Arthur > >>>> > >>>> wget https://snappy.googlecode.com/files/snappy-1.1.1.tar.gz > >>>> tar -vxf snappy-1.1.1.tar.gz > >>>> cd snappy-1.1.1 > >>>> ./configure > >>>> make > >>>> make install > >>>> make[1]: Entering directory > >> `/edh/hadoop_all_sources/snappy-1.1.1' > >>>> test -z "/usr/local/lib" || /bin/mkdir -p "/usr/local/lib" > >>>> /bin/sh ./libtool --mode=install /usr/bin/install -c > >>>> libsnappy.la '/usr/local/lib' > >>>> libtool: install: /usr/bin/install -c .libs/libsnappy.so.1.2.0 > >>>> /usr/local/lib/libsnappy.so.1.2.0 > >>>> libtool: install: (cd /usr/local/lib && { ln -s -f > >>>> libsnappy.so.1.2.0 libsnappy.so.1 || { rm -f libsnappy.so.1 && ln -s > >>>> libsnappy.so.1.2.0 libsnappy.so.1; }; }) > >>>> libtool: install: (cd /usr/local/lib && { ln -s -f > >>>> libsnappy.so.1.2.0 libsnappy.so || { rm -f libsnappy.so && ln -s > >>>> libsnappy.so.1.2.0 libsnappy.so; }; }) > >>>> libtool: install: /usr/bin/install -c .libs/libsnappy.lai > >>>> /usr/local/lib/libsnappy.la > >>>> libtool: install: /usr/bin/install -c .libs/libsnappy.a > >>>> /usr/local/lib/libsnappy.a > >>>> libtool: install: chmod 644 /usr/local/lib/libsnappy.a > >>>> libtool: install: ranlib /usr/local/lib/libsnappy.a > >>>> libtool: finish: > >>>> > >> > PATH="/edh/hadoop/spark/bin:/edh/hadoop/hbase/bin:/edh/hadoop/zookeeper//bin:/edh/hadoop/yarn/hadoop/bin:/edh/hadoop/yarn/hadoop/sbin:/usr/lib64/qt-3.3/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/edh/hadoop/zookeeper//bin:/edh/hadoop/hive//bin:/usr/lib/jvm/jdk1.6.0_45//bin:/root/bin:/sbin" > >>>> ldconfig -n /usr/local/lib > >>>> > >>>> ---------------------------------------------------------------------- > >>>> Libraries have been installed in: > >>>> /usr/local/lib > >>>> If you ever happen to want to link against installed libraries > >>>> in a given directory, LIBDIR, you must either use libtool, and > >>>> specify the full pathname of the library, or use the `-LLIBDIR' > >>>> flag during linking and do at least one of the following: > >>>> - add LIBDIR to the `LD_LIBRARY_PATH' environment variable > >>>> during execution > >>>> - add LIBDIR to the `LD_RUN_PATH' environment variable > >>>> during linking > >>>> - use the `-Wl,-rpath -Wl,LIBDIR' linker flag > >>>> - have your system administrator add LIBDIR to `/etc/ld.so.conf' > >>>> See any operating system documentation about shared libraries > for > >>>> more information, such as the ld(1) and ld.so(8) manual pages. > >>>> > >>>> ---------------------------------------------------------------------- > >>>> test -z "/usr/local/share/doc/snappy" || /bin/mkdir -p > >>>> "/usr/local/share/doc/snappy" > >>>> /usr/bin/install -c -m 644 ChangeLog COPYING INSTALL NEWS > README > >>>> format_description.txt framing_format.txt > '/usr/local/share/doc/snappy' > >>>> test -z "/usr/local/include" || /bin/mkdir -p > >> "/usr/local/include" > >>>> /usr/bin/install -c -m 644 snappy.h snappy-sinksource.h > >>>> snappy-stubs-public.h snappy-c.h '/usr/local/include' > >>>> make[1]: Leaving directory > `/edh/hadoop_all_sources/snappy-1.1.1' > >>>> > >>>> ll /usr/local/lib > >>>> -rw-r--r--. 1 root root 233554 Aug 20 00:14 libsnappy.a > >>>> -rwxr-xr-x. 1 root root 953 Aug 20 00:14 libsnappy.la > >>>> lrwxrwxrwx. 1 root root 18 Aug 20 00:14 libsnappy.so -> > >>>> libsnappy.so.1.2.0 > >>>> lrwxrwxrwx. 1 root root 18 Aug 20 00:14 libsnappy.so.1 -> > >>>> libsnappy.so.1.2.0 > >>>> -rwxr-xr-x. 1 root root 147726 Aug 20 00:14 libsnappy.so.1.2.0 > >>>> > >>>> > >>>> > >>>> On 26 Aug, 2014, at 7:38 pm, Jean-Marc Spaggiari < > >> jean-m...@spaggiari.org> > >>>> wrote: > >>>> > >>>>> Hi Arthur, > >>>>> > >>>>> Do you have snappy libs installed and configured? HBase doesn't come > >> with > >>>>> Snappy. So yo need to have it first. > >>>>> > >>>>> Shameless plug: > >>>>> > >>>> > >> > http://www.spaggiari.org/index.php/hbase/how-to-install-snappy-with-1#.U_xxSqdZuZY > >>>>> > >>>>> This is for 0.96 but should be very similar for 0.98. I will try it > >> soon > >>>>> and post and update, but keep us posted here so we can support you... > >>>>> > >>>>> JM > >>>>> > >>>>> > >>>>> 2014-08-26 7:34 GMT-04:00 arthur.hk.c...@gmail.com < > >>>> arthur.hk.c...@gmail.com > >>>>>> : > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> I need to install snappy to HBase 0.98.4. (my Hadoop version is > >> 2.4.1) > >>>>>> > >>>>>> Can you please advise what would be wrong? Should my pom.xml be > >>>> incorrect > >>>>>> and missing something? > >>>>>> > >>>>>> Regards > >>>>>> Arthur > >>>>>> > >>>>>> > >>>>>> Below are my commands: > >>>>>> bash -x ./dev-support/generate-hadoopX-poms.sh 0.98.4 0.98.4-hadoop2 > >>>>>> mvn -f pom.xml.hadoop2 install -DskipTests assembly:single > >>>>>> -Prelease,hadoop-snappy > >>>>>> > >>>>>> Iog: > >>>>>> [INFO] > >>>>>> > >> ------------------------------------------------------------------------ > >>>>>> [INFO] Building HBase - Server 0.98.4-hadoop2 > >>>>>> [INFO] > >>>>>> > >> ------------------------------------------------------------------------ > >>>>>> [WARNING] The POM for > >> org.apache.hadoop:hadoop-snappy:jar:0.0.1-SNAPSHOT > >>>>>> is missing, no dependency information available > >>>>>> [INFO] > >>>>>> > >> ------------------------------------------------------------------------ > >>>>>> [INFO] Reactor Summary: > >>>>>> [INFO] > >>>>>> [INFO] HBase ............................................. SUCCESS > >>>> [3.129s] > >>>>>> [INFO] HBase - Common .................................... SUCCESS > >>>> [3.105s] > >>>>>> [INFO] HBase - Protocol .................................. SUCCESS > >>>> [0.976s] > >>>>>> [INFO] HBase - Client .................................... SUCCESS > >>>> [0.925s] > >>>>>> [INFO] HBase - Hadoop Compatibility ...................... SUCCESS > >>>> [0.183s] > >>>>>> [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS > >>>> [0.497s] > >>>>>> [INFO] HBase - Prefix Tree ............................... SUCCESS > >>>> [0.407s] > >>>>>> [INFO] HBase - Server .................................... FAILURE > >>>> [0.103s] > >>>>>> [INFO] HBase - Testing Util .............................. SKIPPED > >>>>>> [INFO] HBase - Thrift .................................... SKIPPED > >>>>>> [INFO] HBase - Shell ..................................... SKIPPED > >>>>>> [INFO] HBase - Integration Tests ......................... SKIPPED > >>>>>> [INFO] HBase - Examples .................................. SKIPPED > >>>>>> [INFO] HBase - Assembly .................................. SKIPPED > >>>>>> [INFO] > >>>>>> > >> ------------------------------------------------------------------------ > >>>>>> [INFO] BUILD FAILURE > >>>>>> [INFO] > >>>>>> > >> ------------------------------------------------------------------------ > >>>>>> [INFO] Total time: 9.939s > >>>>>> [INFO] Finished at: Tue Aug 26 19:23:14 HKT 2014 > >>>>>> [INFO] Final Memory: 61M/2921M > >>>>>> [INFO] > >>>>>> > >> ------------------------------------------------------------------------ > >>>>>> [ERROR] Failed to execute goal on project hbase-server: Could not > >>>> resolve > >>>>>> dependencies for project > >>>> org.apache.hbase:hbase-server:jar:0.98.4-hadoop2: > >>>>>> Failure to find org.apache.hadoop:hadoop-snappy:jar:0.0.1-SNAPSHOT > in > >>>>>> http://maven.oschina.net/content/groups/public/ was cached in the > >> local > >>>>>> repository, resolution will not be reattempted until the update > >>>> interval of > >>>>>> nexus-osc has elapsed or updates are forced -> [Help 1] > >>>>>> [ERROR] > >>>>>> [ERROR] To see the full stack trace of the errors, re-run Maven with > >> the > >>>>>> -e switch. > >>>>>> [ERROR] Re-run Maven using the -X switch to enable full debug > logging. > >>>>>> [ERROR] > >>>>>> [ERROR] For more information about the errors and possible > solutions, > >>>>>> please read the following articles: > >>>>>> [ERROR] [Help 1] > >>>>>> > >>>> > >> > http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException > >>>>>> [ERROR] > >>>>>> [ERROR] After correcting the problems, you can resume the build with > >> the > >>>>>> command > >>>>>> [ERROR] mvn <goals> -rf :hbase-server > >>>>>> > >>>>>> > >>>> > >>>> > >> > >> > > -- Sean
