Hi Eric, There isn't a way to completely skip compiling libhdfs++ as part of the native build. You could pass -Dnative_cmake_args="-DHDFSPP_LIBRARY_ONLY=TRUE" to maven to avoid building all of the libhdfs++ tests, examples, and tools though. That cut the native client build time from 4:10 to 2:20 for me.
-Jim On Tue, Mar 27, 2018 at 12:25 PM, Eric Badger <ebad...@oath.com> wrote: > Is there a way to skip the libhdfs++ compilation during a native build? I > just went to build native trunk to test out some container-executor changes > and it spent 7:49 minutes out of 14:31 minutes in Apache Hadoop HDFS Native > Client. For me it basically doubled the compilation time. > > Eric > > On Fri, Mar 16, 2018 at 4:01 PM, Deepak Majeti <majeti.dee...@gmail.com> > wrote: > >> Thanks for all your hard work on getting this feature (with > 200 >> sub-tasks) in James! >> >> On Fri, Mar 16, 2018 at 12:05 PM, Jim Clampffer < >> james.clampf...@gmail.com> >> wrote: >> >> > With 6 +1s, 0 0s, and 0 -1s the vote passes. I'll be merging this into >> > trunk shortly. >> > >> > Thanks everyone who participated in the discussion and vote! And many >> > thanks to everyone who contributed code and feedback throughout the >> > development process! Particularly Bob, Anatoli, Xiaowei and Deepak who >> > provided lots of large pieces of code as well as folks like Owen, Chris >> D, >> > Allen, and Stephen W who provided various support and guidance with the >> > Apache process and project design. >> > >> > On Wed, Mar 14, 2018 at 1:32 PM, Jitendra Pandey < >> jiten...@hortonworks.com >> > > >> > wrote: >> > >> > > +1 (binding) >> > > >> > > On 3/14/18, 9:57 AM, "Anu Engineer" <aengin...@hortonworks.com> >> wrote: >> > > >> > > +1 (binding). Thanks for all the hard work and getting this client >> > > ready. >> > > It is nice to have an official and supported native client for >> HDFS. >> > > >> > > Thanks >> > > Anu >> > > >> > > On 3/13/18, 8:16 PM, "Mukul Kumar Singh" <msi...@hortonworks.com> >> > > wrote: >> > > >> > > +1 (binding) >> > > >> > > Thanks, >> > > Mukul >> > > >> > > On 14/03/18, 2:06 AM, "Owen O'Malley" <owen.omal...@gmail.com >> > >> > > wrote: >> > > >> > > +1 (binding) >> > > >> > > .. Owen >> > > >> > > On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas < >> > > cdoug...@apache.org> wrote: >> > > >> > > > +1 (binding) -C >> > > > >> > > > On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer < >> > > james.clampf...@gmail.com> >> > > > wrote: >> > > > > Hi Everyone, >> > > > > >> > > > > The feedback was generally positive on the discussion >> > > thread [1] so I'd >> > > > > like to start a formal vote for merging HDFS-8707 >> > > (libhdfs++) into trunk. >> > > > > The vote will be open for 7 days and end 6PM EST on >> > > 3/15/18. >> > > > > >> > > > > This branch includes a C++ implementation of an HDFS >> > > client for use in >> > > > > applications that don't run an in-process JVM. Right >> now >> > > the branch only >> > > > > supports reads and metadata calls. >> > > > > >> > > > > Features (paraphrasing the list from the discussion >> > > thread): >> > > > > -Avoiding the JVM means applications that use >> libhdfs++ >> > > can explicitly >> > > > > control resources (memory, FDs, threads). The driving >> > > goal for this >> > > > > project was to let C/C++ applications access HDFS >> while >> > > maintaining a >> > > > > single heap. >> > > > > -Includes support for Kerberos authentication. >> > > > > -Includes a libhdfs/libhdfs3 compatible C API as well >> as >> > a >> > > C++ API that >> > > > > supports asynchronous operations. Applications that >> only >> > > do reads may be >> > > > > able to use this as a drop in replacement for libhdfs. >> > > > > -Asynchronous IO is built on top of boost::asio which >> in >> > > turn uses >> > > > > select/epoll so many sockets can be monitored from a >> > > single thread (or >> > > > > thread pool) rather than spawning a thread to sleep >> on a >> > > blocked socket. >> > > > > -Includes a set of utilities written in C++ that >> mirror >> > > the CLI tools >> > > > (e.g. >> > > > > ./hdfs dfs -ls). These have a 3 order of magnitude >> lower >> > > startup time >> > > > than >> > > > > java client which is useful for scripts that need to >> work >> > > with many >> > > > files. >> > > > > -Support for cancelable reads that release associated >> > > resources >> > > > > immediately. Useful for applications that need to be >> > > responsive to >> > > > > interactive users. >> > > > > >> > > > > Other points: >> > > > > -This is almost all new code in a new subdirectory. >> No >> > > Java source for >> > > > the >> > > > > rest of hadoop was changed so there's no risk of >> > > regressions there. The >> > > > > only changes outside of that subdirectory were >> > integrating >> > > the build in >> > > > > some of the pom files and adding a couple >> dependencies to >> > > the DockerFile. >> > > > > -The library has had plenty of burn-in time. It's >> been >> > > used in >> > > > production >> > > > > for well over a year and is indirectly being >> distributed >> > > as part of the >> > > > > Apache ORC project (in the form of a third party >> > > dependency). >> > > > > -There isn't much in the way of well formatted >> > > documentation right now. >> > > > > The documentation for the libhdfs API is applicable to >> > the >> > > libhdfs++ C >> > > > API. >> > > > > Header files describe various component including >> details >> > > about threading >> > > > > and lifecycle expectations for important objects. >> Good >> > > places to start >> > > > are >> > > > > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h >> > and >> > > rpc_enginel.h. >> > > > > >> > > > > I'll start with my +1 (binding). >> > > > > >> > > > > [1] >> > > > > http://mail-archives.apache. >> > org/mod_mbox/hadoop-hdfs-dev/ >> > > > 201803.mbox/browser >> > > > > (second message in thread, can't figure out how to >> link >> > > directly to mine) >> > > > > >> > > > > Thanks! >> > > > >> > > > ------------------------------ >> > ------------------------------ >> > > --------- >> > > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop. >> > > apache.org >> > > > For additional commands, e-mail: >> > > hdfs-dev-h...@hadoop.apache.org >> > > > >> > > > >> > > >> > > >> > > >> > > ------------------------------------------------------------ >> > > --------- >> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.ap >> ache.org >> > > For additional commands, e-mail: >> hdfs-dev-h...@hadoop.apache.org >> > > >> > > >> > > >> > > >> > > ------------------------------------------------------------ >> > --------- >> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> > > >> > > >> > > >> > > >> > >> >> >> >> -- >> regards, >> Deepak Majeti >> > >
