Would it be possible to add a maven flag like –skipShade, that helps in reducing the compile time for people who does not need to build libhdfs++ ?
Thanks Anu From: Jim Clampffer <james.clampf...@gmail.com> Date: Tuesday, March 27, 2018 at 11:09 AM To: Eric Badger <ebad...@oath.com> Cc: Deepak Majeti <majeti.dee...@gmail.com>, Jitendra Pandey <jiten...@hortonworks.com>, Anu Engineer <aengin...@hortonworks.com>, Mukul Kumar Singh <msi...@hortonworks.com>, Owen O'Malley <owen.omal...@gmail.com>, Chris Douglas <cdoug...@apache.org>, Hdfs-dev <hdfs-dev@hadoop.apache.org> Subject: Re: [VOTE] Merging branch HDFS-8707 (native HDFS client) to trunk Hi Eric, There isn't a way to completely skip compiling libhdfs++ as part of the native build. You could pass -Dnative_cmake_args="-DHDFSPP_LIBRARY_ONLY=TRUE" to maven to avoid building all of the libhdfs++ tests, examples, and tools though. That cut the native client build time from 4:10 to 2:20 for me. -Jim On Tue, Mar 27, 2018 at 12:25 PM, Eric Badger <ebad...@oath.com<mailto:ebad...@oath.com>> wrote: Is there a way to skip the libhdfs++ compilation during a native build? I just went to build native trunk to test out some container-executor changes and it spent 7:49 minutes out of 14:31 minutes in Apache Hadoop HDFS Native Client. For me it basically doubled the compilation time. Eric On Fri, Mar 16, 2018 at 4:01 PM, Deepak Majeti <majeti.dee...@gmail.com<mailto:majeti.dee...@gmail.com>> wrote: Thanks for all your hard work on getting this feature (with > 200 sub-tasks) in James! On Fri, Mar 16, 2018 at 12:05 PM, Jim Clampffer <james.clampf...@gmail.com<mailto:james.clampf...@gmail.com>> wrote: > With 6 +1s, 0 0s, and 0 -1s the vote passes. I'll be merging this into > trunk shortly. > > Thanks everyone who participated in the discussion and vote! And many > thanks to everyone who contributed code and feedback throughout the > development process! Particularly Bob, Anatoli, Xiaowei and Deepak who > provided lots of large pieces of code as well as folks like Owen, Chris D, > Allen, and Stephen W who provided various support and guidance with the > Apache process and project design. > > On Wed, Mar 14, 2018 at 1:32 PM, Jitendra Pandey > <jiten...@hortonworks.com<mailto:jiten...@hortonworks.com> > > > wrote: > > > +1 (binding) > > > > On 3/14/18, 9:57 AM, "Anu Engineer" > > <aengin...@hortonworks.com<mailto:aengin...@hortonworks.com>> wrote: > > > > +1 (binding). Thanks for all the hard work and getting this client > > ready. > > It is nice to have an official and supported native client for HDFS. > > > > Thanks > > Anu > > > > On 3/13/18, 8:16 PM, "Mukul Kumar Singh" > > <msi...@hortonworks.com<mailto:msi...@hortonworks.com>> > > wrote: > > > > +1 (binding) > > > > Thanks, > > Mukul > > > > On 14/03/18, 2:06 AM, "Owen O'Malley" > > <owen.omal...@gmail.com<mailto:owen.omal...@gmail.com>> > > wrote: > > > > +1 (binding) > > > > .. Owen > > > > On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas < > > cdoug...@apache.org<mailto:cdoug...@apache.org>> wrote: > > > > > +1 (binding) -C > > > > > > On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer < > > james.clampf...@gmail.com<mailto:james.clampf...@gmail.com>> > > > wrote: > > > > Hi Everyone, > > > > > > > > The feedback was generally positive on the discussion > > thread [1] so I'd > > > > like to start a formal vote for merging HDFS-8707 > > (libhdfs++) into trunk. > > > > The vote will be open for 7 days and end 6PM EST on > > 3/15/18. > > > > > > > > This branch includes a C++ implementation of an HDFS > > client for use in > > > > applications that don't run an in-process JVM. Right now > > the branch only > > > > supports reads and metadata calls. > > > > > > > > Features (paraphrasing the list from the discussion > > thread): > > > > -Avoiding the JVM means applications that use libhdfs++ > > can explicitly > > > > control resources (memory, FDs, threads). The driving > > goal for this > > > > project was to let C/C++ applications access HDFS while > > maintaining a > > > > single heap. > > > > -Includes support for Kerberos authentication. > > > > -Includes a libhdfs/libhdfs3 compatible C API as well as > a > > C++ API that > > > > supports asynchronous operations. Applications that only > > do reads may be > > > > able to use this as a drop in replacement for libhdfs. > > > > -Asynchronous IO is built on top of boost::asio which in > > turn uses > > > > select/epoll so many sockets can be monitored from a > > single thread (or > > > > thread pool) rather than spawning a thread to sleep on a > > blocked socket. > > > > -Includes a set of utilities written in C++ that mirror > > the CLI tools > > > (e.g. > > > > ./hdfs dfs -ls). These have a 3 order of magnitude lower > > startup time > > > than > > > > java client which is useful for scripts that need to work > > with many > > > files. > > > > -Support for cancelable reads that release associated > > resources > > > > immediately. Useful for applications that need to be > > responsive to > > > > interactive users. > > > > > > > > Other points: > > > > -This is almost all new code in a new subdirectory. No > > Java source for > > > the > > > > rest of hadoop was changed so there's no risk of > > regressions there. The > > > > only changes outside of that subdirectory were > integrating > > the build in > > > > some of the pom files and adding a couple dependencies to > > the DockerFile. > > > > -The library has had plenty of burn-in time. It's been > > used in > > > production > > > > for well over a year and is indirectly being distributed > > as part of the > > > > Apache ORC project (in the form of a third party > > dependency). > > > > -There isn't much in the way of well formatted > > documentation right now. > > > > The documentation for the libhdfs API is applicable to > the > > libhdfs++ C > > > API. > > > > Header files describe various component including details > > about threading > > > > and lifecycle expectations for important objects. Good > > places to start > > > are > > > > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h > and > > rpc_enginel.h. > > > > > > > > I'll start with my +1 (binding). > > > > > > > > [1] > > > > http://mail-archives.apache. > org/mod_mbox/hadoop-hdfs-dev/ > > > 201803.mbox/browser > > > > (second message in thread, can't figure out how to link > > directly to mine) > > > > > > > > Thanks! > > > > > > ------------------------------ > ------------------------------ > > --------- > > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop. > > apache.org<http://apache.org> > > > For additional commands, e-mail: > > hdfs-dev-h...@hadoop.apache.org<mailto:hdfs-dev-h...@hadoop.apache.org> > > > > > > > > > > > > > > ------------------------------------------------------------ > > --------- > > To unsubscribe, e-mail: > > hdfs-dev-unsubscr...@hadoop.apache.org<mailto:hdfs-dev-unsubscr...@hadoop.apache.org> > > For additional commands, e-mail: > > hdfs-dev-h...@hadoop.apache.org<mailto:hdfs-dev-h...@hadoop.apache.org> > > > > > > > > > > ------------------------------------------------------------ > --------- > > To unsubscribe, e-mail: > > hdfs-dev-unsubscr...@hadoop.apache.org<mailto:hdfs-dev-unsubscr...@hadoop.apache.org> > > For additional commands, e-mail: > > hdfs-dev-h...@hadoop.apache.org<mailto:hdfs-dev-h...@hadoop.apache.org> > > > > > > > > > -- regards, Deepak Majeti
