Is there a way to skip the libhdfs++ compilation during a native build? I just went to build native trunk to test out some container-executor changes and it spent 7:49 minutes out of 14:31 minutes in Apache Hadoop HDFS Native Client. For me it basically doubled the compilation time.
Eric On Fri, Mar 16, 2018 at 4:01 PM, Deepak Majeti <majeti.dee...@gmail.com> wrote: > Thanks for all your hard work on getting this feature (with > 200 > sub-tasks) in James! > > On Fri, Mar 16, 2018 at 12:05 PM, Jim Clampffer <james.clampf...@gmail.com > > > wrote: > > > With 6 +1s, 0 0s, and 0 -1s the vote passes. I'll be merging this into > > trunk shortly. > > > > Thanks everyone who participated in the discussion and vote! And many > > thanks to everyone who contributed code and feedback throughout the > > development process! Particularly Bob, Anatoli, Xiaowei and Deepak who > > provided lots of large pieces of code as well as folks like Owen, Chris > D, > > Allen, and Stephen W who provided various support and guidance with the > > Apache process and project design. > > > > On Wed, Mar 14, 2018 at 1:32 PM, Jitendra Pandey < > jiten...@hortonworks.com > > > > > wrote: > > > > > +1 (binding) > > > > > > On 3/14/18, 9:57 AM, "Anu Engineer" <aengin...@hortonworks.com> wrote: > > > > > > +1 (binding). Thanks for all the hard work and getting this client > > > ready. > > > It is nice to have an official and supported native client for > HDFS. > > > > > > Thanks > > > Anu > > > > > > On 3/13/18, 8:16 PM, "Mukul Kumar Singh" <msi...@hortonworks.com> > > > wrote: > > > > > > +1 (binding) > > > > > > Thanks, > > > Mukul > > > > > > On 14/03/18, 2:06 AM, "Owen O'Malley" <owen.omal...@gmail.com> > > > wrote: > > > > > > +1 (binding) > > > > > > .. Owen > > > > > > On Sun, Mar 11, 2018 at 6:20 PM, Chris Douglas < > > > cdoug...@apache.org> wrote: > > > > > > > +1 (binding) -C > > > > > > > > On Thu, Mar 8, 2018 at 9:31 AM, Jim Clampffer < > > > james.clampf...@gmail.com> > > > > wrote: > > > > > Hi Everyone, > > > > > > > > > > The feedback was generally positive on the discussion > > > thread [1] so I'd > > > > > like to start a formal vote for merging HDFS-8707 > > > (libhdfs++) into trunk. > > > > > The vote will be open for 7 days and end 6PM EST on > > > 3/15/18. > > > > > > > > > > This branch includes a C++ implementation of an HDFS > > > client for use in > > > > > applications that don't run an in-process JVM. Right > now > > > the branch only > > > > > supports reads and metadata calls. > > > > > > > > > > Features (paraphrasing the list from the discussion > > > thread): > > > > > -Avoiding the JVM means applications that use libhdfs++ > > > can explicitly > > > > > control resources (memory, FDs, threads). The driving > > > goal for this > > > > > project was to let C/C++ applications access HDFS while > > > maintaining a > > > > > single heap. > > > > > -Includes support for Kerberos authentication. > > > > > -Includes a libhdfs/libhdfs3 compatible C API as well > as > > a > > > C++ API that > > > > > supports asynchronous operations. Applications that > only > > > do reads may be > > > > > able to use this as a drop in replacement for libhdfs. > > > > > -Asynchronous IO is built on top of boost::asio which > in > > > turn uses > > > > > select/epoll so many sockets can be monitored from a > > > single thread (or > > > > > thread pool) rather than spawning a thread to sleep on > a > > > blocked socket. > > > > > -Includes a set of utilities written in C++ that mirror > > > the CLI tools > > > > (e.g. > > > > > ./hdfs dfs -ls). These have a 3 order of magnitude > lower > > > startup time > > > > than > > > > > java client which is useful for scripts that need to > work > > > with many > > > > files. > > > > > -Support for cancelable reads that release associated > > > resources > > > > > immediately. Useful for applications that need to be > > > responsive to > > > > > interactive users. > > > > > > > > > > Other points: > > > > > -This is almost all new code in a new subdirectory. No > > > Java source for > > > > the > > > > > rest of hadoop was changed so there's no risk of > > > regressions there. The > > > > > only changes outside of that subdirectory were > > integrating > > > the build in > > > > > some of the pom files and adding a couple dependencies > to > > > the DockerFile. > > > > > -The library has had plenty of burn-in time. It's been > > > used in > > > > production > > > > > for well over a year and is indirectly being > distributed > > > as part of the > > > > > Apache ORC project (in the form of a third party > > > dependency). > > > > > -There isn't much in the way of well formatted > > > documentation right now. > > > > > The documentation for the libhdfs API is applicable to > > the > > > libhdfs++ C > > > > API. > > > > > Header files describe various component including > details > > > about threading > > > > > and lifecycle expectations for important objects. Good > > > places to start > > > > are > > > > > hdfspp.h, filesystem.h, filehandle.h, rpc_connection.h > > and > > > rpc_enginel.h. > > > > > > > > > > I'll start with my +1 (binding). > > > > > > > > > > [1] > > > > > http://mail-archives.apache. > > org/mod_mbox/hadoop-hdfs-dev/ > > > > 201803.mbox/browser > > > > > (second message in thread, can't figure out how to link > > > directly to mine) > > > > > > > > > > Thanks! > > > > > > > > ------------------------------ > > ------------------------------ > > > --------- > > > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop. > > > apache.org > > > > For additional commands, e-mail: > > > hdfs-dev-h...@hadoop.apache.org > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------ > > > --------- > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache. > org > > > > > > > > > > > > > > > ------------------------------------------------------------ > > --------- > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > > > > > > > > > > > -- > regards, > Deepak Majeti >
