Handling Kerberos is similar to what we have done for WebHDFS now. Kerby will be in the picture but things are much simpler.
If protobuf is a concern, why not shading it into hadoop-common? The generated binaries might not be compatible but the wire format is. On Mon, Feb 29, 2016 at 1:55 AM Steve Loughran <ste...@hortonworks.com> wrote: > > > On 27 Feb 2016, at 19:02, Haohui Mai <ricet...@gmail.com> wrote: > > > > Have we evaluated GRPC? A robust RPC requires significant effort. > Migrating > > to GRPC can save ourselves a lot of headache. > > > > That's the google protobuf 3 based GRPC? More specifically, > protobufVersion = '3.0.0-beta-2'? > > That's successor to the protobuf.jar whose Alejandro-choreographed > cross-project upgrade caused the "great protobuf upgrade of 2013"? That's > the protobuf library where some of us have seriously considered forking the > library so that we could have a version of protobuf which would link across > java classes generated with older versions? > > We have enough problems working with a released version of protobuf > breaking across minor point releases, whose guava JARs are a recurrent > source of cross version compatibility pain? > > > I would rather stab myself in the leg with a fork —repeatedly— than adopt > something based on a beta-release of a google artifact as critical path of > the Hadoop RPC chain. > > While google are pretty obsessive about wire format compatibility across > languages and versions, we just can't trust google to maintain binary > compatibility, primarily due to a build process which clean builds > everything from scratch. They don't have the same problem of trying to > nudge things up across a loosely coupled set of projects, including those > who still have requirements of JAR-sharing compatibility with older hadoop > versions. Indeed, for those projects, being backwards compatible with > Hadoop 1.x (no protobuf) is easier than working with Hadoop 2.205, purely > due to to that protobuf difference. > > > Even when protobuf 3.0 finally ships, we should hold back even adopting > it for its current role until 3.1 comes out so we can asses google's > compatibility policy in the 3.x line. > > > > Haohui > > On Sat, Feb 27, 2016 at 1:35 AM Andrew Purtell <andrew.purt...@gmail.com > > > > wrote: > > > >> I get a excited thinking about the prospect of better performance with > >> auth-conf QoP. HBase RPC is an increasingly distant fork but still close > >> enough to Hadoop in that respect. Our bulk data transfer protocol isn't > a > >> separate thing like in HDFS, which avoids a SASL wrapped > implementation, so > >> we really suffer when auth-conf is negotiated. You'll see the same > impact > >> where there might be a high frequency of NameNode RPC calls or similar > >> still. Throughput drops 3-4x, or worse. > >> > >>> On Feb 22, 2016, at 4:56 PM, Zheng, Kai <kai.zh...@intel.com> wrote: > >>> > >>> Thanks for the confirm and further inputs, Steve. > >>> > >>>>> the latter would dramatically reduce the cost of wire-encrypting IPC. > >>> Yes to optimize Hadoop IPC/RPC encryption is another opportunity Kerby > >> can help with, it's possible because we may hook Chimera or AES-NI thing > >> into the Kerberos layer by leveraging the Kerberos library. As it may be > >> noted, HADOOP-12725 is on the going for this aspect. There may be good > >> result and further update on this recently. > >>> > >>>>> For now, I'd like to see basic steps -upgrading minkdc to krypto, see > >> how it works. > >>> Yes, starting with this initial steps upgrading MiniKDC to use Kerby is > >> the right thing we could do. After some interactions with Kerby > project, we > >> may have more ideas how to proceed on the followings. > >>> > >>>>> Long term, I'd like Hadoop 3 to be Kerby-ized > >>> This sounds great! With necessary support from the community like > >> feedback and patch reviewing, we can speed up the related work. > >>> > >>> Regards, > >>> Kai > >>> > >>> -----Original Message----- > >>> From: Steve Loughran [mailto:ste...@hortonworks.com] > >>> Sent: Monday, February 22, 2016 6:51 PM > >>> To: common-dev@hadoop.apache.org > >>> Subject: Re: Introduce Apache Kerby to Hadoop > >>> > >>> > >>> > >>> I've discussed this offline with Kai, as part of the "let's fix > >> kerberos" project. Not only is it a better Kerberos engine, we can do > more > >> diagnostics, get better algorithms and ultimately get better APIs for > doing > >> Kerberos and SASL —the latter would dramatically reduce the cost of > >> wire-encrypting IPC. > >>> > >>> For now, I'd like to see basic steps -upgrading minkdc to krypto, see > >> how it works. > >>> > >>> Long term, I'd like Hadoop 3 to be Kerby-ized > >>> > >>> > >>>> On 22 Feb 2016, at 06:41, Zheng, Kai <kai.zh...@intel.com> wrote: > >>>> > >>>> Hi folks, > >>>> > >>>> I'd like to mention Apache Kerby [1] here to the community and propose > >> to introduce the project to Hadoop, a sub project of Apache Directory > >> project. > >>>> > >>>> Apache Kerby is a Kerberos centric project and aims to provide a first > >> Java Kerberos library that contains both client and server supports. The > >> relevant features include: > >>>> It supports full Kerberos encryption types aligned with both MIT KDC > >>>> and MS AD; Client APIs to allow to login via password, credential > >>>> cache, keytab file and etc.; Utilities for generate, operate and > >>>> inspect keytab and credential cache files; A simple KDC server that > >>>> borrows some ideas from Hadoop-MiniKDC and can be used in tests but > >>>> with minimal overhead in external dependencies; A brand new token > >> mechanism is provided, can be experimentally used, using it a JWT token > can > >> be used to exchange a TGT or service ticket; Anonymous PKINIT support, > can > >> be experientially used, as the first Java library that supports the > >> Kerberos major extension. > >>>> > >>>> The project stands alone and is ensured to only depend on JRE for > >> easier usage. It has made the first release (1.0.0-RC1) and 2nd release > >> (RC2) is upcoming. > >>>> > >>>> > >>>> As an initial step, this proposal suggests using Apache Kerby to > >> upgrade the existing codes related to ApacheDS for the Kerberos support. > >> The advantageous: > >>>> > >>>> 1. The kerby-kerb library is all the need, which is purely in Java, > >>>> SLF4J is the only dependency, the whole is rather small; > >>>> > >>>> 2. There is a SimpleKDC in the library for test usage, which borrowed > >>>> the MiniKDC idea and implemented all the support existing in MiniKDC. > >>>> We had a POC that rewrote MiniKDC using Kerby SimpleKDC and it works > >>>> fine; > >>>> > >>>> 3. Full Kerberos encryption types (many of them are not available in > >>>> JRE but supported by major Kerberos vendors) and more functionalities > >>>> like credential cache support; > >>>> > >>>> 4. Perhaps the most concerned, Hadoop MiniKDC and etc. depend on the > >>>> old Kerberos implementation in Directory Server project, but the > >>>> implementation is stopped being maintained. Directory project has a > >>>> plan to replace the implementation using Kerby. MiniKDC can use Kerby > >>>> directly to simplify the deps; > >>>> > >>>> 5. Extensively tested with all kinds of unit tests, already being used > >>>> for some time (like PSU), even in production environment; > >>>> > >>>> 6. Actively developed, and can be fixed and released in time if > >> necessary, separately and independently from other components in Apache > >> Directory project. By actively developing Apache Kerby and now applying > it > >> to Hadoop, our side wish to make the Kerberos deploying, troubleshooting > >> and further enhancement can be much easier and thereafter possible. > >>>> > >>>> > >>>> > >>>> Wish this is a good beginning, and eventually Apache Kerby can benefit > >> other projects in the ecosystem as well. > >>>> > >>>> > >>>> > >>>> This Kerberos related work is actually a long time effort led by > Weihua > >> Jiang in Intel, and had been kindly encouraged by Andrew Purtell, Steve > >> Loughran, Gangumalla Uma, Andrew Wang and etc., thanks a lot for their > >> great discussions and inputs in the past. > >>>> > >>>> > >>>> > >>>> Your feedback is very welcome. Thanks in advance. > >>>> > >>>> > >>>> > >>>> [1] https://github.com/apache/directory-kerby > >>>> > >>>> > >>>> > >>>> Regards, > >>>> > >>>> Kai > >>> > >> > >
