Awesome ! Looks like a great distribution. I'd love to add HDP releases to the kafka-python integration test suite.
-Dana On Dec 24, 2015 9:33 AM, "Harsha" <ka...@harsha.io> wrote: > HI Dana, > I worked on that release. Yes HDP-2.3.0 has lot of > additional patches on top of 0.8.2.1 mainly the kerberos > patches. > We did missed KAFKA-1841 which was fixed in later maint release. We > already notified everyone to upgrade HDP-2.3.4 this is the apache kafka > 0.9.0 + additional patches . We are making sure on our side not to miss > any compatibility patches like these with 3rd party developers and have > tests to ensure that. > > Thanks, > Harsha > On Wed, Dec 23, 2015, at 04:11 PM, Dana Powers wrote: > > Hi all, > > > > I've been helping debug an issue filed against kafka-python related to > > compatibility w/ Hortonworks 2.3.0.0 kafka release. As I understand it, > > HDP > > is currently based on snapshots of apache/kafka trunk, merged with some > > custom patches from HDP itself. > > > > In this case, HDP's 2.3.0.0 kafka release missed a compatibility patch > > that > > I believe is critical for third-party library support. Unfortunately the > > patch -- KAFKA-1841 -- was initially only applied to the 0.8.2 branch (it > > was merged to trunk several months later in KAFKA-2068). Because it > > wasn't > > on trunk, it didn't get included in the HDP kafka releases. > > > > If you recall, KAFKA-1841 was needed to maintain backwards and forwards > > compatibility wrt the change from zookeeper to kafka-backed offset > > storage. > > Not having this patch is fine if you only ever use the clients / > > libraries > > distributed in the that release -- and I imagine that is probably most > > folks that are using it. But if you remember the thread on this issue > > back > > in the 0.8.2-beta review, the API incompatibility made third-party > > clients > > hard to develop and maintain if the goal is to support multiple broker > > versions w/ the same client code [this is the goal of kafka-python]. > > Anyways, I'm really glad that the fix made it into the apache release, > > but > > now I'm sad that it didn't make it into HDP's release. > > > > Anyways, I think there's a couple takeaways here: > > > > (1) I'd recommend anyone using HDP who intends to use third-party kafka > > consumers should upgrade to 2.3.4.0 or later. That version appears to > > include the compatibility patch (KAFKA-2068). Of course if anyone is on > > list from HDP, they may be able to provide better help on this. > > > > (2) I think more care should probably be taken to help vendors or anyone > > tracking changes on trunk wrt released versions. Is there a list of all > > KAFKA-XXXX patches that are released but not merged into trunk ? > > KAFKA-1841 > > is obviously near and dear to my heart, but I wonder if there are other > > patches like it? > > > > Happy holidays to all, and may the force be with you > > > > -Dana >
