I have some bad news here which is even though hadoop cut protobuf 2.5 support, hbase team put it back in (HADOOP-17046). I don't know if the shaded hadoop client has removed that dependency on protobuf 2.5.
In HADOOP-18487 i want to allow hadoop to cut that dependency, with hbase having to add it to the classpath if they still want it: https://github.com/apache/hadoop/pull/4996 It's been neglected -if you can help with review/test etc that'd be great. I'd love to get this into the 3.3.6 release. On Sat, 13 May 2023 at 08:36, Cheng Pan <cheng...@apache.org> wrote: > Hi all, > > In SPARK-42452 (apache/spark#41153 [1]), I’m trying to remove protobuf > 2.5.0 from the Spark dependencies. > > Spark does not use protobuf 2.5.0 directly, instead, it comes from other > dependencies, with the following changes, now, Spark does not require > protobuf 2.5.0. > > - SPARK-40323 upgraded ORC 1.8.0, which moved from protobuf 2.5.0 to a > shaded protobuf 3 > > - SPARK-33212 switched from Hadoop vanilla client to Hadoop shaded client, > also removed the protobuf 2 dependency. SPARK-42452 removed the support for > Hadoop 2. > > - SPARK-14421 shaded and relocated protobuf 2.6.1, which is required by > the kinesis client, into the kinesis assembly jar > > - Spark itself's core/connect/protobuf modules use protobuf 3, also shaded > and relocated all protobuf 3 deps. > > Feel free to comment if you still have any concerns. > > [1] https://github.com/apache/spark/pull/41153 > > Thanks, > Cheng Pan >
