we would cut it from the hadoop dependencies, but still allow IPC messages using it to be marshalled if the protoc-compiled classes were using the protobuf-2.5 JAR *and* that JAR was on the classpath.
it'd be come the homework of those apps which need protobuf-2.5, here hbase, to set things up. On Sat, 27 May 2023 at 14:39, 张铎(Duo Zhang) <palomino...@gmail.com> wrote: > For hbase, the problem is we rely on protobuf 2.5 for our coprocessors. > > See HBASE-27436. > > Cheng Pan <pan3...@gmail.com> 于2023年5月24日周三 10:00写道： > >> +CC dev@hbase >> >> Thanks, >> Cheng Pan >> >> On Fri, May 19, 2023 at 4:08 AM Steve Loughran >> <ste...@cloudera.com.invalid> wrote: >> > >> > >> > >> > On Thu, 18 May 2023 at 03:45, Cheng Pan <cheng...@apache.org> wrote: >> >> >> >> Steve, thanks for the information, I think HADOOP-17046 should be fine >> for the Spark case. >> >> >> >> Hadoop put the protobuf 3 into the pre-shaded hadoop-thirdparty, and >> the hadoop-client-runtime shades protobuf 2 during the package, which >> results in protobuf 2 and 3 co-exist in hadoop-client-runtime in different >> packages: >> >> >> >> - protobuf 2: org.apache.hadoop.shaded.com.google.protobuf >> >> - protobuf 3: org.apache.hadoop.thirdparty.protobuf >> > >> > j >> > oh, so in fact that "put it back in unshaded" change doesn't do >> anything useful through the hadoop-client lib. so it is very much useless. >> >> >> >> >> >> As HADOOP-18487 plans to mark the protobuf 2 optional, will this make >> hadoop-client-runtime does not ship protobuf 2? If yes, things become worse >> for downstream projects who consumes hadoop shaded client, like Spark, >> because it requires the user to add vanilla protobuf 2 jar into the >> classpath if they want to access those API. >> > >> > >> > Well, what applications are using >> > org.apache.hadoop.shaded.com.google.protobuf >> ? hadoop itself doesn't; it's only referenced in unshaded form because >> hbase wanted the IPC library to still work with the unshaded version they >> were still using. But if the parquet2 lib is now only available shaded, >> their protobuf compiled .class files aren't going to link to it, are they? >> > >> > does anyone know how spark + hbase + hadoop-client-runtime work so that >> spark can talk to an hbase server? especially: what is needed on the >> classpath, and what gets loaded for a call >> >> >> >> >> >> In summary, I think the current state is fine. But for security >> purposes, the Hadoop community may want to remove the EOL protobuf 2 >> classes from hadoop-client-runtime. >> > >> > >> > +1. the shaded one which is in use also needs upgrading. >> > >> >> >> >> Thanks, >> >> Cheng Pan >> >> >> >> >> >> On May 17, 2023 at 04:10:43, Dongjoon Hyun <dongj...@apache.org> >> wrote: >> >>> >> >>> Thank you for sharing, Steve. >> >>> >> >>> Dongjoon >> >>> >> >>> On Tue, May 16, 2023 at 11:44 AM Steve Loughran >> <ste...@cloudera.com.invalid> wrote: >> >>>> >> >>>> I have some bad news here which is even though hadoop cut protobuf >> 2.5 support, hbase team put it back in (HADOOP-17046). I don't know if the >> shaded hadoop client has removed that dependency on protobuf 2.5. >> >>>> >> >>>> In HADOOP-18487 i want to allow hadoop to cut that dependency, with >> hbase having to add it to the classpath if they still want it: >> >>>> https://github.com/apache/hadoop/pull/4996 >> >>>> >> >>>> It's been neglected -if you can help with review/test etc that'd be >> great. I'd love to get this into the 3.3.6 release. >> >>>> >> >>>> On Sat, 13 May 2023 at 08:36, Cheng Pan <cheng...@apache.org> wrote: >> >>>>> >> >>>>> Hi all, >> >>>>> >> >>>>> In SPARK-42452 (apache/spark#41153 [1]), I’m trying to remove >> protobuf 2.5.0 from the Spark dependencies. >> >>>>> >> >>>>> Spark does not use protobuf 2.5.0 directly, instead, it comes from >> other dependencies, with the following changes, now, Spark does not require >> protobuf 2.5.0. >> >>>>> >> >>>>> - SPARK-40323 upgraded ORC 1.8.0, which moved from protobuf 2.5.0 >> to a shaded protobuf 3 >> >>>>> >> >>>>> - SPARK-33212 switched from Hadoop vanilla client to Hadoop shaded >> client, also removed the protobuf 2 dependency. SPARK-42452 removed the >> support for Hadoop 2. >> >>>>> >> >>>>> - SPARK-14421 shaded and relocated protobuf 2.6.1, which is >> required by the kinesis client, into the kinesis assembly jar >> >>>>> >> >>>>> - Spark itself's core/connect/protobuf modules use protobuf 3, also >> shaded and relocated all protobuf 3 deps. >> >>>>> >> >>>>> Feel free to comment if you still have any concerns. >> >>>>> >> >>>>> [1] https://github.com/apache/spark/pull/41153 >> >>>>> >> >>>>> Thanks, >> >>>>> Cheng Pan >> >
