i am very surprised this works, since spark depends on akka remote which uses protobuf 2.5
On Mon, Nov 9, 2015 at 2:28 PM, Lan Jiang <ljia...@gmail.com> wrote: > I have not run into any linkage problem, but maybe I was lucky. :-). The > reason I wanted to use protobuf 3 is mainly for Map type support. > > On Thu, Nov 5, 2015 at 4:43 AM, Steve Loughran <ste...@hortonworks.com> > wrote: > >> >> > On 5 Nov 2015, at 00:12, Lan Jiang <ljia...@gmail.com> wrote: >> > >> > I have used protobuf 3 successfully with Spark on CDH 5.4, even though >> Hadoop itself comes with protobuf 2.5. I think the steps apply to HDP too. >> You need to do the following >> >> >> Protobuf.jar has been so brittle in the past that the entire hadoop stack >> has effectively frozen @ 2.5, at least until another mass-coordinated >> update across as many projects as possible. If you do update it locally, >> you are highly likely to find linkage problems. I would strongly advise >> staying with 2.5 unless there is some pressing need and you are happy to >> take on all the pain yourself >> > >
