I have used protobuf 3 successfully with Spark on CDH 5.4, even though Hadoop itself comes with protobuf 2.5. I think the steps apply to HDP too. You need to do the following
1. Set the below parameter spark.executor.userClassPathFirst=true spark.driver.userClassPathFirst=true 2. Include protobuf 3 jar file either through —jars during the spark-submit or package it into a uber jar file with your own classes. Lan > On Nov 4, 2015, at 4:07 PM, Cassa L <[email protected]> wrote: > > Hi, > Does spark support protobuff 3.0? I used protobuff 2.5 with spark-1.4 built > for HDP 2.3. Given that protobuff has compatibility issues , want to know if > spark supports protbuff 3.0 > > LCassa --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
