In Hive 13 (which is the default for Spark 1.2), parquet is included and thus we no longer include the Hive parquet bundle. You can now use the included ParquetSerDe: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
If you want to compile Spark 1.2 with Hive 12 instead you can pass -Phive-0.12.0 and parquet.hive.serde.ParquetHiveSerDe will be included as before. Michael On Tue, Dec 2, 2014 at 9:31 AM, Yana Kadiyska <yana.kadiy...@gmail.com> wrote: > Apologies if people get this more than once -- I sent mail to dev@spark > last night and don't see it in the archives. Trying the incubator list > now...wanted to make sure it doesn't get lost in case it's a bug... > > ---------- Forwarded message ---------- > From: Yana Kadiyska <yana.kadiy...@gmail.com> > Date: Mon, Dec 1, 2014 at 8:10 PM > Subject: [Thrift,1.2 RC] what happened to > parquet.hive.serde.ParquetHiveSerDe > To: dev@spark.apache.org > > > Hi all, apologies if this is not a question for the dev list -- figured > User list might not be appropriate since I'm having trouble with the RC > tag. > > I just tried deploying the RC and running ThriftServer. I see the following > error: > > 14/12/01 21:31:42 ERROR UserGroupInformation: PriviledgedActionException > as:anonymous (auth:SIMPLE) > cause:org.apache.hive.service.cli.HiveSQLException: > java.lang.RuntimeException: > MetaException(message:java.lang.ClassNotFoundException Class > parquet.hive.serde.ParquetHiveSerDe not found) > 14/12/01 21:31:42 WARN ThriftCLIService: Error executing statement: > org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: > MetaException(message:java.lang.ClassNotFoundException Class > parquet.hive.serde.ParquetHiveSerDe not found) > at > > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:192) > at > > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231) > at > > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:212) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > > org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79) > at > > org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37) > at > > org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > > > > I looked at a working installation that I have(build master a few weeks > ago) and this class used to be included in spark-assembly: > > ls *.jar|xargs grep parquet.hive.serde.ParquetHiveSerDe > Binary file spark-assembly-1.2.0-SNAPSHOT-hadoop2.0.0-mr1-cdh4.2.0.jar > matches > > but with the RC build it's not there? > > I tried both the prebuilt CDH drop and later manually built the tag with > the following command: > > ./make-distribution.sh --tgz -Phive -Dhadoop.version=2.0.0-mr1-cdh4.2.0 > -Phive-thriftserver > $JAVA_HOME/bin/jar -tvf spark-assembly-1.2.0-hadoop2.0.0-mr1-cdh4.2.0.jar > |grep parquet.hive.serde.ParquetHiveSerDe > > comes back empty... >
