[
https://issues.apache.org/jira/browse/HIVE-10672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14538843#comment-14538843
]
Alexander Pivovarov commented on HIVE-10672:
--------------------------------------------
if we build hive running
{code}
$ mvn clean install -DskipTests -Phadoop-2,dist
{code}
then we will see that
packaging/target/apache-hive-1.3.0-SNAPSHOT-bin/apache-hive-1.3.0-SNAPSHOT-bin/lib
does not have hive-hcatalog-core-1.3.0-SNAPSHOT.jar
if we start hive by
{code}
$ cd
<HIVE_PROJ_HOME>/packaging/target/apache-hive-1.3.0-SNAPSHOT-bin/apache-hive-1.3.0-SNAPSHOT-bin
$ bin/hive
{code}
and try to create the table then create table statements (which refers to
'org.apache.hive.hcatalog.data.JsonSerDe') will fail because hive can not find
JsonSerDe class (because hive-hcatalog-core.jar it's not in hive/lib)
In order to solve this problem we need to add
hive-hcatalog-core-1.3.0-SNAPSHOT.jar as a resource
{code}
hive> ADD JARS
<HIVE_PROJ_HOME>/hcatalog/core/target/hive-hcatalog-core-1.3.0-SNAPSHOT.jar;
{code}
after adding hive-hcatalog-core.jar we can create the table and try to run
'analyze table'.
Analyze table MR job will fail
{code}
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1431383083931_0004, Tracking URL =
http://c11.example.com:8088/proxy/application_1431383083931_0004/
Kill Command = /usr/lib/hadoop-2.6.0/bin/hadoop job -kill
job_1431383083931_0004
Hadoop job information for Stage-0: number of mappers: 1; number of reducers: 0
2015-05-11 15:43:48,524 Stage-0 map = 0%, reduce = 0%
2015-05-11 15:44:13,937 Stage-0 map = 100%, reduce = 0%
Ended Job = job_1431383083931_0004 with errors
...
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.metastore.IMetaStoreClient
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
{code}
This problem also can be solved by adding another jar
hive-metastore-1.3.0-SNAPSHOT.jar as a resource
{code}
hive> ADD JARS
<HIVE_PROJ_HOME>/metastore/target/hive-metastore-1.3.0-SNAPSHOT.jar;
{code}
if we look at hive/lib folder we can see that hive-metastore-1.3.0-SNAPSHOT.jar
exists in the folder.
But for some reason hive-metastore-1.3.0-SNAPSHOT.jar is not included to
MapReduce classpath.
First question. Should hive-metastore.jar be included to MR classpath by
default?
If yes then we need to find the place in hive which controls what jars are
added to MR classpath and add hive-metastore.jar into it.
Otherwise we have to manually add hive-metastore.jar as a resource.
> Analyze command on a table using row format serde JsonSerDe fails with
> NoClassDefFoundError
> -------------------------------------------------------------------------------------------
>
> Key: HIVE-10672
> URL: https://issues.apache.org/jira/browse/HIVE-10672
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Reporter: Jason Dere
> Assignee: Jason Dere
> Attachments: HIVE-10672.1.patch
>
>
> Found by [~deepesh].
> Running analyze command on a table created using the following DDL:
> {noformat}
> create external table all100kjson(
> s string,
> i int,
> d double,
> m map<string, string>,
> bb array<struct<a: int, b: string>>)
> row format serde 'org.apache.hive.hcatalog.data.JsonSerDe'
> STORED AS TEXTFILE
> location '/user/hcat/tests/data/all100kjson';
> {noformat}
> analyze command
> {noformat}
> analyze table all100kjson compute statistics;
> {noformat}
> throws the following error:
> {noformat}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1431071702167_0006_1_00,
> diagnostics=[Task failed, taskId=task_1431071702167_0006_1_00_000000,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:331)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/metastore/IMetaStoreClient
> at
> org.apache.hive.hcatalog.data.schema.HCatFieldSchema.<init>(HCatFieldSchema.java:225)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:122)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
> at
> org.apache.hive.hcatalog.data.JsonSerDe.initialize(JsonSerDe.java:141)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:527)
> at
> org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:191)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.metastore.IMetaStoreClient
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 25 more
> ], TaskAttempt 1 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:331)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/metastore/IMetaStoreClient
> at
> org.apache.hive.hcatalog.data.schema.HCatFieldSchema.<init>(HCatFieldSchema.java:225)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:122)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
> at
> org.apache.hive.hcatalog.data.JsonSerDe.initialize(JsonSerDe.java:141)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:527)
> at
> org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:191)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.metastore.IMetaStoreClient
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 25 more
> ], TaskAttempt 2 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:331)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/metastore/IMetaStoreClient
> at
> org.apache.hive.hcatalog.data.schema.HCatFieldSchema.<init>(HCatFieldSchema.java:225)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:122)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
> at
> org.apache.hive.hcatalog.data.JsonSerDe.initialize(JsonSerDe.java:141)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:527)
> at
> org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:191)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.metastore.IMetaStoreClient
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 25 more
> ], TaskAttempt 3 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:331)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/metastore/IMetaStoreClient
> at
> org.apache.hive.hcatalog.data.schema.HCatFieldSchema.<init>(HCatFieldSchema.java:225)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:122)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
> at
> org.apache.hive.hcatalog.data.JsonSerDe.initialize(JsonSerDe.java:141)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:527)
> at
> org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:191)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.metastore.IMetaStoreClient
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 25 more
> ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
> killedTasks:2, Vertex vertex_1431071702167_0006_1_00 [Map 1] killed/failed
> due to:null]
> DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_1431071702167_0006_1_00, diagnostics=[Task failed,
> taskId=task_1431071702167_0006_1_00_000000, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:java.lang.RuntimeException:
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:331)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/metastore/IMetaStoreClient
> at
> org.apache.hive.hcatalog.data.schema.HCatFieldSchema.<init>(HCatFieldSchema.java:225)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:122)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
> at
> org.apache.hive.hcatalog.data.JsonSerDe.initialize(JsonSerDe.java:141)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:527)
> at
> org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:191)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.metastore.IMetaStoreClient
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 25 more
> ], TaskAttempt 1 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:331)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/metastore/IMetaStoreClient
> at
> org.apache.hive.hcatalog.data.schema.HCatFieldSchema.<init>(HCatFieldSchema.java:225)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:122)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
> at
> org.apache.hive.hcatalog.data.JsonSerDe.initialize(JsonSerDe.java:141)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:527)
> at
> org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:191)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.metastore.IMetaStoreClient
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 25 more
> ], TaskAttempt 2 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:331)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/metastore/IMetaStoreClient
> at
> org.apache.hive.hcatalog.data.schema.HCatFieldSchema.<init>(HCatFieldSchema.java:225)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:122)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
> at
> org.apache.hive.hcatalog.data.JsonSerDe.initialize(JsonSerDe.java:141)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:527)
> at
> org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:191)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.metastore.IMetaStoreClient
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 25 more
> ], TaskAttempt 3 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator
> initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:331)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:229)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> ... 14 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/metastore/IMetaStoreClient
> at
> org.apache.hive.hcatalog.data.schema.HCatFieldSchema.<init>(HCatFieldSchema.java:225)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:122)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
> at
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
> at
> org.apache.hive.hcatalog.data.JsonSerDe.initialize(JsonSerDe.java:141)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:527)
> at
> org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:191)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.metastore.IMetaStoreClient
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 25 more
> ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
> killedTasks:2, Vertex vertex_1431071702167_0006_1_00 [Map 1] killed/failed
> due to:null]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1
> killedVertices:0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
