Willem van Asperen created HIVE-11574:
-----------------------------------------
Summary: ERROR 2245: Cannot get schema from loadFunc
org.apache.hive.hcatalog.pig.HCatLoader
Key: HIVE-11574
URL: https://issues.apache.org/jira/browse/HIVE-11574
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.14.0
Environment: HDP 2.4.4, CentOS 6.5
Reporter: Willem van Asperen
Priority: Minor
When running a job through Oozie, the user is propagated into running a pig
job. If that pig job reads using the {{HCatLoader}} or writes out using the
{{HCatStorer}}, a kerberos delegation ticket request is triggered. This does
not work as expected in a non-kerberized cluster.
One would expect that the value of property {{hive.metastore.sasl.enabled}} is
checked before attempting to obtain the TGT. Instead, the function
{{getHiveMetaClient}} of {{PigHCatUtil}} checks if a kerberos server principal
has been set using the property {{hive.metastore.kerberos.principal}}. Since
that is set to some default value, this results in unexpected behavior.
Setting {{hive.metastore.sasl.enabled}} to false is "undone" by the above
mentioned function as soon as a {{hive.metastore.kerberos.principal}} has been
set. So even though the log-file shows that the property
{{hive.metastore.sasl.enabled}} comes through as false, the system behaves as
if sasl is requested and starts the kerberos delegation ticket request. This
results in the following stack trace:
{{ERROR 2245: Cannot get schema from loadFunc
org.apache.hive.hcatalog.pig.HCatLoader
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during
parsing. Cannot get schema from loadFunc org.apache.hive.hcatalog.pig.HCatLoader
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1748)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1443)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:387)
at org.apache.pig.PigServer.executeBatch(PigServer.java:412)
at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
at
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:495)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:288)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: Failed to parse: Can not retrieve schema from loader
org.apache.hive.hcatalog.pig.HCatLoader@2acf062d
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:201)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1735)
... 27 more
Caused by: java.lang.RuntimeException: Can not retrieve schema from loader
org.apache.hive.hcatalog.pig.HCatLoader@2acf062d
at
org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:91)
at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:901)
at
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
... 28 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245:
Cannot get schema from loadFunc org.apache.hive.hcatalog.pig.HCatLoader
at
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:179)
at
org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89)
... 35 more
Caused by: java.io.IOException: java.lang.Exception: Could not instantiate a
HiveMetaStoreClient connecting to server
uri:[thrift://ip-10-223-8-183.eu-west-1.compute.internal:9083]
at
org.apache.hive.hcatalog.pig.PigHCatUtil.getTable(PigHCatUtil.java:213)
at
org.apache.hive.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:217)
at
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
... 36 more
Caused by: java.lang.Exception: Could not instantiate a HiveMetaStoreClient
connecting to server
uri:[thrift://ip-10-223-8-183.eu-west-1.compute.internal:9083]
at
org.apache.hive.hcatalog.pig.PigHCatUtil.getHiveMetaClient(PigHCatUtil.java:168)
at
org.apache.hive.hcatalog.pig.PigHCatUtil.getTable(PigHCatUtil.java:208)
... 38 more
Caused by: MetaException(message:Could not connect to meta store using any of
the URIs provided. Most recent failure:
org.apache.thrift.transport.TTransportException: GSS initiate failed
at
org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:358)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:215)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:161)
at
org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:302)
at
org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:226)
at
org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:223)
at
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
at
org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:223)
at
org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:199)
at
org.apache.hive.hcatalog.common.HCatUtil.getHiveClient(HCatUtil.java:571)
at
org.apache.hive.hcatalog.pig.PigHCatUtil.getHiveMetaClient(PigHCatUtil.java:166)
at
org.apache.hive.hcatalog.pig.PigHCatUtil.getTable(PigHCatUtil.java:208)
at
org.apache.hive.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:217)
at
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
at
org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89)
at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:901)
at
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1735)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1443)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:387)
at org.apache.pig.PigServer.executeBatch(PigServer.java:412)
at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
at
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:495)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:288)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:404)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:215)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:161)
at
org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:302)
at
org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:226)
at
org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:223)
at
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
at
org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:223)
at
org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:199)
at
org.apache.hive.hcatalog.common.HCatUtil.getHiveClient(HCatUtil.java:571)
at
org.apache.hive.hcatalog.pig.PigHCatUtil.getHiveMetaClient(PigHCatUtil.java:166)
... 39 more}}
The work-around is to un-set property {{hive.metastore.kerberos.principal}} in
oozie by including
<job-xml>/scripts/workflows/000-lib/hive-oozie-site.xml</job-xml> in the global
settings of your workflow and make sure that you do not set
{{hive.metastore.kerberos.principal}} in that hive-oozie-site.xml
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
