[
https://issues.apache.org/jira/browse/HIVE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mithun Radhakrishnan updated HIVE-6409:
---------------------------------------
Attachment: HIVE-6409.patch
I've renamed the patch, removed the changes to the deprecated code and
resubmitted.
About not making changes to org.apache.hcatalog: It is really quite dangerous
to patch fixes only into org.apache.hive.hcatalog. :/ We'll have to make sure
that projects like Pig are using the right version of the packages, for fixes.
Wouldn't it be preferable to remove the package entirely (and visibly break
code) than to silently compile and produce something unusable? (Has this been
discussed already?)
> FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
> ------------------------------------------------------------------------------
>
> Key: HIVE-6409
> URL: https://issues.apache.org/jira/browse/HIVE-6409
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
> Affects Versions: 0.12.0, 0.13.0
> Reporter: Mithun Radhakrishnan
> Assignee: Mithun Radhakrishnan
> Attachments: HIVE-6409.patch
>
>
> When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls
> the underlying OutputCommitter and then attempts to register partitions in
> HCatalog.
> If the commit fails (for example, because of HIVE-4996), commitJob() cancels
> delegation tokens retrieved from HCatalog before the exception is rethrown.
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
> ... 5 more
> Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
> partition to metastore. Cause :
> MetaException(message:java.lang.RuntimeException: commitTransaction was called
> but openTransactionCalls = 0. This probably indicates that there are
> unbalanced
> calls to openTransaction/commitTransaction)
> at
> org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
> {code}
> The problem is that this happens before abortJob() has had a chance to run,
> thus yielding the following error:
> {code}
> MetaException(message:Could not connect to meta store using any of the URIs
> provided. Most recent failure:
> org.apache.thrift.transport.TTransportException: Peer indicated failure:
> DIGEST-MD5: IO error acquiring password
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> at java.security.AccessController.doPrivileged(Native Method)
> ...
> {code}
> I'll have a patch out that only cancels delegation tokens if the commitJob()
> has succeeded.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
