[
https://issues.apache.org/jira/browse/HIVE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918414#comment-13918414
]
Sushanth Sowmyan commented on HIVE-6409:
----------------------------------------
I'm +1 on this new patch, I'll go ahead and commit it.
As to your other concerns, this is per HIVE-5274, and per discussions that I
brought up in the next hive-dev meetup(after that patch) as well.
As to visibly breaking code, we will do that, as of 0.14, which will be trunk
in a couple more days as soon as we branch out for 0.13 - we have HIVE-6432
open for just that. The idea is that org.apache.hcatalog.* works predictably in
the way it has for all of those that have already been using it and find it
usable in the state it is in, whereas to all new users who may be evaluating,
they must be moving to org.apache.hive.hcatalog.
> FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
> ------------------------------------------------------------------------------
>
> Key: HIVE-6409
> URL: https://issues.apache.org/jira/browse/HIVE-6409
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
> Affects Versions: 0.12.0, 0.13.0
> Reporter: Mithun Radhakrishnan
> Assignee: Mithun Radhakrishnan
> Attachments: HIVE-6409.patch
>
>
> When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls
> the underlying OutputCommitter and then attempts to register partitions in
> HCatalog.
> If the commit fails (for example, because of HIVE-4996), commitJob() cancels
> delegation tokens retrieved from HCatalog before the exception is rethrown.
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
> ... 5 more
> Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
> partition to metastore. Cause :
> MetaException(message:java.lang.RuntimeException: commitTransaction was called
> but openTransactionCalls = 0. This probably indicates that there are
> unbalanced
> calls to openTransaction/commitTransaction)
> at
> org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
> {code}
> The problem is that this happens before abortJob() has had a chance to run,
> thus yielding the following error:
> {code}
> MetaException(message:Could not connect to meta store using any of the URIs
> provided. Most recent failure:
> org.apache.thrift.transport.TTransportException: Peer indicated failure:
> DIGEST-MD5: IO error acquiring password
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> at java.security.AccessController.doPrivileged(Native Method)
> ...
> {code}
> I'll have a patch out that only cancels delegation tokens if the commitJob()
> has succeeded.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
