[
https://issues.apache.org/jira/browse/HIVE-7950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127789#comment-14127789
]
Josh Elser commented on HIVE-7950:
----------------------------------
I have found one corner-case which I'm still trying to maneuver around. The DAG
code fails if local resources are added that already exist in the "state" of
extra resources that are already going to be added. The problem is that you
can't get find out what resources are already set to be localized for a DAG.
I can call {{DAG#addTaskLocalFiles(Map<String, LocalResource>)}} to add
resources but that will fail if any of them happen to be already loaded. This
seems to be lacking WRT to Vertex which also has a {{getTaskLocalFiles()}}
method. That's a Tez nit -- I can open a JIRA over there if you think that's
necessary (or not already fixed upstream).
This is the actual stack I'm trying to work around:
{noformat}
org.apache.tez.dag.api.TezUncheckedException: Attempting to add duplicate
resource: accumulo-fate-1.6.0.jar
at
org.apache.tez.common.TezCommonUtils.addAdditionalLocalResources(TezCommonUtils.java:307)
at org.apache.tez.dag.api.Vertex.addTaskLocalFiles(Vertex.java:256)
at org.apache.tez.dag.api.DAG.createDag(DAG.java:643)
at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:372)
at org.apache.tez.client.TezClient.submitDAG(TezClient.java:342)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:385)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:209)
...
{noformat}
Essentially, we have a DAG, we tried to submit it to the Session we have, but
the underlying application was dead (for testing purposes, because I
{{kill}}'ed it, but this happens if you just wait long enough). The code gets a
{{SessionNotRunning}} exception, tries to {{closeAndOpen}}
> StorageHandler resources aren't added to Tez Session if already Session is
> already Open
> ---------------------------------------------------------------------------------------
>
> Key: HIVE-7950
> URL: https://issues.apache.org/jira/browse/HIVE-7950
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler, Tez
> Reporter: Josh Elser
> Assignee: Josh Elser
> Fix For: 0.14.0
>
> Attachments: HIVE-7950-1.diff, hive-7950-tez-WIP.diff
>
>
> Was trying to run some queries using the AccumuloStorageHandler when using
> the Tez execution engine. Some things that classes which were added to
> tmpjars weren't making it into the container. When a Tez Session is already
> open, as is the normal case when simply using the `hive` command, the
> resources aren't added.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
