[
https://issues.apache.org/jira/browse/HIVE-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630689#comment-14630689
]
Chengxiang Li commented on HIVE-11276:
--------------------------------------
Besides, for the case of dynamic allocation, i'm not sure whether it would be
influenced by this. From my point of view, as we use Spark API like
SparkContext::addJar()/addFile() to upload resources to SparkCluster, after
that, it should be Spark's responsibility to make sure it's executor JVM load
these resources. From the experience of my previous test of dynamic allocation,
everything works well.
> Optimization around job submission and adding jars [Spark Branch]
> -----------------------------------------------------------------
>
> Key: HIVE-11276
> URL: https://issues.apache.org/jira/browse/HIVE-11276
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Affects Versions: 1.1.0
> Reporter: Xuefu Zhang
> Assignee: Chengxiang Li
>
> It seems that Hive on Spark has some room for performance improvement on job
> submission. Specifically, we are calling refreshLocalResources() for every
> job submission despite there is are no changes in the jar list. Since Hive on
> Spark is reusing the containers in the whole user session, we might be able
> to optimize that.
> We do need to take into consideration the case of dynamic allocation, in
> which new executors might be added.
> This task is some R&D in this area.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
