[
https://issues.apache.org/jira/browse/HIVE-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147328#comment-14147328
]
Rui Li commented on HIVE-7382:
------------------------------
Hi [~xuefuz],
Just to clarify, we currently use spark local mode to run the tests. And our
goal here is to use local-cluster mode to run the test, right?
So far I've found, local mode runs all the spark executor, backend, etc. in the
same JVM, while in local-cluster mode, master and workers run in the same JVM
and the executors run in separate JVMs. Local-cluster mode resembles the
standalone mode, except that master and workers run in the same JVM and
everything runs on a single machine.
Intuitively, cluster-local mode can catch more errors we may have. But this
mode seems to be intended only for spark tests and not exposed to users. We may
need to make sure if we really want to use it.
> Create a MiniSparkCluster and set up a testing framework [Spark Branch]
> -----------------------------------------------------------------------
>
> Key: HIVE-7382
> URL: https://issues.apache.org/jira/browse/HIVE-7382
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: Rui Li
> Labels: Spark-M1
>
> To automatically test Hive functionality over Spark execution engine, we need
> to create a test framework that can execute Hive queries with Spark as the
> backend. For that, we should create a MiniSparkCluser for this, similar to
> other execution engines.
> Spark has a way to create a local cluster with a few processes in the local
> machine, each process is a work node. It's fairly close to a real Spark
> cluster. Our mini cluster can be based on that.
> For more info, please refer to the design doc on wiki.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
