[
https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225049#comment-14225049
]
Marcelo Vanzin commented on HIVE-8836:
--------------------------------------
I talked briefly with Brock about this, but the main thing here is that, right
now, Spark is not very friendly to applications that are trying to embed it. As
you've noticed, the assembly jar, which contains almost everything you need to
run Spark, is not published in maven or anywhere. And not all artifacts used to
build the assembly are published - for example, the Yarn backend cannot be
found anywhere in maven, so without the assembly you cannot submit jobs to Yarn.
I've suggested it in the past, but I think right now, or until Spark makes
itself more friendly to such use cases, Hive should require a full Spark
install to work. If desired we could use the hacks I added to the remote client
to not need the full install for unit tests, but even those are very limited;
it probably only works with a "local" master as some of you may have noticed.
> Enable automatic tests with remote spark client.[Spark Branch]
> --------------------------------------------------------------
>
> Key: HIVE-8836
> URL: https://issues.apache.org/jira/browse/HIVE-8836
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Chengxiang Li
> Assignee: Rui Li
> Labels: Spark-M3
> Attachments: HIVE-8836-brock-1.patch, HIVE-8836-brock-2.patch,
> HIVE-8836-brock-3.patch, HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch,
> HIVE-8836.3-spark.patch
>
>
> In real production environment, remote spark client should be used to submit
> spark job for Hive mostly, we should enable automatic test with remote spark
> client to make sure the Hive feature workable with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
