[
https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196761#comment-16196761
]
Peter Vary commented on HIVE-17111:
-----------------------------------
Sorry for the delay, I had other tasks which took all of my time, and some :(
I think it would be useful to have the possibility to run every spark test with
the local driver - so that is a plus for the TestLocalSparkCliDriver.
We should only add more tests, if we can be sure that it significantly improves
runtime.
The old method of adding new driver class is changed by HIVE-14444, so this is
the new method.
So this is a +1 from me too.
> Add TestLocalSparkCliDriver
> ---------------------------
>
> Key: HIVE-17111
> URL: https://issues.apache.org/jira/browse/HIVE-17111
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Attachments: HIVE-17111.1.patch
>
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but
> the HoS still uses decides to use the RemoteHiveSparkClient rather than the
> LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
> if (master.equals("local") || master.startsWith("local[")) {
> // With local spark context, all user sessions share the same spark
> context.
> return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
> } else {
> return new RemoteHiveSparkClient(hiveconf, sparkConf);
> }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and
> sees that it doesn't start with {{local[}} and then decides to use the
> RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed
> up some of the tests, and also makes qtests easier to debug since everything
> will now be run in the same process.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
