[
https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738309#comment-15738309
]
Saikat Kanjilal commented on SPARK-9487:
----------------------------------------
[~srowen] I think the above plan is great minus one fundamental flaw, I already
have tests passing uniformly across multiple components locally, the issue I am
running into is trying to get the tests working in jenkins, currently every
change I've made locally passes unit tests. Until the issue with my local
environment and jenkins gets resolved I don't see a clever way to get tests to
pass , let me know your thoughts on a good way to get past this. After we
figure this out I can pick a set of components to work with a uniform number of
threads.
> Use the same num. worker threads in Scala/Python unit tests
> -----------------------------------------------------------
>
> Key: SPARK-9487
> URL: https://issues.apache.org/jira/browse/SPARK-9487
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, Spark Core, SQL, Tests
> Affects Versions: 1.5.0
> Reporter: Xiangrui Meng
> Labels: starter
> Attachments: ContextCleanerSuiteResults, HeartbeatReceiverSuiteResults
>
>
> In Python we use `local[4]` for unit tests, while in Scala/Java we use
> `local[2]` and `local` for some unit tests in SQL, MLLib, and other
> components. If the operation depends on partition IDs, e.g., random number
> generator, this will lead to different result in Python and Scala/Java. It
> would be nice to use the same number in all unit tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
