Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3121#issuecomment-62225490
I'd say it's pretty serious, since it's possible for users to accidentally
create multiple SparkContexts and the presence of multiple active SparkContexts
is a known cause of some broadcast errors.
Here are some references:
- [A user created multiple SparkContexts, which caused "failed to get
broadcast_0_piece_0..."
error](https://issues.apache.org/jira/browse/SPARK-4080?focusedCommentId=14191349&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14191349)
- [SPARK-4133](https://issues.apache.org/jira/browse/SPARK-4133): a user
accidentally created multiple SparkContexts when initializing a Spark Streaming
application:
```
val sparkConf = sparkCtxBuilder.createSparkConf()
val sparkContext = new SparkContext(sparkConf)
val ssc = new StreamingContext(sparkConf, Seconds(5)) // sparkConf
passed, so this creates its own SparkContext
```
This caused `PARSING_ERROR(2)` in TorrentBroadcast (due to an empty byte
array being read) and `FileNotFoundException` in HttpBroadcast.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
