[
https://issues.apache.org/jira/browse/SPARK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangJianfei updated SPARK-17535:
--------------------------------
Description:
I think the singleton pattern of SparkContext is inefficient if there are many
request to get the SparkContext,
So we can write the singleton pattern as below,The second way if more efficient
when there are many request to get the SparkContext.
{code}
// the current version
def getOrCreate(): SparkContext = {
SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
if (activeContext.get() == null) {
setActiveContext(new SparkContext(), allowMultipleContexts = false)
}
activeContext.get()
}
}
// by myself
def getOrCreate(): SparkContext = {
if (activeContext.get() == null) {
SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
if(activeContext.get == null) {
@volatile val sparkContext = new SparkContext()
setActiveContext(sparkContext, allowMultipleContexts = false)
}
}
activeContext.get()
}
}
{code}
was:
I think the singleton pattern of SparkContext is inefficient if there are many
request to get the SparkContext,
So we can write the singleton pattern as below:
{code}
// the current version
def getOrCreate(): SparkContext = {
SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
if (activeContext.get() == null) {
setActiveContext(new SparkContext(), allowMultipleContexts = false)
}
activeContext.get()
}
}
// by myself
def getOrCreate(): SparkContext = {
if (activeContext.get() == null) {
SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
if(activeContext.get == null) {
@volatile val sparkContext = new SparkContext()
setActiveContext(sparkContext, allowMultipleContexts = false)
}
}
activeContext.get()
}
}
{code}
> Performance Improvement of Signleton pattern in SparkContext
> ------------------------------------------------------------
>
> Key: SPARK-17535
> URL: https://issues.apache.org/jira/browse/SPARK-17535
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.0.0
> Reporter: WangJianfei
> Labels: easyfix, performance
>
> I think the singleton pattern of SparkContext is inefficient if there are
> many request to get the SparkContext,
> So we can write the singleton pattern as below,The second way if more
> efficient when there are many request to get the SparkContext.
> {code}
> // the current version
> def getOrCreate(): SparkContext = {
> SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
> if (activeContext.get() == null) {
> setActiveContext(new SparkContext(), allowMultipleContexts = false)
> }
> activeContext.get()
> }
> }
> // by myself
> def getOrCreate(): SparkContext = {
> if (activeContext.get() == null) {
> SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
> if(activeContext.get == null) {
> @volatile val sparkContext = new SparkContext()
> setActiveContext(sparkContext, allowMultipleContexts = false)
> }
> }
> activeContext.get()
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
