[
https://issues.apache.org/jira/browse/SPARK-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Armin Braun updated SPARK-19592:
--------------------------------
Description:
This configuration for Surefire, Scalatest is duplicated in the parent POM as
well as the SBT build.
While this duplication cannot be removed in general it can at least be removed
for all system properties that simply result in a SparkConf setting I think.
Instead of having lines like
{code}
<spark.ui.enabled>false</spark.ui.enabled>
{code}
twice in the pom.xml
and once in SBT as
{code}
javaOptions in Test += "-Dspark.ui.enabled=false",
{code}
it would be a lot cleaner to simply have a
{code}
var conf: SparkConf
{code}
field in
{code}
org.apache.spark.SparkFunSuite
{code}
that has SparkConf set up with all the shared configuration that
`systemProperties` currently provide. Obviously this cannot be done straight
away given that
many subclasses of the parent suit do this, so I think it would be best to
simply add a method to the parent that provides this configuration for now
and start refactoring away duplication in other suit setups from there step by
step until the sys properties can be removed from the pom and sbt.build.
This makes the build a lot easier to maintain and makes tests more readable by
making the environment setup more explicit in the code.
(also it would allow running more tests straight from the IDE which is always a
nice thing imo)
was:
This configuration for Surefire, Scalatest is duplicated in the parent POM as
well as the SBT build.
While this duplication cannot be removed in general it can at least be removed
for all system properties that simply result in a SparkConf setting I think.
Instead of having lines like
{code}
<spark.ui.enabled>false</spark.ui.enabled>
{code}
twice in the pom.xml
and once in SBT as
{code}
javaOptions in Test += "-Dspark.ui.enabled=false",
{code}
it would be a lot cleaner to simply have a
{code}
var conf: SparkConf
{code}
field in
{code}
org.apache.spark.SparkFunSuite
{code}
that has SparkConf set up with all the shared configuration that
`systemProperties` currently provide.
This makes the build a lot easier to maintain and makes tests more readable by
making the environment setup more explicit in the code.
(also it would allow running more tests straight from the IDE which is always a
nice thing imo)
> Duplication in Test Configuration Relating to SparkConf Settings Should be
> Removed
> ----------------------------------------------------------------------------------
>
> Key: SPARK-19592
> URL: https://issues.apache.org/jira/browse/SPARK-19592
> Project: Spark
> Issue Type: Improvement
> Components: Tests
> Affects Versions: 2.1.0
> Environment: Applies to all Environments
> Reporter: Armin Braun
> Priority: Minor
>
> This configuration for Surefire, Scalatest is duplicated in the parent POM as
> well as the SBT build.
> While this duplication cannot be removed in general it can at least be
> removed for all system properties that simply result in a SparkConf setting I
> think.
> Instead of having lines like
> {code}
> <spark.ui.enabled>false</spark.ui.enabled>
> {code}
> twice in the pom.xml
> and once in SBT as
> {code}
> javaOptions in Test += "-Dspark.ui.enabled=false",
> {code}
> it would be a lot cleaner to simply have a
> {code}
> var conf: SparkConf
> {code}
> field in
> {code}
> org.apache.spark.SparkFunSuite
> {code}
> that has SparkConf set up with all the shared configuration that
> `systemProperties` currently provide. Obviously this cannot be done straight
> away given that
> many subclasses of the parent suit do this, so I think it would be best to
> simply add a method to the parent that provides this configuration for now
> and start refactoring away duplication in other suit setups from there step
> by step until the sys properties can be removed from the pom and sbt.build.
> This makes the build a lot easier to maintain and makes tests more readable
> by making the environment setup more explicit in the code.
> (also it would allow running more tests straight from the IDE which is always
> a nice thing imo)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
