date:20161029

Re: Spark has a compile dependency on scalatest

2016-10-29 Thread Sean Owen

Declare your scalatest dependency as test scope (which is correct anyway).
That would override it I think as desired?

On Fri, Oct 28, 2016, 21:22 Shixiong(Ryan) Zhu 
wrote:

> This is my test pom:
>
> 
> 4.0.0
>   foo
> bar
> 1.0
> 
>
> 
> org.apache.spark
> spark-core_2.10
> 2.0.1
> 
>  
>
> 
>
> scalatest is in the compile scope:
>
> [INFO] bar:foo:jar:1.0
> [INFO] \- org.apache.spark:spark-core_2.10:jar:2.0.1:compile
> [INFO]+- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
> [INFO]|  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
> [INFO]|  |  \- org.apache.avro:avro:jar:1.7.7:compile
> [INFO]|  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
> [INFO]|  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO]|  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO]+- com.twitter:chill_2.10:jar:0.8.0:compile
> [INFO]|  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
> [INFO]| +- com.esotericsoftware:minlog:jar:1.3.0:compile
> [INFO]| \- org.objenesis:objenesis:jar:2.1:compile
> [INFO]+- com.twitter:chill-java:jar:0.8.0:compile
> [INFO]+- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
> [INFO]+- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
> [INFO]|  |  +- commons-cli:commons-cli:jar:1.2:compile
> [INFO]|  |  +- org.apache.commons:commons-math:jar:2.1:compile
> [INFO]|  |  +- xmlenc:xmlenc:jar:0.52:compile
> [INFO]|  |  +- commons-io:commons-io:jar:2.1:compile
> [INFO]|  |  +- commons-lang:commons-lang:jar:2.5:compile
> [INFO]|  |  +-
> commons-configuration:commons-configuration:jar:1.6:compile
> [INFO]|  |  |  +-
> commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]|  |  |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO]|  |  |  |  \-
> commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO]|  |  |  \-
> commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO]|  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [INFO]|  |  +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
> [INFO]|  |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
> [INFO]|  | \- org.tukaani:xz:jar:1.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
> [INFO]|  |  \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
> [INFO]|  |  +-
> org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
> [INFO]|  |  |  +-
> org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
> [INFO]|  |  |  |  \- com.google.inject:guice:jar:3.0:compile
> [INFO]|  |  |  | +- javax.inject:javax.inject:jar:1:compile
> [INFO]|  |  |  | \- aopalliance:aopalliance:jar:1.0:compile
> [INFO]|  |  |  \-
> org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
> [INFO]|  |  \-
> org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
> [INFO]|  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
> [INFO]|  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
> [INFO]|  +-
> org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile
> [INFO]|  \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
> [INFO]+- org.apache.spark:spark-launcher_2.10:jar:2.0.1:compile
> [INFO]+- org.apache.spark:spark-network-common_2.10:jar:2.0.1:compile
> [INFO]|  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
> [INFO]|  \-
> com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:compile
> [INFO]+- org.apache.spark:spark-network-shuffle_2.10:jar:2.0.1:compile
> [INFO]+- org.apache.spark:spark-unsafe_2.10:jar:2.0.1:compile
> [INFO]+- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
> [INFO]|  +- commons-codec:commons-codec:jar:1.3:compile
> [INFO]|  \- commons-httpclient:commons-httpclient:jar:3.1:compile
> [INFO]+- org.apache.curator:curator-recipes:jar:2.4.0:compile
> [INFO]|  +- org.apache.curator:curator-framework:jar:2.4.0:compile
> [INFO]|  |  \- org.apache.curator:curator-client:jar:2.4.0:compile
> [INFO]|  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
> [INFO]|  \- com.google.guava:guava:jar:14.0.1:compile
> [INFO]+- javax.servlet:javax.servlet-api:jar:3.1.0:compile
> [INFO]+- org.apache.commons:commons-lang3:jar:3.3.2:compile
> [INFO]+- org.apache.commons:commons-math3:jar:3.4.1:compile
> [INFO]+- com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [INFO]+- org.slf4j:slf4j-api:jar:1.7.16:compile
> [INFO]+- org.slf4j:jul-to-slf4j:jar:1.7.16:compile
> [INFO]+- org.slf4j:jcl-over-slf4j:jar:1.7.16:compile
> [INFO]+- log4j:log4j:jar:1.2.17:compile
> [INFO]+- org.slf4j:s

Re: Spark has a compile dependency on scalatest

2016-10-29 Thread Steve Loughran


On 29 Oct 2016, at 10:50, Sean Owen 
mailto:so...@cloudera.com>> wrote:

Declare your scalatest dependency as test scope (which is correct anyway). That 
would override it I think as desired?


not sure about that, but then mvn dependencies are one of those graph-theory 
problems. It may just add it to the test CP alongside the compile one, so: no 
change. Someone would need to do an experiment there.

If the scalatags dependency is marked as provided, then maven won't put it on 
the transitive CP. And I assume, Ivy will work similarly


On Fri, Oct 28, 2016, 21:22 Shixiong(Ryan) Zhu 
mailto:shixi...@databricks.com>> wrote:
This is my test pom:


4.0.0
  foo
bar
1.0



org.apache.spark
spark-core_2.10
2.0.1

 



scalatest is in the compile scope:

[INFO] bar:foo:jar:1.0
[INFO] \- org.apache.spark:spark-core_2.10:jar:2.0.1:compile
[INFO]+- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
[INFO]|  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
[INFO]|  |  \- org.apache.avro:avro:jar:1.7.7:compile
[INFO]|  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
[INFO]|  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO]|  \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO]+- com.twitter:chill_2.10:jar:0.8.0:compile
[INFO]|  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
[INFO]| +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO]| \- org.objenesis:objenesis:jar:2.1:compile
[INFO]+- com.twitter:chill-java:jar:0.8.0:compile
[INFO]+- org.apache.xbean:xbean-asm5-shaded:jar:4.4:compile
[INFO]+- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
[INFO]|  |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO]|  |  +- org.apache.commons:commons-math:jar:2.1:compile
[INFO]|  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO]|  |  +- commons-io:commons-io:jar:2.1:compile
[INFO]|  |  +- commons-lang:commons-lang:jar:2.5:compile
[INFO]|  |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO]|  |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO]|  |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO]|  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO]|  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO]|  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO]|  |  +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
[INFO]|  |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO]|  | \- org.tukaani:xz:jar:1.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile
[INFO]|  |  \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO]|  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile
[INFO]|  |  +- 
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:compile
[INFO]|  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO]|  |  |  |  \- com.google.inject:guice:jar:3.0:compile
[INFO]|  |  |  | +- javax.inject:javax.inject:jar:1:compile
[INFO]|  |  |  | \- aopalliance:aopalliance:jar:1.0:compile
[INFO]|  |  |  \- 
org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO]|  |  \- 
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile
[INFO]|  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile
[INFO]|  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO]|  +- 
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:compile
[INFO]|  \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
[INFO]+- org.apache.spark:spark-launcher_2.10:jar:2.0.1:compile
[INFO]+- org.apache.spark:spark-network-common_2.10:jar:2.0.1:compile
[INFO]|  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO]|  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:compile
[INFO]+- org.apache.spark:spark-network-shuffle_2.10:jar:2.0.1:compile
[INFO]+- org.apache.spark:spark-unsafe_2.10:jar:2.0.1:compile
[INFO]+- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO]|  +- commons-codec:commons-codec:jar:1.3:compile
[INFO]|  \- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO]+- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO]|  +- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO]|  |  \- org.apache.curator:curator-client:jar:2.4.0:compile
[INFO]|  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO]|  \- com.google.guava:guava:jar:14.0.1:compile
[INFO]+- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO]+- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO]+- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO]+- com.google.code

GSoC projects related to Spark

2016-10-29 Thread aditya1702

Hello all,
I am really interested in Spark. I have been doing small projects in machine
learning using spark and would love to do a project in this year's GSoC. Can
anyone tell me are there any projects related to Spark for this year's GSoC?



--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/GSoC-projects-related-to-Spark-tp19653.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[info] Warning: Unknown ScalaCheck args provided: -oDF

2016-10-29 Thread Jacek Laskowski

Hi,

Just noticed the messages from the recent build of my pull request in Jenkins:

[info] Warning: Unknown ScalaCheck args provided: -oDF

I think we should fix it, right?

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Spark has a compile dependency on scalatest

Re: Spark has a compile dependency on scalatest

GSoC projects related to Spark

[info] Warning: Unknown ScalaCheck args provided: -oDF

4 matches

Site Navigation

Mail list logo

Footer information