SparkPi running slower with more cores on each worker

2015-11-30 Thread yiskylee
Hi, I was running the SparkPi example code and studying their performance difference using different number of cores per worker. I change the number of cores by using start-slave.sh -c CORES on the worker machine for distributed computation. I also use spark-submit --master local[CORES] for the sam

scalastyle violation on mvn install but not on mvn package

2017-05-04 Thread yiskylee
./build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package works, but ./build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean install triggers scalastyle violation error. Is the scalastyle check not used on package but only on install? To install, should I

scalastyle violation on mvn install but not on mvn package

2017-05-17 Thread yiskylee
./build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package works, but ./build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean install triggers scalastyle violation error. Is the scalastyle check not used on package but only on install? To install, should