Hi, I'm trying to finish up a PR (https://github.com/apache/spark/pull/10089) which is currently failing PySpark tests. The instructions to run the test suite seem a little dated. I was able to find these: https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals http://spark.apache.org/docs/latest/building-spark.html
I've tried running `python/run-tests`, but it fails hard at the ORC tests. I suspect it has to do with the external libraries not being compiled or put in the right location. I've tried running `SPARK_TESTING=1 ./bin/pyspark python/pyspark/streaming/tests.py` as suggested, but this doesn't work on Spark 2.0. I've tried running `SPARK_TESTING=1 ./bin/spark-submit python/pyspark/streaming/tests.py`and that worked a little better, but it failed at `pyspark.streaming.tests.KafkaStreamTests`, with `java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaTestUtils`. I suspect the same issue with external libraries. I've compiling Spark with `build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package` with no trouble. Is there any better documentation somewhere about how to run the PySpark tests? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/How-to-run-PySpark-tests-tp16357.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
