What does spark.python.worker.memory affect?

2017-07-20 Thread Cyanny LIANG
worker subprocess, what is the memory limit for each python worker? does spark.python.worker.memory affect the memory of a python worker? -- Best & Regards Cyanny LIANG email: lgrcya...@gmail.com

Fwd: Does pyspark worker really use pipe?

2017-07-19 Thread Cyanny LIANG
anything I missed? why pyspark worker use socket not pipe? for performance reason? -- Best & Regards Cyanny LIANG email: lgrcya...@gmail.com

When will spark 2.0 support dataset python API?

2017-05-31 Thread Cyanny LIANG
/introducing-apache-spark-datasets.html it said, python API will be supported in spark 2.0 -- Best & Regards Cyanny LIANG email: lgrcya...@gmail.com