I have a string spark.driver.maxResultSize=0 in the spark-defaults.conf. But I get an error:
"org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 18 tasks (1070.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)" But if I write --conf spark.driver.maxResultSize=0 in pyspark-shell it works fine. Could anyone know how to fix it? Thank you -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-driver-maxResultSize-doesn-t-work-in-conf-file-tp26279.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
