External shuffle service on K8S

2018-10-26 Thread
Hi all: Does Spark 2.3.2 supports external shuffle service on Kubernetes? I have looked up the documentation( https://spark.apache.org/docs/latest/running-on-kubernetes.html), but couldn't find related suggestions. If suppports, how can I enable it? Best Regards Lijun Cao

Internal Spark class is not registered by Kryo

2018-10-09 Thread
Hi all: I have set spark.kryo.registrationRequired=true, but an exception occured: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run the program. I tried to register it manually by kryo.register() and Sparkcon

Internal Spark class is not registered by Kryo

2018-10-09 Thread
Hi all: I have set spark.kryo.registrationRequired=true, but an exception occured: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run the program. I tried to register it manually by kryo.register() and Spa