Re: UnknownHostException on submission Spark jobs to an AWS EMR cluster

2019-08-03 Thread Jeff Zhang
It depends on which mode do you use? If you use yarn client mode, then driver run in the zeppelin host, the driver needs to connect with executor which run in EMR. If you use yarn-cluster mode, then the driver run in EMR. Then the driver needs to connect with zeppelin server which is outside of EMR

Re: UnknownHostException on submission Spark jobs to an AWS EMR cluster

2019-08-02 Thread Yan Yang
Jeff When we run the Spark interpreter against remote cluster, does the interpreter process run locally or on the Spark cluster? Which port do we need to open on the zeppelin-server for the interpreter? Thanks a lot for the help. Yan

Re: UnknownHostException on submission Spark jobs to an AWS EMR cluster

2019-08-01 Thread Jeff Zhang
Do you see the error in yarn am log ? I suspect it is due to network issue. Because zeppelin needs bidirectional communication between zeppelin-server and interpreter process. Does your EMR cluster able to access your zeppelin server host ? Yan Yang 于2019年8月2日周五 上午12:58写道: > We have been trying

UnknownHostException on submission Spark jobs to an AWS EMR cluster

2019-08-01 Thread Yan Yang
We have been trying to submit Spark jobs to an AWS EMR cluster from our own Zeppelin instance. The YARN job was received and started properly, but ran into below error. The host indicated in the error is a weird 12-char alphanumeric string that does not look like hostname or IP. *Caused by: java.