Help needed.
1. got zeppelin running in a docker container
2. got remote spark standalone cluster I want to run paragraphs against
I have:
- created a setting with
-- master - $MASTER_IP/7077
-- 'spark.driver.host' - ip of the docker container 172.18.0.2
-- 'spark.driver.port' - free port number (I have scanned range of
forwarded ports 40000-40100 for a free one) 40099
-- 'spark.driver.bindAddress' - host IP address
I can see the following in logs when trying to run a paragraph
...WARN [2017-06-22 15:10:21,827] ({pool-2-thread-2}
Logging.scala[logWarning]:66) - Service 'sparkDriver' could not bind on
port 40114. Attempting port 40115.
ERROR [2017-06-22 15:10:21,835] ({pool-2-thread-2}
Logging.scala[logError]:91) - Error initializing SparkContext.
java.net.BindException: Cannot assign requested address: Service
'sparkDriver' failed after 16 retries (starting from 40099)! Consider
explicitly setting the appropriate port for the service 'sparkDriver' (for
example spark.ui.port for SparkUI) to an available port or increasing
spark.port.maxRetries.
But when I remove 'spark.driver.bindAddress' - paragraph job is
successfully submitted but apparently cluster cannot see the driver
Caused by: java.io.IOException: Failed to connect to /172.18.0.2:40099
Please help, any ideas are more than appreciated
Thanks in advance
--
Ivan Shapovalov
Kharkov, Ukraine
