Good afternoon. Perhaps you will be discouraged by what I will write below, but nevertheless, I ask for help in solving my problem. Perhaps the architecture of our solution will not seem correct to you. There are backend services that communicate with a service that implements spark-driver. When the service starts with driver, spark-submit occurs and the session lives until the service stops. The service works constantly. Faced problems when they began to deploy our solution in k8s. The services were located inside the k8s cluster and the Spark StandAlone cluster outside the k8s. When starting the service from spark-driver, spark-submit is executed, which confirms the presence of an application on the UI. But on workers, we get an error that the workman could not connect to a random port to the spark-driver. The ports themselves have learned to override and specify, but these ports must be accessible from outside the cluster. We found a solution in which we open NodePort on workers - it works. BUT this is not suitable for most customers due to internal regulations. How to resolve the issue through ingress was never found.
Faced problems when they began to deploy our solution in k8s. The services were located inside the k8s cluster and the Spark StandAlone cluster outside the k8s. When starting the service from spark-driver, spark-submit is executed, which confirms the presence of an application on the UI. But on workers, we get an error that the workman could not connect to a random port to the spark-driver. The ports themselves have learned to override and specify, but these ports must be accessible from outside the cluster. We found a solution in which we open NodePort on workers - it works. BUT this is not suitable for most customers due to internal regulations. How to resolve the issue through ingress was never found. with best regards, Vitaly Kryukov
