[GitHub] [zeppelin] jongyoul commented on a diff in pull request #4398: [ZEPPELIN-5760] fix passing configs to spark in k8s

[GitBox]

jongyoul commented on code in PR #4398:
URL: https://github.com/apache/zeppelin/pull/4398#discussion_r929623223


##########
zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java:
##########
@@ -319,9 +321,25 @@ Properties getTemplateBindings(String userName) {
     if (isSpark()) {
       int webUiPort = 4040;
       k8sProperties.put("zeppelin.k8s.spark.container.image", sparkImage);
+
+      // There is already initial value following --driver-java-options added 
in interpreter.sh
+      // so we need to pass spark.driver.defaultJavaOptions and 
spark.driver.extraJavaOptions
+      // as SPARK_DRIVER_EXTRAJAVAOPTIONS_CONF env variable to build 
spark-submit command correctly.
+      StringBuilder driverExtraJavaOpts = new StringBuilder();
+      if (properties.containsKey(SPARK_DRIVER_DEFAULTJAVAOPTS)) {
+        driverExtraJavaOpts.append((String) 
properties.remove(SPARK_DRIVER_DEFAULTJAVAOPTS));
+      }
+      if (properties.containsKey(SPARK_DRIVER_EXTRAJAVAOPTS)) {
+        driverExtraJavaOpts.append(" ");
+        driverExtraJavaOpts.append((String) 
properties.remove(SPARK_DRIVER_EXTRAJAVAOPTS));
+      }
+      if (driverExtraJavaOpts.length() > 0) {
+        k8sEnv.put("SPARK_DRIVER_EXTRAJAVAOPTIONS_CONF", 
driverExtraJavaOpts.toString());
+      }
+
       if (isSparkOnKubernetes(properties)) {
-        k8sEnv.put("SPARK_SUBMIT_OPTIONS",
-            getEnv().getOrDefault("SPARK_SUBMIT_OPTIONS", "") + 
buildSparkSubmitOptions(userName));
+        this.addSparkK8sProperties();

Review Comment:
   ```suggestion
           addSparkK8sProperties();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@zeppelin.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org