Thanks Prabodh. I copied the spark connect jar to $SPARK_HOME/jars folder. And passed the location as —jars attr. Its working now. I could submit spark jobs via spark connect.
Really appreciate the help. Thanks, Elango On Tue, 30 Jul 2024 at 11:05 AM, Prabodh Agarwal <prabodh1...@gmail.com> wrote: > Yeah. I understand the problem. One of the ways is to actually place the > spark connect jar in the $SPARK_HOME/jars folder. That is how we run spark > connect. Using the `--packages` or the `--jars` option is flaky in case of > spark connect. > > You can instead manually place the relevant spark connect jar file in the > `$SPARK_HOME/jars` directory and remove the `--packages` or the `--jars` > option from your start command. > > On Mon, Jul 29, 2024 at 7:01 PM Ilango <elango...@gmail.com> wrote: > >> >> Thanks Prabodh, Yes I can see the spark connect logs in $SPARK_HOME/logs >> path. It seems like the spark connect dependency issue. My spark node is >> air gapped node so no internet is allowed. Can I download the spark connect >> jar and pom files locally and share the local paths? How can I share the >> local jars ? >> >> Error message: >> >> :: problems summary :: >> >> :::: WARNINGS >> >> module not found: >> org.apache.spark#spark-connect_2.12;3.5.1 >> >> >> >> ==== local-m2-cache: tried >> >> >> >> >> file:/root/.m2/repository/org/apache/spark/spark-connect_2.12/3.5.1/spark-connect_2.12-3.5.1.pom >> >> >> >> -- artifact >> org.apache.spark#spark-connect_2.12;3.5.1!spark-connect_2.12.jar: >> >> >> >> >> file:/root/.m2/repository/org/apache/spark/spark-connect_2.12/3.5.1/spark-connect_2.12-3.5.1.jar >> >> >> >> ==== local-ivy-cache: tried >> >> >> >> >> /root/.ivy2/local/org.apache.spark/spark-connect_2.12/3.5.1/ivys/ivy.xml >> >> >> >> -- artifact >> org.apache.spark#spark-connect_2.12;3.5.1!spark-connect_2.12.jar: >> >> >> >> >> /root/.ivy2/local/org.apache.spark/spark-connect_2.12/3.5.1/jars/spark-connect_2.12.jar >> >> >> >> ==== central: tried >> >> >> >> >> https://repo1.maven.org/maven2/org/apache/spark/spark-connect_2.12/3.5.1/spark-connect_2.12-3.5.1.pom >> >> >> >> -- artifact >> org.apache.spark#spark-connect_2.12;3.5.1!spark-connect_2.12.jar: >> >> >> >> >> https://repo1.maven.org/maven2/org/apache/spark/spark-connect_2.12/3.5.1/spark-connect_2.12-3.5.1.jar >> >> >> >> ==== spark-packages: tried >> >> >> >> >> https://repos.spark-packages.org/org/apache/spark/spark-connect_2.12/3.5.1/spark-connect_2.12-3.5.1.pom >> >> >> >> -- artifact >> org.apache.spark#spark-connect_2.12;3.5.1!spark-connect_2.12.jar: >> >> >> >> >> https://repos.spark-packages.org/org/apache/spark/spark-connect_2.12/3.5.1/spark-connect_2.12-3.5.1.jar >> >> >> >> :::::::::::::::::::::::::::::::::::::::::::::: >> >> >> >> :: UNRESOLVED DEPENDENCIES :: >> >> >> >> :::::::::::::::::::::::::::::::::::::::::::::: >> >> >> >> :: org.apache.spark#spark-connect_2.12;3.5.1: not found >> >> >> >> :::::::::::::::::::::::::::::::::::::::::::::: >> >> >> >> >> >> :::: ERRORS >> >> Server access error at url >> https://repo1.maven.org/maven2/org/apache/spark/spark-connect_2.12/3.5.1/spark-connect_2.12-3.5.1.pom >> (java.net.ConnectException: >> Connection timed out (Connection timed out)) >> >> >> >> Server access error at url >> https://repo1.maven.org/maven2/org/apache/spark/spark-connect_2.12/3.5.1/spark-connect_2.12-3.5.1.jar(java.net.ConnectException: >> Connection timed out (Connection timed out)) >> >> >> >> Server access error at url >> https://repos.spark-packages.org/org/apache/spark/spark-connect_2.12/3.5.1/spark-connect_2.12-3.5.1.pom >> (java.net.ConnectException: >> Connection timed out (Connection timed out)) >> >> >> >> Server access error at url >> https://repos.spark-packages.org/org/apache/spark/spark-connect_2.12/3.5.1/spark-connect_2.12-3.5.1.jar(java.net.ConnectException: >> Connection timed out (Connection timed out)) >> >> >> >> >> >> :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS >> >> Exception in thread "main" java.lang.RuntimeException: [unresolved >> dependency: org.apache.spark#spark-connect_2.12;3.5.1: not found] >> >> at >> org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1608) >> >> at >> org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185) >> >> at >> org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:334) >> >> at org.apache.spark.deploy.SparkSubmit.org >> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:964) >> >> at >> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194) >> >> at >> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217) >> >> at >> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) >> >> at >> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120) >> >> at >> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129) >> >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> >> >> >> >> Thanks, >> Elango >> >> >> On Mon, 29 Jul 2024 at 6:45 PM, Prabodh Agarwal <prabodh1...@gmail.com> >> wrote: >> >>> The spark connect startup prints the log location. Is that not feasible >>> for you? >>> For me log comes to $SPARK_HOME/logs >>> >>> On Mon, 29 Jul, 2024, 15:30 Ilango, <elango...@gmail.com> wrote: >>> >>>> >>>> Hi all, >>>> >>>> >>>> I am facing issues with a Spark Connect application running on a Spark >>>> standalone cluster (without YARN and HDFS). After executing the >>>> start-connect-server.sh script with the specified packages, I observe a >>>> process ID for a short period but am unable to see the corresponding port >>>> (default 15002) associated with that PID. The process automatically stops >>>> after around 10 minutes. >>>> >>>> Since the Spark History server is not enabled, I am unable to locate >>>> the relevant logs or error messages. The logs for currently running Spark >>>> applications are accessible from the Spark UI, but I am unsure where to >>>> find the logs for the Spark Connect application and service. >>>> >>>> Could you please advise on where to find the logs or error messages >>>> related to Spark Connect? >>>> >>>> >>>> >>>> >>>> Thanks, >>>> Elango >>>> >>>
