Hello team,
We are in the process of upgrading one of our apps to Spring Boot 3.x while
using Spark, and we have encountered an issue with Spark compatibility,
specifically with Jakarta Servlet. Spring Boot 3.x uses Jakarta Servlet
while Spark uses Javax Servlet. Can we get some guidance on how to upgrade
to Spring Boot 3.x while continuing to use Spark.
The specific error is listed below:
java.lang.NoClassDefFoundError: javax/servlet/Servlet
at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:239)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:503)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
The error comes up when we try to run a mvn clean install, and the
issue is in our test cases. This issue happens specifically when we
build our spark session. The line of code it traces down to is as
follows:
*session =
SparkSession.builder().sparkContext(SparkContext.getOrCreate(sparkConf)).getOrCreate();*
What we have tried:
- We noticed according to this post
<https://stackoverflow.com/questions/75690210/apache-spark-with-spring-boot-failed-to-start-exception-factory-method-javasp>,
there are no compatible versions of spark using version 5 of the
Jakarta Servlet API
- We've tried
<https://stackoverflow.com/questions/76618374/spark-3-4-1-jakarta-servlet-6-0-0-compatibility-issue>
using the maven shade plugin to use jakarta instead of javax, but are
running into some other issues with this.
- We've also looked at the following
<https://stackoverflow.com/questions/75350944/dependecy-conflict-apache-spark-and-spring-boot>
to use jakarta 4.x with jersey 2.x and still have an issue with the
servlet
Please let us know if there are any solutions to this issue. Thanks!
--
*Ahmed Albalawi*
Senior Associate Software Engineer • EP2 Tech - CuRE
571-668-3911 • 1680 Capital One Dr.
______________________________________________________________________
The information contained in this e-mail may be confidential and/or proprietary
to Capital One and/or its affiliates and may only be used solely in performance
of work or services for Capital One. The information transmitted herewith is
intended only for use by the individual or entity to which it is addressed. If
the reader of this message is not the intended recipient, you are hereby
notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error, please
contact the sender and delete the material from your computer.
