[ 
https://issues.apache.org/jira/browse/SPARK-51603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17939118#comment-17939118
 ] 

Yang Jie edited comment on SPARK-51603 at 3/28/25 3:22 AM:
-----------------------------------------------------------

Regarding the issue described in Jira, here are the possible solutions:

1. Try to create a new module named `connect-jvm-integration-testing` and move 
the tests related to `RemoteSparkSession` into this module. At the same time, 
make the `connect-jvm-integration-testing` module dependent on 
`connect-client-jvm`, `connect`, and `assembly`. The feasibility of this 
solution remains to be verified.

2. As I suggested in the pull request at 
[https://github.com/apache/spark/pull/50396], automatically skip the tests 
related to `RemoteSparkSession` when the directory 
`assembly/target/scala-2.13/jars/` does not exist.

[~gurwls223]  WDYT? Additionally, do you think this issue could potentially 
block the release of version 4.0(Personally, I think it is)?


was (Author: luciferyang):
Regarding the issue described in Jira, here are the possible solutions:

1. Try to create a new module named `connect-jvm-integration-testing` and move 
the tests related to `RemoteSparkSession` into this module. At the same time, 
make the `connect-jvm-integration-testing` module dependent on 
`connect-client-jvm`, `connect`, and `assembly`. The feasibility of this 
solution remains to be verified.

2. As I suggested in the pull request at 
https://github.com/apache/spark/pull/50396, automatically skip the tests 
related to `RemoteSparkSession` when the directory 
`assembly/target/scala-2.13/jars/` does not exist.

[~gurwls223]  WDYT? Additionally, do you think this issue could potentially 
block the release of version 4.0?

> logical circular dependency between modules `connect-client-jvm` and 
> `assembly`
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-51603
>                 URL: https://issues.apache.org/jira/browse/SPARK-51603
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect, Tests
>    Affects Versions: 4.0.0
>            Reporter: Yang Jie
>            Priority: Major
>              Labels: pull-request-available
>
> After SPARK-48936, the `assembly` module is compiled prior to 
> `connect-client-jvm` to facilitate the packaging output. However, tests 
> related to RemoteSparkSession in the `connect-client-jvm` module depend on 
> the packaged results from the `assembly` module, specifically the JARs 
> located in `assembly/target/scala-2.13/jars/`. Since the latter is a 
> convention, executing 
>  
> {code:java}
> build/mvn clean install -DskipTests {code}
>  
> will not produce any errors. But if we directly execute 
> {code:java}
> build/mvn clean install {code}
> we will encounter test errors similar to the following when testing the 
> `connect-client-jvm` module:
>  
> {code:java}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.044 
> s <<< FAILURE! – in org.apache.spark.sql.JavaEncoderSuite
> [ERROR] org.apache.spark.sql.JavaEncoderSuite – Time elapsed: 0.044 s <<< 
> FAILURE!
> java.lang.AssertionError: assertion failed: Fail to locate the target folder: 
> '/Users/yangjie01/SourceCode/git/spark-maven/sql/connect/server/target'. 
> SPARK_HOME='/Users/yangjie01/SourceCode/git/spark-maven'. Make sure the spark 
> project jars has been built (e.g. using build/sbt package)and the env 
> variable `SPARK_HOME` is set correctly.
>   at scala.Predef$.assert(Predef.scala:279)
>   at 
> org.apache.spark.sql.connect.test.IntegrationTestUtils$.tryFindJar(IntegrationTestUtils.scala:138)
>   at 
> org.apache.spark.sql.connect.test.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:116)
>   at 
> org.apache.spark.sql.connect.test.SparkConnectServerUtils$.sparkConnect$lzycompute(RemoteSparkSession.scala:65)
>   at 
> org.apache.spark.sql.connect.test.SparkConnectServerUtils$.sparkConnect(RemoteSparkSession.scala:62)
>   at 
> org.apache.spark.sql.connect.test.SparkConnectServerUtils$.start(RemoteSparkSession.scala:135)
>   at 
> org.apache.spark.sql.connect.test.SparkConnectServerUtils$.createSparkSession(RemoteSparkSession.scala:181)
>   at 
> org.apache.spark.sql.connect.test.SparkConnectServerUtils.createSparkSession(RemoteSparkSession.scala)
>   at org.apache.spark.sql.JavaEncoderSuite.setup(JavaEncoderSuite.java:42)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:569)
>   at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
>   Suppressed: java.lang.NullPointerException: Cannot invoke 
> "org.apache.spark.sql.SparkSession.stop()" because 
> "org.apache.spark.sql.JavaEncoderSuite.spark" is null
>     at 
> org.apache.spark.sql.JavaEncoderSuite.tearDown(JavaEncoderSuite.java:47)
>     at java.base/java.lang.reflect.Method.invoke(Method.java:569)
>     at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
>     at 
> java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1092)
>     ... 1 more
> [INFO] 
> [INFO] Results:
> [INFO] 
> [ERROR] Failures: 
> [ERROR]   JavaEncoderSuite.setup:42 assertion failed: Fail to locate the 
> target folder: 
> '/Users/yangjie01/SourceCode/git/spark-maven/sql/connect/server/target'. 
> SPARK_HOME='/Users/yangjie01/SourceCode/git/spark-maven'. Make sure the spark 
> project jars has been built (e.g. using build/sbt package)and the env 
> variable `SPARK_HOME` is set correctly.
> [INFO] 
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to