Re: [Question] Apache Beam Spark Runner Support - Spark 3.5 Environment

I already added Spark 3.5.0 version to Beam Spark version tests [1] and I didn’t notice any regression. The next Beam release (2.53.0) should be available in a couple on months, depending on release preparation process. — Alexey [1] https://github.com/apache/beam/pull/29327 > On 9 Nov 2023,

RE: Re: [Question] Apache Beam Spark Runner Support - Spark 3.5 Environment

Thank you Alexey for the response. We are using Beam 2.41.0 with Spark 3.3.0 cluster. We did not run into any issues. is it because in Beam 2.41.0, compatibility tests were run against spark 3.3.0 ? https://github.com/apache/beam/blob/release-2.41.0/runners/spark/3/build.gradle If so, since co

Re: [Question] Apache Beam Spark Runner Support - Spark 3.5 Environment

AFAICT, the latest tested (compatibility tests) version for now is 3.4.1 [1] We may try to add 3.5.x version there. I believe that ValidateRunners tests are run only against default Spark 3.2.2 version. — Alexey [1] https://github.com/apache/beam/blob/2aaf09c0eb6928390d861ba228447338b8ca92d3

[Question] Apache Beam Spark Runner Support - Spark 3.5 Environment

Does Apache Beam version (2.41.0) or latest (2.51.0) support Spark 3.5 environment for spark runner ? Apache Beam - Spark Runner Documentation states - The Spark runner currently supports Spark’s 3.2.x branch Thanks Sri Ganesh V