Hi everyone, For the moment, Beam Spark Runner supports two versions of Spark - 2.x and 3.x.
Taking into account the several things that: - almost all cloud providers already mostly moved to Spark 3.x as a main supported version; - the latest Spark 2.x release (Spark 2.4.8, maintenance release) was done almost a year ago; - Spark 3 is considered as a mainstream Spark version for development and bug fixing; - better to avoid the burden of maintenance (there are some incompatibilities between Spark 2 and 3) of two versions; I’d suggest to stop support Spark 2 for the Spark Runner in the one of the next Beam releases. What are your thoughts on this? Are there any principal objections or reasons for not doing this that I probably missed? — Alexey
