Thanx Cheng Pan for sharing the pointers, Do you have any list of issues or pointers on what are the challenges for Spark to move to a higher Hive version? I know upgrading libraries is quite challenging but it is inevitable.
Hive is already in discussion of marking Hive-2.x EOL, so at very best we would have one release and immediately after that we will announce it EOL, maintaining a release line is quite an effort for us at Hive & doing it because other projects doesn't want to upgrade isn't a convincing reason for most of us. The best we can do is or are trying is to address issues for Spark whatever we can do as part of Hive code & would definitely need help/support from Spark side as well, since the move is from 2.x to 4.x, it would be a big change and would offer resistance on both sides. So, it would be great help if any pointers can be shared from Spark side for the move, if there is no help/interest from Spark then we can't do anything & there is no need for Hive-2.x either in that case :-) -Ayush On Wed, 17 Apr 2024 at 15:00, Cheng Pan <pan3...@gmail.com> wrote: > > … we are exploring ways to get Spark move from 2.3.9 to 4.0, Our initial > hunch is that it would be quite challenging without a hive-exec slim jar … > > It should be challenging to upgrade Spark’s built-in Hive version. > Actually, we already did lots of work on branch-2.3 which focuses on CVE > reduction, for example, allowing Spark to upgrade Guava to modern versions > to get rid of Guava 14, it was tested with the latest Spark master > branch[1], maybe we need a release for 2.3.10 now. > > [1] https://github.com/apache/spark/pull/45372 > > Thanks, > Cheng Pan > > >
