Bumping this again, to ask please some of you guys if there is any chance someone can help to get the version of Spark upgraded so the Avro upgrade tests pass. Or give me info if there is another workaround to make this happen.
On Fri, Dec 6, 2019 at 4:44 PM Ismaël Mejía <ieme...@gmail.com> wrote: > Hello, just pinging again to see if anyone can help me with this. > Just to give some more context, I am trying to upgrade Avro to version > 1.9.1 to tackle some security issues in the dependencies as well as to get > some of the improvements of Avro since the latest release that happened 2y > ago. > > An extra goal is to get this cherry picked into the 2.x branch (if > possible) so we can get the upgrade done in Spark 3 too. The Spark upgrade > is partially blocked by Hive leaking an older version of Avro and an unlock > that too. > > > On Tue, Nov 26, 2019 at 3:26 PM Ismaël Mejía <ieme...@gmail.com> wrote: > >> Hello, >> >> I am trying to upgrade the Avro dependency on Hive to version 1.9.1. More >> details in >> https://issues.apache.org/jira/browse/HIVE-21737 >> >> The Hive CI tests for my patch failed because of some class issue then I >> tried to tackle this by upgrading the Spark version installed during the IT >> tests from 2.3.0 to version 2.4.4. However I discovered that the Jenkins >> job was failing to download the Spark version from some AWS storage bucket >> >> https://github.com/apache/hive/blob/209096a71b25a3d75dca1930f825c8c10b99d2b9/itests/bin/download-spark.sh#L37 >> >> It seems this version is somehow a different from upstream Spark. Does >> anyone know more about why we use that version? And more important, any >> chance this version can be upgraded to version 2.4.x (with the hope this >> may unblock my upgrade issue). >> >> Regards, >> Ismaël >> >>
