Thanks for bringing this subject Michael, we have vastly improved our communication with downstream projects (Spark/Parquet/Hive) and most of the issues you and other users have had will be available soon in upcoming versions of those projects.
Spark finally merged the upgrade to Avro 1.10.1 recently [1], the Parquet upgrade will be there too. Dongjoon (one Spark committer) has been contributing to the zstd upgrades in the three projects (Avro/Parquet/Spark) so we should be good to go. Actually the 1.10.2 release is motivated in part because of the needs of Spark [2] I expect to open the PR with the Avro upgrade once published so it will be included in Spark 3.2. Regards, Ismaël [1] https://github.com/apache/spark/pull/31232 [2] https://github.com/apache/avro/pull/1129 On Wed, Mar 10, 2021 at 7:28 PM Michael Heuer <[email protected]> wrote: > > Hello, > > The zstd-jni dependency version has been moving quickly lately, and there are > API changes (although hopefully all compatible). Any worry about version > conflicts between Avro, Parquet, Spark, commons-compress, and other projects > that all have a dependency on zstd-jni? > > I would test this RC with our stack, but we're still stuck on Avro 1.8.2 due > to other version conflicts between Avro and Parquet in Spark (which I believe > have been resolved in the master branch towards a future Spark 3.2 release). > > Sorry I can't be more help other than raising a possible concern. > > michael > > > > On Mar 10, 2021, at 12:17 PM, Ryan Skraba <[email protected]> wrote: > > > > Hello! > > > > I finished some validation and I'm pleased to add my own vote on the > > release candidate! > > > > I checked the Avro 1.10.2-rc2 artifacts by: > > > > * Running a large set of internal Java code and unit tests (already on > > 1.10) with these candidates. > > * Building parquet-mr with the candidates. > > * We have some internal python code and unit tests that I ran on versions > > python3.5 - python3.9. > > * Checked that all of the artifacts were signed and verified my own > > signature (FWIW) > > * Checked that the artifacts in the staging nexus were signed by my own > > signature (FWIW) > > > > It looks good for me! > > > > [X] +1 Release this as Apache Avro 1.10.2 > > [ ] +0 > > [ ] -1 Do not release this because... > > > > All my best, Ryan > > > > > > On Tue, Mar 9, 2021 at 6:55 PM Ryan Skraba <[email protected]> wrote: > > > >> Hi everyone, > >> > >> I'd like to propose the following RC2 to be released as the official Apache > >> Avro 1.10.2 release. > >> > >> There have been two commits since RC1, addressing the following issues: > >> - [AVRO-3072] ZSTD Performance improvements > >> - [AVRO-3073] Potential vulnerability with velocity > >> > >> The commit id is 8111cdc35430ff68dcb644306362859de40999d9 > >> * This corresponds to the tag: release-1.10.2-rc2 > >> * https://github.com/apache/avro/releases/tag/release-1.10.2-rc2 > >> > >> The release tarball, signature, and checksums are here (revision 46543): > >> * https://dist.apache.org/repos/dist/dev/avro/avro-1.10.2-rc2/ > >> > >> You can find the KEYS file here: > >> * https://dist.apache.org/repos/dist/dev/avro/KEYS > >> > >> Binary artifacts for Java are staged in Nexus here: > >> * > >> https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.10.2/ > >> > >> This release includes ~30 Jira issues: > >> * > >> https://jira.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20fixVersion%20%3D%201.10.2 > >> > >> Some interesting highlights: > >> > >> Avro specification > >> - [AVRO-3028] Clarify that records encode values even if they equal their > >> default > >> > >> C# > >> - [AVRO-3005][AVRO-2983] BinaryDecoder fails to read large strings > >> > >> C++ > >> - [AVRO-3031] avrocppgen does not generate correct C++ code when the > >> schema contains > >> C++ reserved words > >> > >> Java > >> - [AVRO-2471] Java code generation doesn't add conversion for > >> timestamp-micros > >> - [AVRO-2860] More Closely Adhere to ASF Parent POM > >> - [AVRO-2944] DataFileReader has incorrect logic reading magic header > >> - [AVRO-3024] Bump Jackson to 2.12.1 > >> - [AVRO-3060] Support ZSTD level and BufferPool options > >> - [AVRO-3049] BinaryDecoder lacks checks on bytes array length > >> - [AVRO-3072] ZSTD Performance improvements > >> > >> Python > >> - [AVRO-3006] Update PyPI documentation to deprecate avro-python3 > >> *** The avro package supports Python 3, and avro-python3 will be removed > >> in > >> the next major release *** > >> > >> Ruby > >> - [AVRO-2984] Unnecessary memory allocations during serialization > >> - [AVRO-2998] Records with symbol keys fail validation > >> - [AVRO-2999] Optimize Ruby union serialization > >> - [AVRO-3000] Avoid unnecessary schema compatibility checks > >> - [AVRO-3023] Validate with Ruby 3 > >> > >> * Upgrade dependencies to latest versions, including CVE fixes. > >> * Multiple fixes, better documentation and more... > >> > >> Avro 1.10 is still using Travis, but the status isn't necessarily > >> reflected on the branch in github: > >> * https://travis-ci.com/github/apache/avro/builds/219514042 > >> > >> Please download, verify, and test. This vote will remain open for at least > >> 72 hours. Given sufficient votes, I would like to close after the weekend > >> on > >> 18h00 UTC Friday, March 12th, 2021 > >> > >> [ ] +1 Release this as Apache Avro 1.10.2 > >> [ ] +0 > >> [ ] -1 Do not release this because... > >> > >> Best regards, > >> Ryan Skraba > >> >
