Thanks Nandor, Upgrading Hive 2.x (Hadoop 2) is going to be painful since this is still on Java 1.7. It is still relying on Avro 1.7.7. This is an issue because this is also blocking upgrading Parquet, this implies that Hive 2.x does not support reading lz4 and zstd compression.
>From Hive 3.x the Java version is 1.8 and we could upgrade Avro. The current version is 1.8.2, so that should be doable. Cheers, Fokko Op wo 15 mei 2019 om 23:51 schreef Nandor Kollar <nkol...@cloudera.com.invalid>: > If Spark is upgraded, then I'm afraid first Hive should be upgraded to > 1.9.0. When I did a quick release verification, I remember I saw several > failures due to Hive using deprecated and removed Avro methods (related to > Jackson removal from public API). > > On Wed, May 15, 2019 at 11:04 PM Ismaël Mejía <ieme...@gmail.com> wrote: > > > Getting a new release of Avro feels amazing, thanks a lot Fokko for > > all your wokr to get this out with the release. > > > > An interesting next step is downstreaming. I saw Fokko created a > > ticket for Parquet, is anyone working on the Spark one? It will be > > really nice to get this into Spark 3. Anything I can do to help there > > or that we can sync work? > > I filled https://issues.apache.org/jira/browse/SPARK-27733 for the > moment. > > > > I am tackling the upgrade on Beam (the other project I work mostly on > too. > > https://issues.apache.org/jira/browse/BEAM-7328 > > > > Any other project worth the sync/work? > > > > > > On Wed, May 15, 2019 at 8:54 PM Driesprong, Fokko <fo...@driesprong.frl> > > wrote: > > > > > > Hi Jacob, > > > > > > This looks off, I'll dive into it. I think it got released for some > > reason. > > > While releasing it today, I also noticed that the artifact wasn't > staged > > in > > > Nexus anymore. However, RC4 is the same as the release. > > > > > > Cheers, Fokko > > > > > > Op wo 15 mei 2019 om 20:27 schreef Jacob Tolar > > > <jto...@verizonmedia.com.invalid>: > > > > > > > Great news! We have been looking forward to the 1.9 release. > > > > > > > > I see 1.9.0 published on Maven Central. It appears to have been > > published > > > > 5/8 > > > > (the same day as RC4 was announced but ~1week before vote passed). > > > > > > > > Is this the release build (presumably the same as RC4) or an earlier > > RC? > > > > > > > > https://repo1.maven.org/maven2/org/apache/avro/avro/1.9.0/ > > > > > > > > Would it be helpful to have maven include the git sha somewhere in > > > > MANIFEST.MF? > > > > > > > > jacob > > > > > > > > On Tue, May 14, 2019 at 1:04 PM Driesprong, Fokko > <fo...@driesprong.frl > > > > > > > wrote: > > > > > > > > > Thank you all, > > > > > > > > > > I'm happy to announce, the vote for Avro 1.9.0 has passed. > > > > > > > > > > Binding: > > > > > +1 Daniel Kulp > > > > > +1 Sean Busbey > > > > > +1 Fokko Driesprong > > > > > > > > > > Non-binding: > > > > > +1 Lunjie Jin > > > > > +1 Michael A. Smith > > > > > +1 Ismaël Mejía > > > > > +1 Brian Lachniet > > > > > > > > > > I would like to thank the community from participating in the > > release. > > > > Both > > > > > in providing patches for Apache Avro and giving the RC's a try. > > > > > > > > > > Yesterday I did some additional testing against Apache Spark > > (similar to > > > > > Nandor :-) and Apache Flink and did not find any blocking issues. > > There > > > > are > > > > > some incompatibilities, but most of them are; revoking Jackson from > > the > > > > > public API and deprecating Joda. > > > > > > > > > > I'll publish the artifacts to the different repositories soon. On > > this, I > > > > > consider myself a polyglot programmer, but my knowledge is limited > on > > > > .Net, > > > > > Perl, and Javascript, so I might ask some help from the community > > here as > > > > > well. > > > > > > > > > > Thanks all, and keep up the good work. > > > > > > > > > > Cheers, Fokko Driesprong > > > > > > > > > > Op di 14 mei 2019 om 14:48 schreef Nandor Kollar > > > > > <nkol...@cloudera.com.invalid>: > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > * verified signatures and checksums > > > > > > * ran unit tests, passed > > > > > > * did a quick sanity test against Spark master: upgraded Spark > Avro > > > > > > dependency to this release candidate, ran spark-avro tests, all > > passed > > > > > (for > > > > > > the record, two minor Maven changes are required due to removal > of > > > > > Hadoop 1 > > > > > > support and making org.tukaani.xz provided dependency - looks > like > > > > Spark > > > > > > tests against each codec). spark-hive tests failed, because > > deprecated > > > > > > methods were removed in Avro (related to Jackson removal from > > public > > > > API, > > > > > > and Hive uses them), but I think it is expected, upgrading to a > > major > > > > > > release could have breaking changes. > > > > > > > > > > > > Nandor > > > > > > > > > > > > On Sun, May 12, 2019 at 7:11 AM Sean Busbey <bus...@apache.org> > > wrote: > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > -- Good: > > > > > > > * signatures > > > > > > > * checksums > > > > > > > * tag/ref lines up with src artifact (except for one file > > mentioned > > > > > > below) > > > > > > > * LICENSE/NOTICE spot check > > > > > > > * apache rat says the license files for all the artifacts are > > fine > > > > > > (except > > > > > > > it couldn't understand the gem; using the gem cli confirms it's > > also > > > > > > > correct) > > > > > > > > > > > > > > -- We Can Be Better Later: > > > > > > > * filed AVRO-2395 because the "java" convenience binary part of > > the > > > > > dist > > > > > > > section is redundant > > > > > > > * in the future please post the specific staged maven > repository > > > > > instead > > > > > > > of somewhere within the staged repository group. that'll help > us > > > > avoid > > > > > > > possible conflicts should someone forget to cancel a staged RC > or > > > > > > > accidentally stage an additional repository after the vote is > > called. > > > > > > FWIW > > > > > > > as far as I can tell from the Nexus UI, this is the staged repo > > for > > > > > this > > > > > > > VOTE (and it's what I verified): > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheavro-1020/ > > > > > > > > > > > > > > * When unpacking the source tarball I got this warning on OSX. > > > > > > > > > > > > > > > Busbey-MBA:1.9.0-RC4 busbey$ tar -C src_untar -xzf > > > > > > > > > > > > > > dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc4/avro-src-1.9.0.tar.gz > > > > > > > > tar: copyfile unpack > > > > > > > > > > > > > > > > > > > > > > > > > (avro-src-1.9.0/lang/java/mapred/src/test/resources/org/apache/avro/mapreduce/mapreduce-test-input.avro/SUCCESS.crc) > > > > > > > failed: No such file or directory > > > > > > > > > > > > > > I don't think it should be a blocker because it's shown up in > > prior > > > > RCs > > > > > > > since 2012 and AFAICT things are fine despite it, save a unit > > test. > > > > > > > > > > > > > > * An upgrade guide will help a bunch of folks given the time > > since > > > > last > > > > > > > release and this being a major version. > > > > > > > > > > > > > > For example, someone asked about how incompatible things are. > So > > I > > > > ran > > > > > > the > > > > > > > Java API Compliance Checker on the java libraries, since it's > > > > > > (relatively) > > > > > > > easy. After filtering out the "avro.shaded" package and > > excluding the > > > > > > > hadoop1 specific jars from 1.8.2: > > > > > > > > > > > > > > > Busbey-MBA:1.9.0-RC4 busbey$ japi-compliance-checker -l > "apache > > > > avro" > > > > > > > -d1 avro-1.8.2-jacc.xml -d2 avro-1.9.0-jacc.xml -skip-packages > > > > > > > skip-packages.txt > > > > > > > > Preparing, please wait ... > > > > > > > > Using Java 1.8.0_161 > > > > > > > > Reading classes 1.8.2 ... > > > > > > > > WARNING: skipping "internal" packages > > > > > > > > NOTE: use --keep-internal option to check them > > > > > > > > Reading classes 1.9.0-rc4 ... > > > > > > > > WARNING: skipping "internal" packages > > > > > > > > Comparing classes ... > > > > > > > > Creating compatibility report ... > > > > > > > > Binary compatibility: 94.6% > > > > > > > > Source compatibility: 93.6% > > > > > > > > Total binary compatibility problems: 155, warnings: 26 > > > > > > > > Total source compatibility problems: 160, warnings: 4 > > > > > > > > Report: compat_reports/apache > > > > > > avro/1.8.2_to_1.9.0-rc4/compat_report.html > > > > > > > > > > > > > > The report is here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://people.apache.org/~busbey/avro/1.9.0-RC4/1.8.2_to_1.9.0RC4_compat_report.html > > > > > > > > > > > > > > This is a major version, so I think breaking things is fine. > But > > this > > > > > > > report is still pretty long and so I'm guessing downstream > could > > use > > > > > some > > > > > > > help. Maybe the current summary list covers all of the breakage > > > > > listed; I > > > > > > > didn't try to compare them. > > > > > > > > > > > > > > On 2019/05/08 19:45:15, "Driesprong, Fokko" > <fo...@driesprong.frl > > > > > > > > > wrote: > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > Since the last release of Apache Avro 1.8.2 on May 31, 2017. > > Two > > > > > years > > > > > > > > later, > > > > > > > > I'm thrilled to propose the following RC to be released as > > official > > > > > > > Apache > > > > > > > > Avro 1.9.0 release. > > > > > > > > > > > > > > > > The commit id is 3c76495e9524ef322726d03d7ee406be89e8fde0 > > > > > > > > * This corresponds to the tag: release-1.9.0-rc4 > > > > > > > > * > > https://github.com/apache/avro/releases/tag/release-1.9.0-rc4 > > > > > > > > > > > > > > > > The release tarball, signature, and checksums are here: > > > > > > > > * > https://dist.apache.org/repos/dist/dev/avro/avro-1.9.0-rc4/ > > > > > > > > > > > > > > > > You can find the KEYS file here: > > > > > > > > * https://dist.apache.org/repos/dist/dev/avro/KEYS > > > > > > > > > > > > > > > > Binary artifacts for Java are staged in Nexus here: > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.9.0/ > > > > > > > > > > > > > > > > This release includes 272 Jira issues: > > > > > > > > > https://issues.apache.org/jira/projects/AVRO/versions/12333394 > > > > > > > > * Deprecate Joda-Time in favor of Java8 JSR310 and setting it > > as > > > > > > default > > > > > > > > * Remove support for Hadoop 1.x > > > > > > > > * Move from Jackson 1.x to 2.9 > > > > > > > > * Add ZStandard Codec > > > > > > > > * Lots of updates on the dependencies to fix CVE's > > > > > > > > * Remove Jackson classes from public API > > > > > > > > * Apache Avro is built by default with Java 8 > > > > > > > > * Apache Avro is compiled and tested with Java 11 to > guarantee > > > > > > > compatibility > > > > > > > > * Apache Avro MapReduce is compiled and tested with Hadoop 3 > > > > > > > > * Apache Avro is now leaner, multiple dependencies were > > removed: > > > > > guava, > > > > > > > > paranamer, commons-codec, and commons-logging > > > > > > > > * Introduce JMH Performance Testing Framework > > > > > > > > * Add Snappy support for C++ DataFile > > > > > > > > * and many, many more! > > > > > > > > > > > > > > > > Since RC1, two commits have been added: > > > > > > > > * https://jira.apache.org/jira/browse/AVRO-2381 > > > > > > > > * https://jira.apache.org/jira/browse/AVRO-2383 > > > > > > > > > > > > > > > > Since RC2: The SHA1/MD5 checksums have been replaced with > > SHA512 > > > > > > > > > > > > > > > > Since RC3: > > > > > > > > * Regression failure, the customEncode methods are public > > again. > > > > > > > > * The release tarball does not contain snapshot anymore > > > > > > > > > > > > > > > > Please download, verify, and test. This vote will remain open > > for > > > > at > > > > > > > least > > > > > > > > 72 hours. Given sufficient votes, I would like to close it on > > or > > > > > about > > > > > > > > midnight > > > > > > > > on Saturday, 11th of May 2019. > > > > > > > > > > > > > > > > [ ] +1 Release this as Apache Avro 1.9.0 > > > > > > > > [ ] +0 > > > > > > > > [ ] -1 Do not release this because... > > > > > > > > > > > > > > > > Consider this a +1 (non-binding) from my side: > > > > > > > > * Compiled the new version of Parquet against the Divolte > > collector > > > > > and > > > > > > > > Apache Parquet > > > > > > > > > > > > > > > > Cheers, Fokko Driesprong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
