While doing some extra testing and investigating another issue, I found that the predicate pushdown is not working for MERGE operations in Spark 3.2 as we fail to extract the required filters. It only affects Spark 3.2 and runtime filtering is there but it will have a serious performance impact for targeted MERGE operations. I think we may want to release 0.13.1 rather quickly.
Here is the PR to fix this. https://github.com/apache/iceberg/pull/4023 <https://github.com/apache/iceberg/pull/4023> Sorry I did not catch it earlier. - Anton > On 31 Jan 2022, at 14:15, Jack Ye <yezhao...@gmail.com> wrote: > > Thanks to everyone who participated in the vote for Release Apache Iceberg > 0.13.0 RC2. > > The vote result is: > > +1: 4 (binding), 7 (non-binding) > +0: 0 (binding), 0 (non-binding) > -1: 0 (binding), 0 (non-binding) > > Therefore, the release candidate is passed. > > Cheers, > Jack Ye > > On Mon, Jan 31, 2022 at 2:07 PM Jack Ye <yezhao...@gmail.com > <mailto:yezhao...@gmail.com>> wrote: > +1 (binding) > > * checked signature, checksum > * Ran unit test, AWS integration test > * Tested basic table operations and RewriteDataFiles on AWS EMR Spark 3.0 and > 3.1 environments > > Best, > Jack Ye > > On Mon, Jan 31, 2022 at 1:52 PM Ryan Blue <b...@tabular.io > <mailto:b...@tabular.io>> wrote: > +1 (binding) > > * Checked signature, checksums, ran RAT > * Ran tests > * Tested Spark 3.2 CREATE, INSERT, and INSERT OVERWRITE (same table) > * Tested merge-on-read with v1 table > * Tested update to v2 > * Tested DELETE FROM with merge-on-read > * Tested rewrite_data_files to merge deletes > * Tested adding a partition field and rewriting > * Tested metadata column projection > * Tested custom catalogs > > Everything looks great! > > There was a warning about ANTLR runtime version 4.7.1 instead of 4.8, but I > don't think that's a blocker. > > On Mon, Jan 31, 2022 at 12:54 PM Yufei Gu <flyrain...@gmail.com > <mailto:flyrain...@gmail.com>> wrote: > +1 non-binding > > 1. Verify the signature, checksum > 2. All tests passed. > > Best, > > Yufei > > > On Mon, Jan 31, 2022 at 9:45 AM Daniel Weeks <dwe...@apache.org > <mailto:dwe...@apache.org>> wrote: > +1 (binding) > > Verified sigs/sums/license/build/tests. > > I ran some basic tests using spark 3.2 with the new runtime and everything > worked as expected. > > -Dan > > On Sun, Jan 30, 2022 at 5:53 PM Kyle Bendickson <kjbendick...@gmail.com > <mailto:kjbendick...@gmail.com>> wrote: > +1 (non-binding) > > Verified signature, checksum, rat check, build and ran tests, and tested the > relevant JAR on both Spark 3.1 and 3.2. > > - Kyle > > On Sun, Jan 30, 2022 at 12:45 AM Szehon Ho <szehon.apa...@gmail.com > <mailto:szehon.apa...@gmail.com>> wrote: > +1 (non-binding) > > Verified signature > Verified checksum > Rat check > Built and ran test, all succeed, after some temporary local HMS timeout > Tested relevant jar with Spark 3.2, created various tables and ran queries > > Thanks > Szehon > > On Fri, Jan 28, 2022 at 12:19 PM Russell Spitzer <russell.spit...@gmail.com > <mailto:russell.spit...@gmail.com>> wrote: > +1 > All tests passed for me and signatures, checksum and license all were good to > go > > On Fri, Jan 28, 2022 at 12:32 PM John Zhuge <jzh...@apache.org > <mailto:jzh...@apache.org>> wrote: > +1 (non-binding) > > Checked signature, checksum, and license. > Ran build and test with OpenJDK 1.8.0_312-b07. > > Ignoring the mr test failures. Maybe my env is not set up correctly? > 19 in TestHiveIcebergStorageHandlerWithEngine > 3 in TestHiveIcebergStorageHandlerWithMultipleCatalogs > > On Fri, Jan 28, 2022 at 8:41 AM Jack Ye <yezhao...@gmail.com > <mailto:yezhao...@gmail.com>> wrote: > Hi Everyone, > > I propose that we release the following RC as the official Apache Iceberg > 0.13.0 release. > > The commit ID is 72237429ba164c054480dcfbdb9fe1c86c04dcda > * This corresponds to the tag: apache-iceberg-0.13.0-rc2 > * https://github.com/apache/iceberg/commits/apache-iceberg-0.13.0-rc2 > <https://github.com/apache/iceberg/commits/apache-iceberg-0.13.0-rc2> > * > https://github.com/apache/iceberg/tree/72237429ba164c054480dcfbdb9fe1c86c04dcda > > <https://github.com/apache/iceberg/tree/72237429ba164c054480dcfbdb9fe1c86c04dcda> > > The release tarball, signature, and checksums are here: > * https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.0-rc2 > <https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.0-rc2> > > You can find the KEYS file here: > * https://dist.apache.org/repos/dist/dev/iceberg/KEYS > <https://dist.apache.org/repos/dist/dev/iceberg/KEYS> > > Convenience binary artifacts are staged on Nexus. The Maven repository URL is: > * https://repository.apache.org/content/repositories/orgapacheiceberg-1080/ > <https://repository.apache.org/content/repositories/orgapacheiceberg-1080/> > > Please download, verify, and test. > > Please vote in the next 72 hours. > > [ ] +1 Release this as Apache Iceberg 0.13.0 > [ ] +0 > [ ] -1 Do not release this because... > > > -- > John Zhuge > > > -- > Ryan Blue > Tabular
