Yeah, I agree. We should fix this for the 0.12.0 release. That said, I plan to continue testing this RC because it won't change that much since this affects the Spark extensions in 3.1. Other engines and Spark 3.0 or older should be fine.
I left a comment on the PR. I think it looks good, but we should try to refactor to make sure we don't have more issues like this. I think when we update our extensions to be compatible with multiple Spark versions, we should introduce a factory method to create the Catalyst plan node and use that everywhere. That will hopefully cut down on the number of times this happens. Thank you, Wing Yew! On Sun, Aug 8, 2021 at 2:52 PM Carl Steinbach <cwsteinb...@gmail.com> wrote: > Hi Wing Yew, > > I will create a new RC once this patch is committed. > > Thanks. > > - Carl > > On Sat, Aug 7, 2021 at 4:29 PM Wing Yew Poon <wyp...@cloudera.com.invalid> > wrote: > >> Sorry to bring this up so late, but this just came up: there is a Spark >> 3.1 (runtime) compatibility issue (not found by existing tests), which I >> have a fix for in https://github.com/apache/iceberg/pull/2954. I think >> it would be really helpful if it can go into 0.12.0. >> - Wing Yew >> >> >> On Fri, Aug 6, 2021 at 11:36 AM Jack Ye <yezhao...@gmail.com> wrote: >> >>> +1 (non-binding) >>> >>> Verified release test and AWS integration test, issue found in test but >>> not blocking for release (https://github.com/apache/iceberg/pull/2948) >>> >>> Verified Spark 3.1 and 3.0 operations and new SQL extensions and >>> procedures on EMR. >>> >>> Thanks, >>> Jack Ye >>> >>> On Fri, Aug 6, 2021 at 1:19 AM Kyle Bendickson <kjbendick...@gmail.com> >>> wrote: >>> >>>> +1 (binding) >>>> >>>> I verified: >>>> - KEYS signature & checksum >>>> - ./gradlew clean build (tests, etc) >>>> - Ran Spark jobs on Kubernetes after building from the tarball at >>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.12.0-rc2/ >>>> - Spark 3.1.1 batch jobs against both Hadoop and Hive tables, >>>> using HMS for Hive catalog >>>> - Verified default FileIO and S3FileIO >>>> - Basic read and writes >>>> - Jobs using Spark procedures (remove unreachable files) >>>> - Special mention: verified that Spark catalogs can override hadoop >>>> configurations using configs prefixed with >>>> "spark.sql.catalog.(catalog-name).hadoop." >>>> - one of my contributions to this release that has been asked >>>> about by several customers internally >>>> - tested using >>>> `spark.sql.catalog.(catalog-name).hadoop.fs.s3a.impl` for two catalogs, >>>> both values respected as opposed to the default globally configured value >>>> >>>> Thank you Carl! >>>> >>>> - Kyle, Data OSS Dev @ Apple =) >>>> >>>> On Thu, Aug 5, 2021 at 11:49 PM Szehon Ho <szehon.apa...@gmail.com> >>>> wrote: >>>> >>>>> +1 (non-binding) >>>>> >>>>> * Verify Signature Keys >>>>> * Verify Checksum >>>>> * dev/check-license >>>>> * Build >>>>> * Run tests (though some timeout failures, on Hive MR test..) >>>>> >>>>> Thanks >>>>> Szehon >>>>> >>>>> On Thu, Aug 5, 2021 at 2:23 PM Daniel Weeks <dwe...@apache.org> wrote: >>>>> >>>>>> +1 (binding) >>>>>> >>>>>> I verified sigs/sums, license, build, and test >>>>>> >>>>>> -Dan >>>>>> >>>>>> On Wed, Aug 4, 2021 at 2:53 PM Ryan Murray <rym...@gmail.com> wrote: >>>>>> >>>>>>> After some wrestling w/ Spark I discovered that the problem was with >>>>>>> my test. Some SparkSession apis changed. so all good here now. >>>>>>> >>>>>>> +1 (non-binding) >>>>>>> >>>>>>> On Wed, Aug 4, 2021 at 11:29 PM Ryan Murray <rym...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks for the help Carl, got it sorted out. The gpg check now >>>>>>>> works. For those who were interested I used a canned wget command in my >>>>>>>> history and it pulled the RC0 :-) >>>>>>>> >>>>>>>> Will have a PR to fix the Nessie Catalog soon. >>>>>>>> >>>>>>>> Best, >>>>>>>> Ryan >>>>>>>> >>>>>>>> On Wed, Aug 4, 2021 at 9:21 PM Carl Steinbach < >>>>>>>> cwsteinb...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Ryan, >>>>>>>>> >>>>>>>>> Can you please run the following command to see which keys in your >>>>>>>>> public keyring are associated with my UID? >>>>>>>>> >>>>>>>>> % gpg --list-keys c...@apache.org >>>>>>>>> pub rsa4096/5A5C7F6EB9542945 2021-07-01 [SC] >>>>>>>>> 160F51BE45616B94103ED24D5A5C7F6EB9542945 >>>>>>>>> uid [ultimate] Carl W. Steinbach (CODE SIGNING >>>>>>>>> KEY) <c...@apache.org> >>>>>>>>> sub rsa4096/4158EB8A4F03D2AA 2021-07-01 [E] >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> - Carl >>>>>>>>> >>>>>>>>> On Wed, Aug 4, 2021 at 11:12 AM Ryan Murray <rym...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> Unfortunately I have to give -1 >>>>>>>>>> >>>>>>>>>> I had trouble w/ the keys: >>>>>>>>>> >>>>>>>>>> gpg: assuming signed data in 'apache-iceberg-0.12.0.tar.gz' >>>>>>>>>> gpg: Signature made Mon 02 Aug 2021 03:36:30 CEST >>>>>>>>>> gpg: using RSA key >>>>>>>>>> FAFEB6EAA60C95E2BB5E26F01FF0803CB78D539F >>>>>>>>>> gpg: Can't check signature: No public key >>>>>>>>>> >>>>>>>>>> And I have discovered a bug in NessieCatalog. It is unclear what >>>>>>>>>> is wrong but the NessieCatalog doesn't play nice w/ Spark3.1. I will >>>>>>>>>> raise >>>>>>>>>> a patch ASAP to fix it. Very sorry for the inconvenience. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Ryan >>>>>>>>>> >>>>>>>>>> On Wed, Aug 4, 2021 at 3:20 AM Carl Steinbach <c...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi everyone, >>>>>>>>>>> >>>>>>>>>>> I propose that we release RC2 as the official Apache Iceberg >>>>>>>>>>> 0.12.0 release. Please note that RC0 and RC1 were DOA. >>>>>>>>>>> >>>>>>>>>>> The commit id for RC2 is 7c2fcfd893ab71bee41242b46e894e6187340070 >>>>>>>>>>> * This corresponds to the tag: apache-iceberg-0.12.0-rc2 >>>>>>>>>>> * >>>>>>>>>>> https://github.com/apache/iceberg/commits/apache-iceberg-0.12.0-rc2 >>>>>>>>>>> * >>>>>>>>>>> https://github.com/apache/iceberg/tree/7c2fcfd893ab71bee41242b46e894e6187340070 >>>>>>>>>>> >>>>>>>>>>> The release tarball, signature, and checksums are here: >>>>>>>>>>> * >>>>>>>>>>> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.12.0-rc2/ >>>>>>>>>>> >>>>>>>>>>> You can find the KEYS file here: >>>>>>>>>>> * https://dist.apache.org/repos/dist/dev/iceberg/KEYS >>>>>>>>>>> >>>>>>>>>>> Convenience binary artifacts are staged in Nexus. The Maven >>>>>>>>>>> repository URL is: >>>>>>>>>>> * >>>>>>>>>>> https://repository.apache.org/content/repositories/orgapacheiceberg-1017/ >>>>>>>>>>> >>>>>>>>>>> Please download, verify, and test. >>>>>>>>>>> >>>>>>>>>>> Please vote in the next 72 hours. >>>>>>>>>>> >>>>>>>>>>> [ ] +1 Release this as Apache Iceberg 0.12.0 >>>>>>>>>>> [ ] +0 >>>>>>>>>>> [ ] -1 Do not release this because... >>>>>>>>>>> >>>>>>>>>> -- Ryan Blue Tabular
