Update: Running the test suite in IntelliJ that was (and is) having consistent test failures via CLI, the issue seems to be resolved. So I do think it is indeed a local JVM set up issue.
Investigating the differences now, but the class in question is *org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine* It seems to be caused by a NoClassDefFoundError, specifically for org.xerial.snappy.Snappy. It also happens for ORC, but not for parquet. Included is a sample output: ``` java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99) ~[snappy-java-1.1.8.jar:1.1.8] at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91) ~[snappy-java-1.1.8.jar:1.1.8] at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81) ~[snappy-java-1.1.8.jar:1.1.8] at org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:81) ~[tez-api-0.10.1.jar:0.10.1] ``` Apologies for speaking too soon. *I'm now +0 [non-binding] *provided we fix the 0.13.x branch and associated commitId to not be in a detached state. The tag *apache-iceberg-0.13.2-rc1 *works just fine, but the 0.13.x branch doesn't have the commit ID in question. Not sure if that's a major concern or not. Cheers, Kyle On Sun, Jun 5, 2022 at 11:51 AM Kyle Bendickson <k...@tabular.io> wrote: > Thanks Eduard! > > I have: > - verified the signature > - verified the checksum in the file given as well as of the artifact > - ran all unit tests on Java 11, all passed > - ran all unit tests on Java 8, some hive-3 tests consistently fail (I do > notice they passed on Github - but the tests which fail are consistent > despite giving the JVM more memory and checking for OOM) > - ran a simple smoke test suite of CRUD on namespaces and v1 and v2 tables > with Spark (3.2, 3.1) and Flink (1.13 and 1.14). > - ran some upsert related tests on Flink 1.13 and 1.14 (1.12 is provided a > deprecation notice) > > *Problems:* > I did notice that the *given commit ID is considered unattached (and I > wasn't able to check it out).* I am running my tests by using the > provided JAR with engines and then running unit tests locally for the > commit just prior (with commit ID > *fae977a9f0a79266a04647b0df2ab540cf0dcff4*). > > Not sure if this is a huge issue, but outside of this unattached commit, > my only concern is the `iceberg-hive3` failing tests, but as they passed in > CI it's possibly an issue with my local setup locally. > > Running hive-3 test suite alone, the same tests failed multiple times but > again might be something to do with my computer / JVM configuration. > > *I am -1 (non-binding)*, primarily based on the detached commit (as I had > quite a good bit of trouble trying to fetch it through my normal processes) > as well as the failing hive3 tests (though that's not exactly within my > area of expertise). > > If the hive3 test failures are only something that occurs for me, then if > we fix the "Add version.txt commit" in branch 0.3.x such that when I fetch > branch 0.3.x it's present, I'd be +1. Unfortunately, I can't help with > cleaning up with the release branch outside of advising somebody else (if > desired), but I'm happy to help with that. > > The hive3 test failures for me seem to be OOM related, but I raised my > > Find attached a picture of the detached commit ID, > *0784d64a659abd4fdaa82cdb599a250a7514facf*, per Github. > > [image: image.png] > > Example test failures > org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine > > testCBOWithSelectedColumnsOverlapJoin[fileFormat=AVRO, engine=tez, > catalog=HIVE_CATALOG, isVectorized=false] FAILED > java.lang.IllegalArgumentException: Failed to execute Hive query > 'SELECT c.first_name, o.order_id FROM default.orders o JOIN > default.customers c ON o.customer_id = c.customer_id ORDER BY o.order_id > DESC': Error while processing statement: FAILED: Execution Error, return > code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask > at > org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:152) > at > org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testCBOWithSelectedColumnsOverlapJoin(TestHiveIcebergStorageHandlerWithEngine.java:236) > > Caused by: > org.apache.hive.service.cli.HiveSQLException: Error while > processing statement: FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.tez.TezTask > at > org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226) > at > org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:263) > at > org.apache.hive.service.cli.operation.Operation.run(Operation.java:247) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541) > at > org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:510) > at > org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:267) > at > org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:139) > ... 1 more > > Thanks you for working on this, > Kyle > > > > > On Wed, Jun 1, 2022 at 11:12 PM Eduard Tudenhoefner <edu...@dremio.com> > wrote: > >> Hi Everyone, >> >> I propose that we release the following RC as the official Apache Iceberg >> 0.13.2 release. >> >> The commit ID is *0784d64a659abd4fdaa82cdb599a250a7514facf* >> >> >> - This corresponds to the tag: *apache-iceberg-0.13.2-rc1* >> - https://github.com/apache/iceberg/commits/apache-iceberg-0.13.2-rc1 >> - >> >> https://github.com/apache/iceberg/tree/0784d64a659abd4fdaa82cdb599a250a7514facf >> >> >> The release tarball, signature, and checksums are here: >> >> - >> https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.2-rc1 >> >> >> You can find the KEYS file here: >> >> - https://dist.apache.org/repos/dist/dev/iceberg/KEYS >> >> >> Convenience binary artifacts are staged on Nexus. The Maven repository >> URL is: >> >> - >> https://repository.apache.org/content/repositories/orgapacheiceberg-1088/ >> >> >> Please download, verify, and test. >> >> Please vote in the next 72 hours. >> >> [ ] +1 Release this as Apache Iceberg <VERSION> >> [ ] +0 >> [ ] -1 Do not release this because... >> >
