Hi Ajantha I will help on the Flink PR. However I don’t think we need to hold 1.9.0 for Spark and Flink updates. We can always include that in the next release cycle.
I propose to focus on issues and check the milestone on GH. Regards JB Le jeu. 27 mars 2025 à 07:11, Ajantha Bhat <ajanthab...@gmail.com> a écrit : > Latest update on the release: > > As discussed in the last community sync, I have waited for the Spark 4.0 > release. But the RC3 has failed. > Flink 2.0 has been released, there is an open PR > <https://github.com/apache/iceberg/pull/12527> to include it at the > Iceberg. It didn't make any progress this week. > > We have recently found a few issues and small requirements for Auth > manager, > Added them to the milestone: > https://github.com/apache/iceberg/milestone/53 > <https://github.com/apache/iceberg/milestone/53> > If we get good review support, I think 2-3 days should be sufficient to > close these and prepare a release. > > > - Ajantha > > On Thu, Mar 20, 2025 at 1:42 PM Alex Dutra <alex.du...@dremio.com.invalid> > wrote: > >> Hi Yuya, >> >> Thanks for reporting this issue, which is indeed a defect in the SigV4 >> auth manager. The fix was hopefully easy: >> >> https://github.com/apache/iceberg/pull/12582 >> >> Sorry for the inconvenience, >> >> Alex >> >> On Thu, Mar 20, 2025 at 5:20 AM Yuya Ebihara < >> yuya.ebih...@starburstdata.com> wrote: >> >>> S3 Tables tests in Trino with 1.9.0 nightly release throws >>> ClassCastException (RESTSigV4AuthSession → OAuth2Util$AuthSession). The >>> test works fine in 1.8.1. >>> I’m going to check if adjusting our test settings based on this PR ( >>> https://github.com/apache/iceberg/pull/11995) can fix the issue. >>> >>> >>> https://github.com/trinodb/trino/actions/runs/13960201796/job/39080056606?pr=25331 >>> Error: >>> io.trino.plugin.iceberg.catalog.rest.TestIcebergS3TablesConnectorSmokeTest >>> -- Time elapsed: 3.038 s <<< ERROR! >>> io.trino.testing.QueryFailedException: class >>> org.apache.iceberg.aws.RESTSigV4AuthSession cannot be cast to class >>> org.apache.iceberg.rest.auth.OAuth2Util$AuthSession >>> (org.apache.iceberg.aws.RESTSigV4AuthSession and >>> org.apache.iceberg.rest.auth.OAuth2Util$AuthSession are in unnamed module >>> of loader 'app') >>> at >>> io.trino.testing.AbstractTestingTrinoClient.execute(AbstractTestingTrinoClient.java:138) >>> at >>> io.trino.testing.DistributedQueryRunner.executeInternal(DistributedQueryRunner.java:565) >>> at >>> io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:548) >>> at io.trino.testing.QueryRunner.execute(QueryRunner.java:82) >>> at >>> io.trino.plugin.iceberg.SchemaInitializer.accept(SchemaInitializer.java:54) >>> at >>> io.trino.plugin.iceberg.IcebergQueryRunner$Builder.lambda$build$3(IcebergQueryRunner.java:178) >>> at java.base/java.util.Optional.ifPresent(Optional.java:178) >>> at >>> io.trino.plugin.iceberg.IcebergQueryRunner$Builder.build(IcebergQueryRunner.java:178) >>> at >>> io.trino.plugin.iceberg.catalog.rest.TestIcebergS3TablesConnectorSmokeTest.createQueryRunner(TestIcebergS3TablesConnectorSmokeTest.java:83) >>> at >>> io.trino.testing.AbstractTestQueryFramework.init(AbstractTestQueryFramework.java:119) >>> at java.base/java.lang.reflect.Method.invoke(Method.java:580) >>> at >>> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507) >>> at >>> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.tryRemoveAndExec(ForkJoinPool.java:1501) >>> at >>> java.base/java.util.concurrent.ForkJoinPool.helpJoin(ForkJoinPool.java:2274) >>> at >>> java.base/java.util.concurrent.ForkJoinTask.awaitDone(ForkJoinTask.java:495) >>> at >>> java.base/java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:662) >>> at >>> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507) >>> at >>> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1460) >>> at >>> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2036) >>> at >>> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189) >>> Suppressed: java.lang.Exception: SQL: CREATE SCHEMA IF NOT EXISTS "tpch" >>> at >>> io.trino.testing.DistributedQueryRunner.executeInternal(DistributedQueryRunner.java:572) >>> ... 18 more >>> Caused by: java.lang.ClassCastException: class >>> org.apache.iceberg.aws.RESTSigV4AuthSession cannot be cast to class >>> org.apache.iceberg.rest.auth.OAuth2Util$AuthSession >>> (org.apache.iceberg.aws.RESTSigV4AuthSession and >>> org.apache.iceberg.rest.auth.OAuth2Util$AuthSession are in unnamed module >>> of loader 'app') >>> at >>> org.apache.iceberg.rest.auth.OAuth2Manager.contextualSession(OAuth2Manager.java:142) >>> at >>> org.apache.iceberg.rest.auth.OAuth2Manager.contextualSession(OAuth2Manager.java:40) >>> at >>> org.apache.iceberg.aws.RESTSigV4AuthManager.contextualSession(RESTSigV4AuthManager.java:68) >>> at >>> org.apache.iceberg.rest.RESTSessionCatalog.loadNamespaceMetadata(RESTSessionCatalog.java:608) >>> at >>> org.apache.iceberg.catalog.SessionCatalog.namespaceExists(SessionCatalog.java:358) >>> at >>> org.apache.iceberg.rest.RESTSessionCatalog.namespaceExists(RESTSessionCatalog.java:595) >>> at >>> io.trino.plugin.iceberg.catalog.rest.TrinoRestCatalog.namespaceExists(TrinoRestCatalog.java:162) >>> at >>> io.trino.plugin.iceberg.IcebergMetadata.schemaExists(IcebergMetadata.java:512) >>> at >>> io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.schemaExists(ClassLoaderSafeConnectorMetadata.java:193) >>> at >>> io.trino.tracing.TracingConnectorMetadata.schemaExists(TracingConnectorMetadata.java:125) >>> at >>> io.trino.metadata.MetadataManager.lambda$schemaExists$1(MetadataManager.java:248) >>> at >>> java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) >>> at >>> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215) >>> at >>> java.base/java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:1034) >>> at >>> java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:147) >>> at >>> java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:588) >>> at >>> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:574) >>> at >>> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560) >>> at >>> java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) >>> at >>> java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) >>> at >>> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) >>> at >>> java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:672) >>> at >>> io.trino.metadata.MetadataManager.schemaExists(MetadataManager.java:248) >>> at >>> io.trino.tracing.TracingMetadata.schemaExists(TracingMetadata.java:165) >>> at >>> io.trino.execution.CreateSchemaTask.internalExecute(CreateSchemaTask.java:119) >>> at io.trino.execution.CreateSchemaTask.execute(CreateSchemaTask.java:82) >>> at io.trino.execution.CreateSchemaTask.execute(CreateSchemaTask.java:54) >>> at >>> io.trino.execution.DataDefinitionExecution.start(DataDefinitionExecution.java:152) >>> at >>> io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:272) >>> at >>> io.trino.dispatcher.LocalDispatchQuery.startExecution(LocalDispatchQuery.java:150) >>> at >>> io.trino.dispatcher.LocalDispatchQuery.lambda$waitForMinimumWorkers$2(LocalDispatchQuery.java:134) >>> at >>> io.airlift.concurrent.MoreFutures.lambda$addSuccessCallback$12(MoreFutures.java:570) >>> at io.airlift.concurrent.MoreFutures$3.onSuccess(MoreFutures.java:545) >>> at >>> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1139) >>> at io.trino.$gen.Trino_testversion____20250320_014939_2038.run(Unknown >>> Source) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) >>> at java.base/java.lang.Thread.run(Thread.java:1575) >>> >>> BR, >>> Yuya >>> >>> On Wed, Mar 19, 2025 at 12:38 AM Manu Zhang <owenzhang1...@gmail.com> >>> wrote: >>> >>>> Hi Ajantha, >>>> >>>> Thanks for driving the release. Can we include >>>> https://github.com/apache/iceberg/pull/12120? >>>> >>>> On Tue, Mar 18, 2025 at 3:18 AM Steve Loughran >>>> <ste...@cloudera.com.invalid> wrote: >>>> >>>>> >>>>> Can I get this reviewed and merged; gives all hadoop filesystems with >>>>> bulk delete calls the ability to issue bulk deletes up to their page >>>>> sizes; >>>>> off by default. Tested all the way through iceberg to AWS S3 london. >>>>> >>>>> https://github.com/apache/iceberg/pull/10233 >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, 17 Mar 2025 at 12:32, Yuya Ebihara < >>>>> yuya.ebih...@starburstdata.com> wrote: >>>>> >>>>>> Hi, can we include https://github.com/apache/iceberg/pull/12264 to >>>>>> fix the S3-compatible storage issue? >>>>>> We downgraded the problematic library in >>>>>> https://github.com/apache/iceberg/pull/12339, but the issue is still >>>>>> present in the main branch. >>>>>> >>>>> >>>>> I'd go with downgrading the AWS SDK to 2.29. x and then get on the >>>>> relevant AWS SDK discussion to express your concerns: >>>>> https://github.com/aws/aws-sdk-java-v2/discussions/5802 >>>>> >>>>> The problem here is that there's such a broad set of implementations >>>>> of the S3 API that it'll take testing to see how even the suggestions from >>>>> the SDK team work everywhere -and we now have explicit confirmation that >>>>> the SDK team leave all such testing to downstream users. >>>>> >>>>> >>>>> >>>>> *The AWS SDKs and CLI are designed for usage with official AWS >>>>> services.We may introduce and enable new features by default, such as >>>>> these >>>>> new default integrity protections,prior to them being supported or >>>>> otherwise handled by third-party service implementations.* >>>>> >>>>> I think ASF projects need to make clear how dangerous this is -that >>>>> projects will end up shipping releases which don't work, and the "set an >>>>> env var or a system property" workarounds are not enough. >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> On Mon, Mar 17, 2025 at 8:47 PM Ajantha Bhat <ajanthab...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> We decided to have a fast follow up on the things that missed >>>>>>> release train of 1.8.0 during community sync. >>>>>>> >>>>>>> More details here: >>>>>>> https://lists.apache.org/thread/wvz5sd7pmh5ww1yqhsxpt1kwf993276j >>>>>>> >>>>>>> On Mon, Mar 17, 2025 at 4:53 PM Russell Spitzer < >>>>>>> russell.spit...@gmail.com> wrote: >>>>>>> >>>>>>>> Can you please rehash the plan? I thought we just did a release >>>>>>>> last month and were aiming for a three month schedule. I may have >>>>>>>> missed >>>>>>>> something >>>>>>>> >>>>>>>> On Mon, Mar 17, 2025 at 6:00 AM Ajantha Bhat <ajanthab...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hey everyone, >>>>>>>>> >>>>>>>>> Following the plan from the 1.8.0 release, I'll be handling the >>>>>>>>> 1.9.0 release. We have several major updates ready: >>>>>>>>> >>>>>>>>> - *Partition stats APIs:* All core APIs for partition >>>>>>>>> statistics have been merged, unblocking engines like Dremio, >>>>>>>>> Trino, and >>>>>>>>> Hive that were waiting for this feature. >>>>>>>>> - *REST catalog authentication:* The refactoring PR for the >>>>>>>>> REST catalog authentication manager has been merged, improving >>>>>>>>> authentication support. >>>>>>>>> - *Spark 3.3 support removed:* We've officially dropped >>>>>>>>> support for Spark 3.3. >>>>>>>>> - *InternalData support in the core module:* The core module >>>>>>>>> now supports InternalData, leveraging the internal Parquet and >>>>>>>>> Avro readers >>>>>>>>> added in the previous release. This allows metadata to be written >>>>>>>>> in >>>>>>>>> Parquet. >>>>>>>>> - *Bug fixes:* Many important bug fixes have been merged. >>>>>>>>> >>>>>>>>> A 1.9.0 milestone has been created with additional "good-to-have" >>>>>>>>> issues: >>>>>>>>> https://github.com/apache/iceberg/milestone/53 >>>>>>>>> >>>>>>>>> If there's anything urgent that needs to be included for this >>>>>>>>> release, please let me know or reply to this thread. >>>>>>>>> I'm aiming to start the release cut by the end of this week. >>>>>>>>> >>>>>>>>> >>>>>>>>> - Ajantha >>>>>>>>> >>>>>>>>
