I started a vote thread for 1.10.0 RC2. I have to fix a couple of release script issues. Hence the first release candidate is RC2 to vote.
On Fri, Aug 29, 2025 at 9:53 AM Kevin Liu <kevinjq...@apache.org> wrote: > Thanks Steven! I did another pass to check for feature parity between > spark 3.5 and spark 4.0 for this release and everything looks good. There > are a few test cases that have not been ported, but we can punt those for > now. > > Best, > Kevin Liu > > On Thu, Aug 28, 2025 at 7:08 PM Steven Wu <stevenz...@gmail.com> wrote: > >> Thanks to Fokko and Ryan, the unknown type support PR was merged today. >> >> Everything in the 1.10.0 milestone is closed now. >> >> I will work on a release candidate next. >> >> On Fri, Aug 8, 2025 at 6:14 AM Fokko Driesprong <fo...@apache.org> wrote: >> >>> Hi Steven, >>> >>> Thanks for updating this thread. >>> >>> I've updated the UnknownType PR >>> <https://github.com/apache/iceberg/pull/13445> to first block on the >>> complex cases that will require some more discussion. This way we can >>> revisit this also after the 1.10.0 release. >>> >>> Kind regards, >>> Fokko >>> >>> >>> >>> >>> Op do 7 aug 2025 om 23:56 schreef Steven Wu <stevenz...@gmail.com>: >>> >>>> edited the subject line as we are into August. >>>> >>>> We are still waiting for the following two changes for the 1.10.0 >>>> release >>>> * Anton's fix for the data frame join using the same snapshot, which >>>> will introduce a slight behavior change in spark 4.0. >>>> * unknown type support. >>>> >>>> >>>> On Fri, Aug 1, 2025 at 6:56 AM Alexandre Dutra <adu...@apache.org> >>>> wrote: >>>> >>>>> Hi Steven, >>>>> >>>>> A small regression with S3 signing has been reported to me. The fix is >>>>> simple: >>>>> >>>>> https://github.com/apache/iceberg/pull/13718 >>>>> >>>>> Would it be still possible to have it in 1.10 please? >>>>> >>>>> Thanks, >>>>> Alex >>>>> >>>>> >>>>> On Thu, Jul 31, 2025 at 7:19 PM Steven Wu <stevenz...@gmail.com> >>>>> wrote: >>>>> > >>>>> > Currently, the 1.10.0 milestone have no open PRs >>>>> > https://github.com/apache/iceberg/milestone/54 >>>>> > >>>>> > The variant PR was merged this and last week. There are still some >>>>> variant testing related PRs, which are probably not blockers for 1.10.0 >>>>> release. >>>>> > * Spark variant read: https://github.com/apache/iceberg/pull/13219 >>>>> > * use short strings: https://github.com/apache/iceberg/pull/13284 >>>>> > >>>>> > We are still waiting for the following two changes >>>>> > * Anton's fix for the data frame join using the same snapshot, which >>>>> will introduce a slight behavior change in spark 4.0. >>>>> > * unknown type support. Fokko raised a discussion thread on a >>>>> blocking issue. >>>>> > >>>>> > Anything else did I miss? >>>>> > >>>>> > >>>>> > >>>>> > On Sat, Jul 26, 2025 at 5:52 AM Fokko Driesprong <fo...@apache.org> >>>>> wrote: >>>>> >> >>>>> >> Hey all, >>>>> >> >>>>> >> The read path for the UnknownType needs some community discussion. >>>>> I've raised a separate thread. PTAL >>>>> >> >>>>> >> Kind regards from Belgium, >>>>> >> Fokko >>>>> >> >>>>> >> Op za 26 jul 2025 om 00:58 schreef Ryan Blue <rdb...@gmail.com>: >>>>> >>> >>>>> >>> I thought that we said we wanted to get support out for v3 >>>>> features in this release unless there is some reasonable blocker, like >>>>> Spark not having geospatial types. To me, I think that means we should aim >>>>> to get variant and unknown done so that we have a complete implementation >>>>> with a major engine. And it should not be particularly difficult to get >>>>> unknown done so I'd opt to get it in. >>>>> >>> >>>>> >>> On Fri, Jul 25, 2025 at 11:28 AM Steven Wu <stevenz...@gmail.com> >>>>> wrote: >>>>> >>>> >>>>> >>>> > I believe we also wanted to get in at least the read path for >>>>> UnknownType. Fokko has a WIP PR for that. >>>>> >>>> I thought in the community sync the consensus is that this is not >>>>> a blocker, because it is a new feature implementation. If it is ready, it >>>>> will be included. >>>>> >>>> >>>>> >>>> On Fri, Jul 25, 2025 at 9:43 AM Kevin Liu <kevinjq...@apache.org> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> I think Fokko's OOO. Should we help with that PR? >>>>> >>>>> >>>>> >>>>> On Fri, Jul 25, 2025 at 9:38 AM Eduard Tudenhöfner < >>>>> etudenhoef...@apache.org> wrote: >>>>> >>>>>> >>>>> >>>>>> I believe we also wanted to get in at least the read path for >>>>> UnknownType. Fokko has a WIP PR for that. >>>>> >>>>>> >>>>> >>>>>> On Fri, Jul 25, 2025 at 6:13 PM Steven Wu <stevenz...@gmail.com> >>>>> wrote: >>>>> >>>>>>> >>>>> >>>>>>> 3. Spark: fix data frame join based on different versions of >>>>> the same table that may lead to weird results. Anton is working on a fix. >>>>> It requires a small behavior change (table state may be stale up to >>>>> refresh >>>>> interval). Hence it is better to include it in the 1.10.0 release where >>>>> Spark 4.0 is first supported. >>>>> >>>>>>> 4. Variant support in core and Spark 4.0. Ryan thinks this is >>>>> very close and will prioritize the review. >>>>> >>>>>>> >>>>> >>>>>>> We still have the above two issues pending. 3 doesn't have a >>>>> PR yet. PR for 4 is not associated with the milestone yet. >>>>> >>>>>>> >>>>> >>>>>>> On Fri, Jul 25, 2025 at 9:02 AM Kevin Liu < >>>>> kevinjq...@apache.org> wrote: >>>>> >>>>>>>> >>>>> >>>>>>>> Thanks everyone for the review. The 2 PRs are both merged. >>>>> >>>>>>>> Looks like there's only 1 PR left in the 1.10 milestone :) >>>>> >>>>>>>> >>>>> >>>>>>>> Best, >>>>> >>>>>>>> Kevin Liu >>>>> >>>>>>>> >>>>> >>>>>>>> On Thu, Jul 24, 2025 at 7:44 PM Manu Zhang < >>>>> owenzhang1...@gmail.com> wrote: >>>>> >>>>>>>>> >>>>> >>>>>>>>> Thanks Kevin. The first change is not in the versioned doc >>>>> so it can be released anytime. >>>>> >>>>>>>>> >>>>> >>>>>>>>> Regards, >>>>> >>>>>>>>> Manu >>>>> >>>>>>>>> >>>>> >>>>>>>>> On Fri, Jul 25, 2025 at 4:21 AM Kevin Liu < >>>>> kevinjq...@apache.org> wrote: >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> The 3 PRs above are merged. Thanks everyone for the review. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> I've added 2 more PRs to the 1.10 milestone. These are both >>>>> nice-to-haves. >>>>> >>>>>>>>>> - docs: add subpage for REST Catalog Spec in >>>>> "Specification" #13521 >>>>> >>>>>>>>>> - REST-Fixture: Ensure strict mode on jdbc catalog for rest >>>>> fixture #13599 >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> The first one changes the link for "REST Catalog Spec" on >>>>> the left nav of https://iceberg.apache.org/spec/ from the swagger.io >>>>> link to a dedicated page for IRC. >>>>> >>>>>>>>>> The second one fixes the default behavior of >>>>> `iceberg-rest-fixture` image to align with the general expectation when >>>>> creating a table in a catalog. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> Please take a look. I would like to have both of these as >>>>> part of the 1.10 release. >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> Best, >>>>> >>>>>>>>>> Kevin Liu >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> >>>>>>>>>> On Wed, Jul 23, 2025 at 1:31 PM Kevin Liu < >>>>> kevinjq...@apache.org> wrote: >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> Here are the 3 PRs to add corresponding tests. >>>>> >>>>>>>>>>> https://github.com/apache/iceberg/pull/13648 >>>>> >>>>>>>>>>> https://github.com/apache/iceberg/pull/13649 >>>>> >>>>>>>>>>> https://github.com/apache/iceberg/pull/13650 >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> I've tagged them with the 1.10 milestone, waiting for CI >>>>> to complete :) >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> Best, >>>>> >>>>>>>>>>> Kevin Liu >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> On Wed, Jul 23, 2025 at 1:08 PM Steven Wu < >>>>> stevenz...@gmail.com> wrote: >>>>> >>>>>>>>>>>> >>>>> >>>>>>>>>>>> Kevin, thanks for checking that. I will take a look at >>>>> your backport PRs. Can you add them to the 1.10.0 milestone? >>>>> >>>>>>>>>>>> >>>>> >>>>>>>>>>>> On Wed, Jul 23, 2025 at 12:27 PM Kevin Liu < >>>>> kevinjq...@apache.org> wrote: >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> Thanks again for driving this Steven! We're very close!! >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> As mentioned in the community sync today, I wanted to >>>>> verify feature parity between Spark 3.5 and Spark 4.0 for this release. >>>>> >>>>>>>>>>>>> I was able to verify that Spark 3.5 and Spark 4.0 have >>>>> feature parity for this upcoming release. More details in the other >>>>> devlist >>>>> thread >>>>> https://lists.apache.org/thread/7x7xcm3y87y81c4grq4nn9gdjd4jm05f >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> Thanks, >>>>> >>>>>>>>>>>>> Kevin Liu >>>>> >>>>>>>>>>>>> >>>>> >>>>>>>>>>>>> On Wed, Jul 23, 2025 at 12:17 PM Steven Wu < >>>>> stevenz...@gmail.com> wrote: >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> Another update on the release. >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> The existing blocker PRs are almost done. >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> During today's community sync, we identified the >>>>> following issues/PRs to be included in the 1.10.0 release. >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> backport of PR 13100 to the main branch. I have created >>>>> a cherry-pick PR for that. There is a one line difference compared to the >>>>> original PR due to the removal of the deprecated RemoveSnapshot class in >>>>> main branch for 1.10.0 target. Amogh has suggested using RemoveSnapshots >>>>> with a single snapshot id, which should be supported by all REST catalog >>>>> servers. >>>>> >>>>>>>>>>>>>> Flink compaction doesn't support row lineage. Fail the >>>>> compaction for V3 tables. I created a PR for that. Will backport after it >>>>> is merged. >>>>> >>>>>>>>>>>>>> Spark: fix data frame join based on different versions >>>>> of the same table that may lead to weird results. Anton is working on a >>>>> fix. It requires a small behavior change (table state may be stale up to >>>>> refresh interval). Hence it is better to include it in the 1.10.0 release >>>>> where Spark 4.0 is first supported. >>>>> >>>>>>>>>>>>>> Variant support in core and Spark 4.0. Ryan thinks this >>>>> is very close and will prioritize the review. >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> Thanks, >>>>> >>>>>>>>>>>>>> steven >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> The 1.10.0 milestone can be found here. >>>>> >>>>>>>>>>>>>> https://github.com/apache/iceberg/milestone/54 >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> On Wed, Jul 16, 2025 at 9:15 AM Steven Wu < >>>>> stevenz...@gmail.com> wrote: >>>>> >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> Ajantha/Robin, thanks for the note. We can include the >>>>> PR in the 1.10.0 milestone. >>>>> >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> On Wed, Jul 16, 2025 at 3:20 AM Robin Moffatt >>>>> <ro...@confluent.io.invalid> wrote: >>>>> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> Thanks Ajantha. Just to confirm, from a Confluent >>>>> point of view, we will not be able to publish the connector on Confluent >>>>> Hub until this CVE[1] is fixed. >>>>> >>>>>>>>>>>>>>>> Since we would not publish a snapshot build, if the >>>>> fix doesn't make it into 1.10 then we'd have to wait for 1.11 (or a dot >>>>> release of 1.10) to be able to include the connector on Confluent Hub. >>>>> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> Thanks, Robin. >>>>> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> [1] >>>>> https://github.com/apache/iceberg/issues/10745#issuecomment-3074300861 >>>>> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> On Wed, 16 Jul 2025 at 04:03, Ajantha Bhat < >>>>> ajanthab...@gmail.com> wrote: >>>>> >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> I have approached Confluent people to help us >>>>> publish the OSS Kafka Connect Iceberg sink plugin. >>>>> >>>>>>>>>>>>>>>>> It seems we have a CVE from dependency that blocks >>>>> us from publishing the plugin. >>>>> >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> Please include the below PR for 1.10.0 release which >>>>> fixes that. >>>>> >>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/13561 >>>>> >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> - Ajantha >>>>> >>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>> On Tue, Jul 15, 2025 at 10:48 AM Steven Wu < >>>>> stevenz...@gmail.com> wrote: >>>>> >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> > Engines may model operations as >>>>> deleting/inserting rows or as modifications to rows that preserve row ids. >>>>> >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> Manu, I agree this sentence probably lacks some >>>>> context. The first half (as deleting/inserting rows) is probably about the >>>>> row lineage handling with equality deletes, which is described in another >>>>> place. >>>>> >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> "Row lineage does not track lineage for rows >>>>> updated via Equality Deletes, because engines using equality deletes avoid >>>>> reading existing data before writing changes and can't provide the >>>>> original >>>>> row ID for the new rows. These updates are always treated as if the >>>>> existing row was completely removed and a unique new row was added." >>>>> >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 5:49 PM Manu Zhang < >>>>> owenzhang1...@gmail.com> wrote: >>>>> >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> Thanks Steven, I missed that part but the >>>>> following sentence is a bit hard to understand (maybe just me) >>>>> >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> Engines may model operations as deleting/inserting >>>>> rows or as modifications to rows that preserve row ids. >>>>> >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> Can you please help to explain? >>>>> >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> Steven Wu <stevenz...@gmail.com>于2025年7月15日 >>>>> 周二04:41写道： >>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>> Manu >>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>> The spec already covers the row lineage carry >>>>> over (for replace) >>>>> >>>>>>>>>>>>>>>>>>>> https://iceberg.apache.org/spec/#row-lineage >>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>> "When an existing row is moved to a different >>>>> data file for any reason, writers should write _row_id and >>>>> _last_updated_sequence_number according to the following rules:" >>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>> >>>>>>>>>>>>>>>>>>>> Steven >>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 1:38 PM Steven Wu < >>>>> stevenz...@gmail.com> wrote: >>>>> >>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>> another update on the release. >>>>> >>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>> We have one open PR left for the 1.10.0 >>>>> milestone (with 25 closed PRs). Amogh is actively working on the last >>>>> blocker PR. >>>>> >>>>>>>>>>>>>>>>>>>>> Spark 4.0: Preserve row lineage information on >>>>> compaction >>>>> >>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>> I will publish a release candidate after the >>>>> above blocker is merged and backported. >>>>> >>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>> Thanks, >>>>> >>>>>>>>>>>>>>>>>>>>> Steven >>>>> >>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>> On Mon, Jul 7, 2025 at 11:56 PM Manu Zhang < >>>>> owenzhang1...@gmail.com> wrote: >>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>> Hi Amogh, >>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>> Is it defined in the table spec that "replace" >>>>> operation should carry over existing lineage info insteading of assigning >>>>> new IDs? If not, we'd better firstly define it in spec because all engines >>>>> and implementations need to follow it. >>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 8, 2025 at 11:44 AM Amogh >>>>> Jahagirdar <2am...@gmail.com> wrote: >>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>>> One other area I think we need to make sure >>>>> works with row lineage before release is data file compaction. At the >>>>> moment, it looks like compaction will read the records from the data files >>>>> without projecting the lineage fields. What this means is that on write of >>>>> the new compacted data files we'd be losing the lineage information. >>>>> There's no data change in a compaction but we do need to make sure the >>>>> lineage info from carried over records is materialized in the newly >>>>> compacted files so they don't get new IDs or inherit the new file sequence >>>>> number. I'm working on addressing this as well, but I'd call this out as a >>>>> blocker as well. >>>>> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>> -- >>>>> >>>>>>>>>>>>>>>> Robin Moffatt >>>>> >>>>>>>>>>>>>>>> Sr. Principal Advisor, Streaming Data Technologies >>>>> >>>>
