Hi Steven,

Thanks for updating this thread.

I've updated the UnknownType PR
<https://github.com/apache/iceberg/pull/13445> to first block on the
complex cases that will require some more discussion. This way we can
revisit this also after the 1.10.0 release.

Kind regards,
Fokko




Op do 7 aug 2025 om 23:56 schreef Steven Wu <stevenz...@gmail.com>:

> edited the subject line as we are into August.
>
> We are still waiting for the following two changes for the 1.10.0 release
> * Anton's fix for the data frame join using the same snapshot, which will
> introduce a slight behavior change in spark 4.0.
> * unknown type support.
>
>
> On Fri, Aug 1, 2025 at 6:56 AM Alexandre Dutra <adu...@apache.org> wrote:
>
>> Hi Steven,
>>
>> A small regression with S3 signing has been reported to me. The fix is
>> simple:
>>
>> https://github.com/apache/iceberg/pull/13718
>>
>> Would it be still possible to have it in 1.10 please?
>>
>> Thanks,
>> Alex
>>
>>
>> On Thu, Jul 31, 2025 at 7:19 PM Steven Wu <stevenz...@gmail.com> wrote:
>> >
>> > Currently, the 1.10.0 milestone have no open PRs
>> > https://github.com/apache/iceberg/milestone/54
>> >
>> > The variant PR was merged this and last week. There are still some
>> variant testing related PRs, which are probably not blockers for 1.10.0
>> release.
>> > * Spark variant read: https://github.com/apache/iceberg/pull/13219
>> > * use short strings: https://github.com/apache/iceberg/pull/13284
>> >
>> > We are still waiting for the following two changes
>> > * Anton's fix for the data frame join using the same snapshot, which
>> will introduce a slight behavior change in spark 4.0.
>> > * unknown type support. Fokko raised a discussion thread on a blocking
>> issue.
>> >
>> > Anything else did I miss?
>> >
>> >
>> >
>> > On Sat, Jul 26, 2025 at 5:52 AM Fokko Driesprong <fo...@apache.org>
>> wrote:
>> >>
>> >> Hey all,
>> >>
>> >> The read path for the UnknownType needs some community discussion.
>> I've raised a separate thread. PTAL
>> >>
>> >> Kind regards from Belgium,
>> >> Fokko
>> >>
>> >> Op za 26 jul 2025 om 00:58 schreef Ryan Blue <rdb...@gmail.com>:
>> >>>
>> >>> I thought that we said we wanted to get support out for v3 features
>> in this release unless there is some reasonable blocker, like Spark not
>> having geospatial types. To me, I think that means we should aim to get
>> variant and unknown done so that we have a complete implementation with a
>> major engine. And it should not be particularly difficult to get unknown
>> done so I'd opt to get it in.
>> >>>
>> >>> On Fri, Jul 25, 2025 at 11:28 AM Steven Wu <stevenz...@gmail.com>
>> wrote:
>> >>>>
>> >>>> > I believe we also wanted to get in at least the read path for
>> UnknownType. Fokko has a WIP PR for that.
>> >>>> I thought in the community sync the consensus is that this is not a
>> blocker, because it is a new feature implementation. If it is ready, it
>> will be included.
>> >>>>
>> >>>> On Fri, Jul 25, 2025 at 9:43 AM Kevin Liu <kevinjq...@apache.org>
>> wrote:
>> >>>>>
>> >>>>> I think Fokko's OOO. Should we help with that PR?
>> >>>>>
>> >>>>> On Fri, Jul 25, 2025 at 9:38 AM Eduard Tudenhöfner <
>> etudenhoef...@apache.org> wrote:
>> >>>>>>
>> >>>>>> I believe we also wanted to get in at least the read path for
>> UnknownType. Fokko has a WIP PR for that.
>> >>>>>>
>> >>>>>> On Fri, Jul 25, 2025 at 6:13 PM Steven Wu <stevenz...@gmail.com>
>> wrote:
>> >>>>>>>
>> >>>>>>> 3. Spark: fix data frame join based on different versions of the
>> same table that may lead to weird results. Anton is working on a fix. It
>> requires a small behavior change (table state may be stale up to refresh
>> interval). Hence it is better to include it in the 1.10.0 release where
>> Spark 4.0 is first supported.
>> >>>>>>> 4. Variant support in core and Spark 4.0. Ryan thinks this is
>> very close and will prioritize the review.
>> >>>>>>>
>> >>>>>>> We still have the above two issues pending. 3 doesn't have a PR
>> yet. PR for 4 is not associated with the milestone yet.
>> >>>>>>>
>> >>>>>>> On Fri, Jul 25, 2025 at 9:02 AM Kevin Liu <kevinjq...@apache.org>
>> wrote:
>> >>>>>>>>
>> >>>>>>>> Thanks everyone for the review. The 2 PRs are both merged.
>> >>>>>>>> Looks like there's only 1 PR left in the 1.10 milestone :)
>> >>>>>>>>
>> >>>>>>>> Best,
>> >>>>>>>> Kevin Liu
>> >>>>>>>>
>> >>>>>>>> On Thu, Jul 24, 2025 at 7:44 PM Manu Zhang <
>> owenzhang1...@gmail.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Thanks Kevin. The first change is not in the versioned doc so
>> it can be released anytime.
>> >>>>>>>>>
>> >>>>>>>>> Regards,
>> >>>>>>>>> Manu
>> >>>>>>>>>
>> >>>>>>>>> On Fri, Jul 25, 2025 at 4:21 AM Kevin Liu <
>> kevinjq...@apache.org> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> The 3 PRs above are merged. Thanks everyone for the review.
>> >>>>>>>>>>
>> >>>>>>>>>> I've added 2 more PRs to the 1.10 milestone. These are both
>> nice-to-haves.
>> >>>>>>>>>> - docs: add subpage for REST Catalog Spec in "Specification"
>> #13521
>> >>>>>>>>>> - REST-Fixture: Ensure strict mode on jdbc catalog for rest
>> fixture #13599
>> >>>>>>>>>>
>> >>>>>>>>>> The first one changes the link for "REST Catalog Spec" on the
>> left nav of https://iceberg.apache.org/spec/ from the swagger.io link to
>> a dedicated page for IRC.
>> >>>>>>>>>> The second one fixes the default behavior of
>> `iceberg-rest-fixture` image to align with the general expectation when
>> creating a table in a catalog.
>> >>>>>>>>>>
>> >>>>>>>>>> Please take a look. I would like to have both of these as part
>> of the 1.10 release.
>> >>>>>>>>>>
>> >>>>>>>>>> Best,
>> >>>>>>>>>> Kevin Liu
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Wed, Jul 23, 2025 at 1:31 PM Kevin Liu <
>> kevinjq...@apache.org> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Here are the 3 PRs to add corresponding tests.
>> >>>>>>>>>>> https://github.com/apache/iceberg/pull/13648
>> >>>>>>>>>>> https://github.com/apache/iceberg/pull/13649
>> >>>>>>>>>>> https://github.com/apache/iceberg/pull/13650
>> >>>>>>>>>>>
>> >>>>>>>>>>> I've tagged them with the 1.10 milestone, waiting for CI to
>> complete :)
>> >>>>>>>>>>>
>> >>>>>>>>>>> Best,
>> >>>>>>>>>>> Kevin Liu
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Wed, Jul 23, 2025 at 1:08 PM Steven Wu <
>> stevenz...@gmail.com> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Kevin, thanks for checking that. I will take a look at your
>> backport PRs. Can you add them to the 1.10.0 milestone?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Wed, Jul 23, 2025 at 12:27 PM Kevin Liu <
>> kevinjq...@apache.org> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Thanks again for driving this Steven! We're very close!!
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> As mentioned in the community sync today, I wanted to
>> verify feature parity between Spark 3.5 and Spark 4.0 for this release.
>> >>>>>>>>>>>>> I was able to verify that Spark 3.5 and Spark 4.0 have
>> feature parity for this upcoming release. More details in the other devlist
>> thread https://lists.apache.org/thread/7x7xcm3y87y81c4grq4nn9gdjd4jm05f
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>> Kevin Liu
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Wed, Jul 23, 2025 at 12:17 PM Steven Wu <
>> stevenz...@gmail.com> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Another update on the release.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> The existing blocker PRs are almost done.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> During today's community sync, we identified the following
>> issues/PRs to be included in the 1.10.0 release.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> backport of PR 13100 to the main branch. I have created a
>> cherry-pick PR for that. There is a one line difference compared to the
>> original PR due to the removal of the deprecated RemoveSnapshot class in
>> main branch for 1.10.0 target. Amogh has suggested using RemoveSnapshots
>> with a single snapshot id, which should be supported by all REST catalog
>> servers.
>> >>>>>>>>>>>>>> Flink compaction doesn't support row lineage. Fail the
>> compaction for V3 tables. I created a PR for that. Will backport after it
>> is merged.
>> >>>>>>>>>>>>>> Spark: fix data frame join based on different versions of
>> the same table that may lead to weird results. Anton is working on a fix.
>> It requires a small behavior change (table state may be stale up to refresh
>> interval). Hence it is better to include it in the 1.10.0 release where
>> Spark 4.0 is first supported.
>> >>>>>>>>>>>>>> Variant support in core and Spark 4.0. Ryan thinks this is
>> very close and will prioritize the review.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>> steven
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> The 1.10.0 milestone can be found here.
>> >>>>>>>>>>>>>> https://github.com/apache/iceberg/milestone/54
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Wed, Jul 16, 2025 at 9:15 AM Steven Wu <
>> stevenz...@gmail.com> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Ajantha/Robin, thanks for the note. We can include the PR
>> in the 1.10.0 milestone.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Wed, Jul 16, 2025 at 3:20 AM Robin Moffatt
>> <ro...@confluent.io.invalid> wrote:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Thanks Ajantha. Just to confirm, from a Confluent point
>> of view, we will not be able to publish the connector on Confluent Hub
>> until this CVE[1] is fixed.
>> >>>>>>>>>>>>>>>> Since we would not publish a snapshot build, if the fix
>> doesn't make it into 1.10 then we'd have to wait for 1.11 (or a dot release
>> of 1.10) to be able to include the connector on Confluent Hub.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Thanks, Robin.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> [1]
>> https://github.com/apache/iceberg/issues/10745#issuecomment-3074300861
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On Wed, 16 Jul 2025 at 04:03, Ajantha Bhat <
>> ajanthab...@gmail.com> wrote:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I have approached Confluent people to help us publish
>> the OSS Kafka Connect Iceberg sink plugin.
>> >>>>>>>>>>>>>>>>> It seems we have a CVE from dependency that blocks us
>> from publishing the plugin.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Please include the below PR for 1.10.0 release which
>> fixes that.
>> >>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/13561
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> - Ajantha
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> On Tue, Jul 15, 2025 at 10:48 AM Steven Wu <
>> stevenz...@gmail.com> wrote:
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> > Engines may model operations as deleting/inserting
>> rows or as modifications to rows that preserve row ids.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Manu, I agree this sentence probably lacks some
>> context. The first half (as deleting/inserting rows) is probably about the
>> row lineage handling with equality deletes, which is described in another
>> place.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> "Row lineage does not track lineage for rows updated
>> via Equality Deletes, because engines using equality deletes avoid reading
>> existing data before writing changes and can't provide the original row ID
>> for the new rows. These updates are always treated as if the existing row
>> was completely removed and a unique new row was added."
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 5:49 PM Manu Zhang <
>> owenzhang1...@gmail.com> wrote:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Thanks Steven, I missed that part but the following
>> sentence is a bit hard to understand (maybe just me)
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Engines may model operations as deleting/inserting
>> rows or as modifications to rows that preserve row ids.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Can you please help to explain?
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Steven Wu <stevenz...@gmail.com>于2025年7月15日
>> 周二04:41写道:
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Manu
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> The spec already covers the row lineage carry over
>> (for replace)
>> >>>>>>>>>>>>>>>>>>>> https://iceberg.apache.org/spec/#row-lineage
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> "When an existing row is moved to a different data
>> file for any reason, writers should write _row_id and
>> _last_updated_sequence_number according to the following rules:"
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>>>>>>> Steven
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 1:38 PM Steven Wu <
>> stevenz...@gmail.com> wrote:
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> another update on the release.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> We have one open PR left for the 1.10.0 milestone
>> (with 25 closed PRs). Amogh is actively working on the last blocker PR.
>> >>>>>>>>>>>>>>>>>>>>> Spark 4.0: Preserve row lineage information on
>> compaction
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> I will publish a release candidate after the above
>> blocker is merged and backported.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>>>>>>>> Steven
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> On Mon, Jul 7, 2025 at 11:56 PM Manu Zhang <
>> owenzhang1...@gmail.com> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Hi Amogh,
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Is it defined in the table spec that "replace"
>> operation should carry over existing lineage info insteading of assigning
>> new IDs? If not, we'd better firstly define it in spec because all engines
>> and implementations need to follow it.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> On Tue, Jul 8, 2025 at 11:44 AM Amogh Jahagirdar <
>> 2am...@gmail.com> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> One other area I think we need to make sure works
>> with row lineage before release is data file compaction. At the moment, it
>> looks like compaction will read the records from the data files without
>> projecting the lineage fields. What this means is that on write of the new
>> compacted data files we'd be losing the lineage information. There's no
>> data change in a compaction but we do need to make sure the lineage info
>> from carried over records is materialized in the newly compacted files so
>> they don't get new IDs or inherit the new file sequence number. I'm working
>> on addressing this as well, but I'd call this out as a blocker as well.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>>>> Robin Moffatt
>> >>>>>>>>>>>>>>>> Sr. Principal Advisor, Streaming Data Technologies
>>
>

Reply via email to