Thanks xxchan’s summary, we also reached consensus about msrv: 1 At least three months from latest rust release 2 Update msrv when we release iceberg-rust
On Fri, Feb 28, 2025 at 00:16 xxchan <xxchan...@gmail.com> wrote: > Conclusion in today's meeting: > - Test against latest version in CI (by updating Cargo.lock with > dependabot) > - Have a wider range of versions support, to allow users to choose their > dep version (by not updating Cargo.toml too often). Perhaps need to add a > CI to test against oldest version. > > On Thu, Feb 27, 2025 at 2:03 PM Renjie Liu <liurenjie2...@gmail.com> > wrote: > >> Hi, Xuanwo: >> >> If we detect that iceberg is not compatible with lib A 0.1.10, then we >> could know that in advance, and we could figure out ways to fix it. >> >> Users have several choices, like using Cargo.lock to pin the version >> used. >> >> We could have a discussion in this week's community sync. >> >> On Tue, Feb 25, 2025 at 12:05 PM Xuanwo <xua...@apache.org> wrote: >> >>> Hi, renjie >>> >>> > My point is still the same, if iceberg uses lib A 0.1.9, and the >>> 0.1.10 version of lib A doesn't work with iceberg 0.3, the user of iceberg >>> will face either of: >>> > 1. Forced to use the old version of lib A, without enjoying the >>> performance / bug fix of the new version of lib A. >>> > 2. Upgrade to newer version of lib A, but surprised to fail in tests >>> or production. >>> >>> I'm not sure what we can do about this. >>> >>> If Iceberg uses version 0.1.9, users can choose which version of lib A >>> to use—either staying at 0.1.9 or upgrading to 0.1.10 (even it's broken). >>> However, if Iceberg enforces the use of 0.1.10, users will always face >>> issues, with no option to revert to 0.1.9. >>> >>> Regardless of the actions we take, we still need the upstream to fix it. >>> Alternatively, we may need to migrate to other libraries. >>> >>> My point is to let users make this decision themselves. If we now know >>> iceberg-rust doesn't work with lib A versions lower than 0.1.11. We can set >>> it to 0.1.11 to prevent users from relying on the outdated 0.1.10 version. >>> >>> On Tue, Feb 25, 2025, at 10:53, Renjie Liu wrote: >>> >>> Hi, Xuanwo: >>> >>> Thanks for pointing that out, and you're right about the version used. >>> >>> However, if iceberg depends on lib A 0.1.10, but app B depends on lib A >>> 0.1.9, users will finally get: >>> - iceberg 0.3 >>> - lib A 0.1.10 (the biggest if all lib A 0.1.X) >>> >>> >>> My point is still the same, if iceberg uses lib A 0.1.9, and the 0.1.10 >>> version of lib A doesn't work with iceberg 0.3, the user of iceberg will >>> face either of: >>> 1. Forced to use the old version of lib A, without enjoying the >>> performance / bug fix of the new version of lib A. >>> 2. Upgrade to newer version of lib A, but surprised to fail in tests or >>> production. >>> >>> >>> >>> On Mon, Feb 24, 2025 at 6:38 PM Xuanwo <xua...@apache.org> wrote: >>> >>> >>> Hi, renjie >>> >>> > 1. Iceberg 0.3 dependents library a 0.1 >>> > 2. Application b dependents on iceberg 0.3 and library a 0.2 >>> > 3. Library a 0.2 doesn't work with iceberg 0.3 >>> > 4. Application b crashed in production. >>> >>> This example can't happen in rust. In rust, every version within 0.X is >>> considered a breaking change. >>> >>> So if iceberg 0.3 depends on lib A 0.1, and app B depends on iceberg 0.3 >>> and ;ib A 0.2, this user will finally get: >>> >>> - iceberg 0.3 >>> - lib A 0.1 >>> - lib A 0.2 >>> >>> Which means the two version of lib A can coexists in the same app B. >>> >>> However, if iceberg depends on lib A 0.1.10, but app B depends on lib A >>> 0.1.9, users will finally get: >>> >>> - iceberg 0.3 >>> - lib A 0.1.10 (the biggest if all lib A 0.1.X) >>> >>> >>> On Mon, Feb 24, 2025, at 18:12, Renjie Liu wrote: >>> >>> Thanks to xxchan's explanation, it seems that we need to resolve the >>> dependency version first. >>> >>> 2.1 Always use the latest dependency version in Cargo.toml, and force >>> downstream applications to upgrade transitive dependencies at the same time. >>> Benefits: let downstream enjoy latest bug fixes and performance >>> improvements of deps >>> Drawback: downstream is forced to upgrade deps together; They will need >>> to audit the changes of all transitive deps, instead of upgrading 1 by 1. >>> >>> >>> For the dependency version, I prefer this one. Though in theory the >>> dependencies of iceberg are supposed to maintain compatibility if the major >>> version is not changed, it would still be better to test against them. >>> Using a dependent bot to upgrade the dependency version would help to avoid >>> cases like this. For example: >>> 1. Iceberg 0.3 dependents library a 0.1 >>> 2. Application b dependents on iceberg 0.3 and library a 0.2 >>> 3. Library a 0.2 doesn't work with iceberg 0.3 >>> 4. Application b crashed in production. >>> >>> I know this approach doesn't solve all problems, but at least it helps >>> us to detect such a problem and save our time to fix it. >>> >>> >>> >>> On Mon, Feb 24, 2025 at 1:47 PM Xuanwo <xua...@apache.org> wrote: >>> >>> >>> > 2.2 Do not upgrade Cargo.toml, and let the application be responsible >>> for the transitive dependencies' version. Instead, we upgrade in Cargo.lock >>> which means "we've tested the latest version". >>> >>> I prefer this approach. I believe we should disable dependabot from >>> upgrading our patch versions, as it doesn't make sense to me. Users should >>> be able to decide exactly which versions they want to use. As a library, we >>> specify the lowest version that we are confident works well with >>> iceberg-rust or includes all the APIs that iceberg-rust relies on. >>> >>> On Mon, Feb 24, 2025, at 13:28, xxchan wrote: >>> >>> Hi Renjie, >>> >>> Yes you are right that `datafusion-iceberg` will not break. I somehow >>> mistakenly thought that we cannot compile `datafusion` with a newer rust >>> toolchain... Sorry for the mistake. >>> >>> Regarding the motivations >>> >>> > 1. Use new(not latest) features in stable rust as rust is still >>> evolving. >>> >>> In theory this can be done case by case, and we don't have a >>> strong motivation yet. >>> But +1 to upgrade periodically to avoid wasting time discussing whether >>> a feature is "worth upgrading". >>> >>> > 2. Use newer versions of iceberg-rust's dependencies, which may >>> include important bug fixes and performance improvements. I've seen >>> dependent bot auto upgrading failed several times due to too old msrv. >>> >>> Dependencies is a complex problem. I want to clarify some points: >>> >>> 1. iceberg-rust is a library. The source of truth of the dependencies >>> versions will be the consumer application's Cargo.lock. The version in >>> iceberg-rust's Cargo.toml is a semver lower bound. MSRV or old dep version >>> in Cargo.toml will not prevent applications using the latest version. >>> >>> 2. So there are different policies a library can do (ref: >>> https://arc.net/l/quote/xghkkpqt) >>> >>> 2.1 Always use the latest dependency version in Cargo.toml, and force >>> downstream applications to upgrade transitive dependencies at the same time. >>> Benefits: let downstream enjoy latest bug fixes and performance >>> improvements of deps >>> Drawback: downstream is forced to upgrade deps together; They will need >>> to audit the changes of all transitive deps, instead of upgrading 1 by 1. >>> >>> 2.2 Do not upgrade Cargo.toml, and let the application be responsible >>> for the transitive dependencies' version. Instead, we upgrade in Cargo.lock >>> which means "we've tested the latest version". >>> >>> I'm not sure which dependency upgrade policy we want to adopt. (BTW, >>> we've once also discussed this in avro-rs, and they prefer the first one >>> https://github.com/apache/avro-rs/issues/4) >>> Although this is a little off-topic, but it seems similar to the MSRV >>> problem. Dependency policy may also conflict with MSRV policy. e.g., as >>> mentioned by Renjie, we cannot upgrade to latest version of a dep with a >>> low MSRV. (So if we want to maintain low MSRV, we have to adopt the second >>> dependency policy.) >>> >>> xxchan >>> >>> On Mon, Feb 24, 2025 at 11:45 AM Renjie Liu <liurenjie2...@gmail.com> >>> wrote: >>> >>> Hi, xxchan: >>> >>> In fact, datafusion is not a consumer of iceberg-rust, in fact, >>> `datafusion-iceberg` is a consumer of both `iceberg` and `datafusion`, >>> which is maintained in `iceberg-rust` repo, so a more aggressive msrv >>> policy will not break `datafusion-iceberg`. >>> >>> On Sun, Feb 23, 2025 at 8:40 PM xxchan <xxchan...@gmail.com> wrote: >>> >>> Hi, >>> >>> To clarify, if we want to adopt the "at least three months" policy, we >>> will need to wait for 3 months before upgrading to 1.85. >>> And I believe we don't want to break Datafusion integration, so we will >>> have to use the "at least six months" policy (or more conservative). >>> >>> xxchan >>> >>> On Sun, Feb 23, 2025 at 5:50 PM NOTME ZE <st810918...@gmail.com> wrote: >>> >>> Hi, >>> >>> >>> As Xuanwo mentions the "at least three months," policy looks good to me. >>> Also, it's time to upgrade to Rust 1.85 (the first version to support >>> Edition 2024!). See: https://github.com/apache/iceberg-rust/pull/987 >>> >>> >>> Xuanwo >>> >>> https://xuanwo.io/ >>> >>> Xuanwo >>> >>> https://xuanwo.io/ >>> >>> Xuanwo >>> >>> https://xuanwo.io/ >>> >>>
