Hi, renjie > 1. Iceberg 0.3 dependents library a 0.1 > 2. Application b dependents on iceberg 0.3 and library a 0.2 > 3. Library a 0.2 doesn't work with iceberg 0.3 > 4. Application b crashed in production.
This example can't happen in rust. In rust, every version within 0.X is considered a breaking change. So if iceberg 0.3 depends on lib A 0.1, and app B depends on iceberg 0.3 and ;ib A 0.2, this user will finally get: - iceberg 0.3 - lib A 0.1 - lib A 0.2 Which means the two version of lib A can coexists in the same app B. However, if iceberg depends on lib A 0.1.10, but app B depends on lib A 0.1.9, users will finally get: - iceberg 0.3 - lib A 0.1.10 (the biggest if all lib A 0.1.X) On Mon, Feb 24, 2025, at 18:12, Renjie Liu wrote: > Thanks to xxchan's explanation, it seems that we need to resolve the > dependency version first. > >> 2.1 Always use the latest dependency version in Cargo.toml, and force >> downstream applications to upgrade transitive dependencies at the same time. >> Benefits: let downstream enjoy latest bug fixes and performance improvements >> of deps >> Drawback: downstream is forced to upgrade deps together; They will need to >> audit the changes of all transitive deps, instead of upgrading 1 by 1. > > For the dependency version, I prefer this one. Though in theory the > dependencies of iceberg are supposed to maintain compatibility if the major > version is not changed, it would still be better to test against them. Using > a dependent bot to upgrade the dependency version would help to avoid cases > like this. For example: > 1. Iceberg 0.3 dependents library a 0.1 > 2. Application b dependents on iceberg 0.3 and library a 0.2 > 3. Library a 0.2 doesn't work with iceberg 0.3 > 4. Application b crashed in production. > > I know this approach doesn't solve all problems, but at least it helps us to > detect such a problem and save our time to fix it. > > > > On Mon, Feb 24, 2025 at 1:47 PM Xuanwo <xua...@apache.org> wrote: >> __ >> > 2.2 Do not upgrade Cargo.toml, and let the application be responsible for >> > the transitive dependencies' version. Instead, we upgrade in Cargo.lock >> > which means "we've tested the latest version". >> >> I prefer this approach. I believe we should disable dependabot from >> upgrading our patch versions, as it doesn't make sense to me. Users should >> be able to decide exactly which versions they want to use. As a library, we >> specify the lowest version that we are confident works well with >> iceberg-rust or includes all the APIs that iceberg-rust relies on. >> >> On Mon, Feb 24, 2025, at 13:28, xxchan wrote: >>> Hi Renjie, >>> >>> Yes you are right that `datafusion-iceberg` will not break. I somehow >>> mistakenly thought that we cannot compile `datafusion` with a newer rust >>> toolchain... Sorry for the mistake. >>> >>> Regarding the motivations >>> >>> > 1. Use new(not latest) features in stable rust as rust is still evolving. >>> >>> In theory this can be done case by case, and we don't have a strong >>> motivation yet. >>> But +1 to upgrade periodically to avoid wasting time discussing whether a >>> feature is "worth upgrading". >>> >>> > 2. Use newer versions of iceberg-rust's dependencies, which may include >>> > important bug fixes and performance improvements. I've seen dependent bot >>> > auto upgrading failed several times due to too old msrv. >>> >>> Dependencies is a complex problem. I want to clarify some points: >>> >>> 1. iceberg-rust is a library. The source of truth of the dependencies >>> versions will be the consumer application's Cargo.lock. The version in >>> iceberg-rust's Cargo.toml is a semver lower bound. MSRV or old dep version >>> in Cargo.toml will not prevent applications using the latest version. >>> >>> 2. So there are different policies a library can do (ref: >>> https://arc.net/l/quote/xghkkpqt) >>> >>> 2.1 Always use the latest dependency version in Cargo.toml, and force >>> downstream applications to upgrade transitive dependencies at the same time. >>> Benefits: let downstream enjoy latest bug fixes and performance >>> improvements of deps >>> Drawback: downstream is forced to upgrade deps together; They will need to >>> audit the changes of all transitive deps, instead of upgrading 1 by 1. >>> >>> 2.2 Do not upgrade Cargo.toml, and let the application be responsible for >>> the transitive dependencies' version. Instead, we upgrade in Cargo.lock >>> which means "we've tested the latest version". >>> >>> I'm not sure which dependency upgrade policy we want to adopt. (BTW, we've >>> once also discussed this in avro-rs, and they prefer the first one >>> https://github.com/apache/avro-rs/issues/4) >>> Although this is a little off-topic, but it seems similar to the MSRV >>> problem. Dependency policy may also conflict with MSRV policy. e.g., as >>> mentioned by Renjie, we cannot upgrade to latest version of a dep with a >>> low MSRV. (So if we want to maintain low MSRV, we have to adopt the second >>> dependency policy.) >>> >>> xxchan >>> >>> On Mon, Feb 24, 2025 at 11:45 AM Renjie Liu <liurenjie2...@gmail.com> wrote: >>>> Hi, xxchan: >>>> >>>> In fact, datafusion is not a consumer of iceberg-rust, in fact, >>>> `datafusion-iceberg` is a consumer of both `iceberg` and `datafusion`, >>>> which is maintained in `iceberg-rust` repo, so a more aggressive msrv >>>> policy will not break `datafusion-iceberg`. >>>> >>>> On Sun, Feb 23, 2025 at 8:40 PM xxchan <xxchan...@gmail.com> wrote: >>>>> Hi, >>>>> >>>>> To clarify, if we want to adopt the "at least three months" policy, we >>>>> will need to wait for 3 months before upgrading to 1.85. >>>>> And I believe we don't want to break Datafusion integration, so we will >>>>> have to use the "at least six months" policy (or more conservative). >>>>> >>>>> xxchan >>>>> >>>>> On Sun, Feb 23, 2025 at 5:50 PM NOTME ZE <st810918...@gmail.com> wrote: >>>>>> Hi, >>>>>> >>>>>> >>>>>> As Xuanwo mentions the "at least three months," policy looks good to me. >>>>>> Also, it's time to upgrade to Rust 1.85 (the first version to support >>>>>> Edition 2024!). See: https://github.com/apache/iceberg-rust/pull/987 >> >> Xuanwo >> >> https://xuanwo.io/ >> Xuanwo https://xuanwo.io/
