I am also on the side of separate repos for different languages. otherwise, the main repo can grow too big. iceberg.apache.org website can provide proper links to repos for different languages.
I would be -1 on renaming apache/iceberg to apache/iceberg-java, as it can break external links to the main/original github repo. the tradeoff may not be worth it. On Thu, Aug 10, 2023 at 8:16 AM Fokko Driesprong <fo...@apache.org> wrote: > Hi everyone, > > Today I took a stab at the generation of wheels in Python (here's the PR > <https://github.com/apache/iceberg/pull/8287> if anyone is interested), > and when testing this it would also kick off many unrelated CI jobs. This > is just for two languages, and I'm not convinced that it will scale to many > languages. Also, having a different release cycle for each of the languages > will clutter up the tags, releases, etc. I'm convinced that > separate repositories are more scalable in the future, we just have to make > sure that they can be found easily (rename apache/iceberg to > apache/iceberg-java?). > > Cheers, Fokko > > > > Op do 10 aug 2023 om 14:18 schreef Jan Kaul <jank...@mailbox.org.invalid>: > >> Hi all, >> >> first off, thanks Brian for starting the conversation and thanks Renjie >> for the write up. >> >> I'm also in the camp multi-repo because of the already mentioned >> benefits. >> >> One point I would like to add is that the potential drawback of having >> less visibility with multi-repos can be mitigated to some extent. I think >> that if the different repos are clearly and visibly presented on the >> iceberg website people should be able to find the desired implementation. >> >> Best wishes, >> >> Jan >> On 10.08.23 13:43, Brian Olsen wrote: >> >> Renjie, you're amazing. >> >> I think you summarized this better than I could, so thank you for that. >> >> I'd like to pull in a user's feedback on Slack >> >> FWIW, I’m personally a fan of separate repos for the client libraries. >>> It keeps things more a bit more isolated (in a good way) and explorable >>> (rather than overwhelming). GitHub search is a bit easier to use. And I >>> think it generally lowers the bar to contributing. Independent versioning, >>> and GitHub releases are a big win too, I think. >>> >> >> Right now, I don’t actually know where to find PyIceberg release notes. >>> Would love to see release notes in the GitHub releases for them. >> >> >> >> IMO, The most important measurement of success for choosing either of >> these options is about making the contributor experience as smooth as >> possible. >> >> Monorepo has the advantage of one place to look, all changes across >> core/clients can be modeled in a single PR, and sharing resources. At >> first, I considered managing the build to only be a problem for Iceberg >> committers managing the build, but ultimately this is setting us up for a >> longer build and running unnecessary infrastructure for unrelated tasks. >> There is definitely ways that we can verify what parts of the code have >> been changed and which code should be run, but it will not always be clear >> or simple to know if we tested too much or not enough. >> >> For that, I am also in the multi-repo camp (for clients). I think despite >> having to manage different repos for each client, I generally consider the >> work of each client to be independent of the work happening in the main >> repo. In this view, it's possibly better that the work be independent and >> seen on its own. The biggest win IMO is the intentional separation of >> testing and deployment infrastructure. This will make for a better >> experience when folks are contributing, testing, and looking for release >> notes. >> >> But I also really don't care as long as we do the same things across >> clients. ;) >> >> Bits >> >> >> On Thu, Aug 10, 2023 at 2:38 AM Renjie Liu <liurenjie2...@gmail.com> >> wrote: >> >>> Hi, all: >>> >>> >>> >>> In yesterday’s community sync we talked about the location of different >>> language clients, and I think we all agree that there should be consistent >>> behavior for these clients, but the decision has not been made yet. I want >>> to continue the discussion here on the pros and cons of different sides: >>> mono repo(all in one big repo) or multi small repos( one for each language >>> client) >>> >>> >>> >>> To make things clear, currently we have four language libraries under >>> development: >>> >>> >>> >>> 1. Java: in main repo(https://github.com/apache/iceberg) >>> 2. Python: in main repo (https://github.com/apache/iceberg) >>> 3. Go: in main repo (https://github.com/apache/iceberg) >>> 4. Rust: in standalone repo (https://github.com/apache/iceberg-rust/) >>> >>> >>> >>> Currently I mainly contribute rust client and I can share the thoughts >>> on why I voted for standalone repo: >>> >>> >>> >>> 1. Easier project setup. Iceberg is a complex project with several >>> components, and mainly written in java. As someone not quite familiar >>> with >>> this project structure, I feel easier to start a new one rather fitting >>> into an existing one. >>> 2. Faster ci workflow. In early days of rust client’s development, >>> we only need to touch rust related code. If we all live in one mono repo, >>> it will trigger unnecessary ci to run for other components. >>> >>> >>> >>> I admit that these reasons may not stand for long term maintains, but >>> it’s good for fast-paced development in early days. >>> >>> >>> >>> After reviewing some discussions on the web, I have a summary about the >>> pros and cons of two sides: >>> >>> >>> >>> Mono Repo >>> >>> >>> >>> Pros >>> >>> - *Visibility and transparency*. It would be easier to follow >>> progresses of all clients, and prs can have more reviews and attractions. >>> - *Easier sharing of resources*. It would be easier to share >>> resources for integration tests. >>> >>> Cons >>> >>> - *Increases complexity of project structure*. The project structure >>> would be more complex when coupling different languages and toolchain >>> setup. >>> - *Longer build/ci time. *Unnecessary ci checks maybe triggered for >>> small prs in different languages. >>> >>> >>> >>> Multi Repo >>> >>> >>> >>> Pros >>> >>> - *Simplifies project structure*. Different language may have >>> toolchains and project setup, one repo for one language makes project >>> structure easier to understand and follow. >>> - *Independent versioning and releases*. Different language may have >>> different versioning and releases process. It’s also possible in >>> monorepo, >>> but I guess it would be easier in standalone multi repo. >>> - *Improved build/ci time*. No unnecessary ci checks will be >>> triggered. >>> >>> Cons >>> >>> - *Difficult to track the overall progress. *Multi repos makes it >>> harder to track what’s happening in different teams. >>> - *Difficult to share common resources.* It maybe more difficult to >>> share resources and do integration tests cross different languages. >>> >>> >>> >>> >>> >>> Welcome to share your ideas and thoughts in this discussion! >>> >>> >>> >>> References >>> >>> >>> >>> 1. >>> >>> https://www.coforge.com/blog/mono-repo-vs.-multi-repo-in-git-unravelling-the-key-differences >>> >>>
