Re: Intra-project dependencies

On Thu, 16 Feb 2023 at 21:43, David Capwell wrote: > After a lot of effort I think this branch is in a good state, accord feels > mostly like its in-tree and all the complexity of git is hidden mostly. I > would love more feedback as the patch is in a usable state > This work is very good, tha

Re: Intra-project dependencies

After a lot of effort I think this branch is in a good state, accord feels mostly like its in-tree and all the complexity of git is hidden mostly. I would love more feedback as the patch is in a usable state > On Jan 30, 2023, at 3:16 PM, David Capwell wrote: > > I took a stab at creating a p

Re: Intra-project dependencies

I took a stab at creating a patch that I think addresses most of the comments I saw in this thread, would love feedback in https://issues.apache.org/jira/browse/CASSANDRA-18204 Given that the leading solution is git submodules I went down

Re: Intra-project dependencies

> Both a git post-checkout and a build fail-fast will protect us here. But >>> the post-checkout will need to fail silently if the .git subdirectory >>> doesn't exist. >>> >> >> Correction: the build fail-fast will need to fail silently if the .git >> subdirectory doesn't exist. >> > > How will thi

Re: Intra-project dependencies

On Fri, Jan 20, 2023, 8:31 AM Mick Semb Wever wrote: > Both a git post-checkout and a build fail-fast will protect us here. But >> the post-checkout will need to fail silently if the .git subdirectory >> doesn't exist. >> > > > Correction: the build fail-fast will need to fail silently if the .gi

Re: Intra-project dependencies

> > Both a git post-checkout and a build fail-fast will protect us here. But > the post-checkout will need to fail silently if the .git subdirectory > doesn't exist. > Correction: the build fail-fast will need to fail silently if the .git subdirectory doesn't exist.

Re: Intra-project dependencies

Thanks Mick and David. I've been following this silently for a few days because we already exhausted my knowledge on the topic. But it seems your collective knowledge is uncovering a nice solution. If I summarize, I like all of this: - link to SHA, not library version - use git submodules because

Re: Intra-project dependencies

replies are inline to your inline replies to my inline replies 🥁 > We can ask INFRA to set up a separate snapshots repository just for us, > with a longer expiry policy. I'd rather not create extra work for infra if > there's other ways we can do this, and this approach would always require > so

Re: Intra-project dependencies

Thanks for the reply, my replies are inline to your inline replies =D > On Jan 19, 2023, at 2:39 PM, Mick Semb Wever wrote: > > > Thanks David for the detailed write up. Replies inline… > > > We tried in-tree for in-jvm dtest and found that this broke every other > commit… maintaining the A

Re: Intra-project dependencies

Thanks David for the detailed write up. Replies inline… > We tried in-tree for in-jvm dtest and found that this broke every other > commit… maintaining the APIs across all our supported branches was too hard > to do and moving it outside of the tree helped make the upgrade tests more > stable (t

Re: Intra-project dependencies

If we make sure all branches are using the latest “stable” accord then this is 6 commits (4 for C*, 1 for accord the stable branch, then 1 to merge into trunk)If we’re modifying stable, we only need one commit per C* branch per release. We don’t need to immediately point C* to it. So there could pl

Re: Intra-project dependencies

Been out, sorry for just catching up now… I feel this thread pidgin hold on the word Accord and ignored the fact we are dealing with this pain today with python/jvm dtest and trying to improve that would help the project…. We also have other related projects that we are developing in parallel t

Re: Intra-project dependencies

> Linking or merging while it is still also being a separate library and repo. I am still unclear why you think this is “a significant thing”? > On 18 Jan 2023, at 10:41, Mick Semb Wever wrote: > >  > > >>> You would reference the snapshot dependency by the timestamped snapshot. >>> This ma

Re: Intra-project dependencies

You would reference the snapshot dependency by the timestamped snapshot. > This makes it a reproducible build. > > > How confident are we that the repository will not alter or delete them? > They cannot be altered. I see artefacts there that are more than a decade old. But we cannot rely on thei

Re: Intra-project dependencies

On Tue, Jan 17, 2023 at 11:40 PM Mick Semb Wever wrote: > >> It introduces some overhead when bisecting to go from the snapshot's > timestamp to the other repo's SHA (this is easily solvable by putting the > SHA inside the jarfile). > Whatever system we choose, the link should be the SHA. It sho

Re: Intra-project dependencies

> You would reference the snapshot dependency by the timestamped snapshot. This > makes it a reproducible build. How confident are we that the repository will not alter or delete them? > linking in the source code into in-tree is a significant thing to do Could you explain why? I thought your p

Re: Intra-project dependencies

> Josh, bundling releases gets tricky in that you need to include the library > sources, because the cassandra release is essentially being voted on (because > it has been built) with non-released dependencies. Arguably, one shouldn't vote on a release of Accord unless there's something that's i

Re: Intra-project dependencies

> > Regarding the use of snapshots, this isn’t impossible Henrik - I floated > this as an option. But besides the additional overhead during development, > this does not maintain reproducible builds, as the snapshot can change. > You would reference the snapshot dependency by the timestamped snaps

Re: Intra-project dependencies

I am certainly not proposing any certainty about outside interest, but I think as the only full implementation of a leaderless protocol in existence, as well as an open source pluggable distributed transaction protocol, the chance of some interest is not vanishingly small (once it is proven in Cass

Re: Intra-project dependencies

Is there any reason we couldn't "bundle" a release vote to include both an Accord release and ASF C* in one voting round as a combined release? My reading of the release process w/the ASF doesn't speak to that (if anything it implies this might be a valid approach): https://www.apache.org/legal

Re: Intra-project dependencies

Hi Derek Somewhat of a newcomer myself, it seems the answers to your excellent questions are: * We don't all agree with the premise that Accord will attract substantial outside interest. Even so, my personal opinion is that whether that happens or not, it's not wrong to aspire toward or plan for

Re: Intra-project dependencies

Actually, re-reading the thread, I think I missed the initial point brought up and got lost in the discussion specific to submodules. What is the technical reason for bringing Accord in-tree? While I think submodules are the best way to include source in-tree, I'm not sure this is actually the corr

Re: Intra-project dependencies

I'd like to go back to Benedict's initial point: if we have a new consensus protocol that other projects would potentially be interested in, then by all means it should be its own project. Let's start with that as a basis for discussion, because from my reading it seems like people might be disagre

Re: Intra-project dependencies

The answer to all your questions is “like any other library” - this is a procedural hack to ease development. There are alternative isomorphic hacks, like compiling source jars from Accord and including them in the C* tree, if it helps your mental model. > you stated that a goal was to avoid ma

Re: Intra-project dependencies

> > … extrapolating this experience to multiple C* versions > > To include forward-merging, bisecting old history, etc etc. that's a leap of faith that I believe deserves the discussion. - patches are off submodule SHAs, not the submodule's HEAD, > > > A SHA would point to the HEAD of a given bran

Re: Intra-project dependencies

 Benedict, experience based on developing one feature against one branch doesn't face the problems of working, and switching frequently, between branches.Mick, please take a look at the ongoing development. Over the last week I have been actively developing five separate PRs against each repository

Re: Intra-project dependencies

> - permanence from a git SHA no longer exists > > With the caveat that I haven't worked w/submodules before and only know > about them from a cursory search, it looks like git-submodule status would > show us the sha for submodules and … > That isn't one SHA, but a collection of SHAs. I'm thin

Re: Intra-project dependencies

We have a build script that is invoked by ant to grab a specific SHA (or HEAD of a branch). We were previously just grabbing HEAD but this has the problems mentioned elsewhere in the thread, amongst others. I don’t think it probably matters much if we use a build script or submodules.I am driven in

Re: Intra-project dependencies

Hi Benedict At least for my part, again, I'm not (yet) trying to argue for or against a particular alternative. So I think you'll find that if you allow a few more iterations of discussion, we can gravitate to some good consensus. Or failing that, we can at least gravitate around a small number o

Re: Intra-project dependencies

How often have we modified Paxos? There are currently no proposals to develop Accord further after the initial release. So I think it is very likely that Accord development will decouple from Cassandra version, unless there is significant external interest that drives it.Furthermore, the idea of re

Re: Intra-project dependencies

Hi all I was invited to share my thoughts just as an additional and somewhat fresh point of view... On a high level: We talked through this with Mick and a few other colleagues, and I/we came to the conclusion that fundamentally all of the mentioned options 1-5 are just variations of the same pro

Re: Intra-project dependencies

> - permanence from a git SHA no longer exists With the caveat that I haven't worked w/submodules before and only know about them from a cursory search, it looks like git-submodule status would show us the sha for submodules and we could have parent projects reference specific shas to pull for

Re: Intra-project dependencies

I guess option 5 is what we have today in cep-15, have the build file grab the relevant SHA for the library. This way you maintain a precise SHA for builds and scripts don’t have to be modified. I believe this is also possible with git submodules, but I’m happy to bake this into our build file

Re: Intra-project dependencies

> > I think (4) is the only sensible option. It permits different development > branches to easily reference different versions of a library and also to > easily co-develop them - from within the same IDE project, even. > I've only heard horror stories about submodules. The challenges they bring

Intra-project dependencies

Those of us who have developed the in-jvm-dtest-api will know that the project’s approach to developing libraries is untenable for more complex projects. Accord makes this a pressing concern, but we would also benefit from separating utilities to their own library for use by the dtest-api and Ac