Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread Caleb Rackliffe
I agree there’s little point in litigating right now, given test stability (or lack thereof) in cep-21-tcm. Eventually, though, I’m more or less aligned w/ David in the sense that getting ourselves planted on top of TCM as soon as possible is a good idea.On Mar 24, 2023, at 3:04 PM, Benedict wrote

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread Caleb Rackliffe
To be clear, I’m not suggesting that the CEP-7 branch be rebased on cep-21-tcm rather than trunk.On Mar 24, 2023, at 2:12 PM, Josh McKenzie wrote:FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration risk while accepting the possible delivery risk.What does the chain of rebases

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

2023-03-24 Thread Yifan Cai
Hi Jeremiah, There are good reasons to not have these inside Cassandra. Consider the following. - Resources isolation. Having the said service running within the same JVM may negatively impact Cassandra storage's performance. It could be more beneficial to have them in Sidecar, which offers strong

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

2023-03-24 Thread Doug Rohrer
I agree that the analytics library will need to support vnodes. To be clear, there’s nothing preventing the solution from working with vnodes right now, and no assumptions about a 1:1 topology between a token and a node. However, we don’t, today, have the ability to test vnode support end-to-end

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread Benedict
It’s not even clear such an effort would need to be different from that used by cep-21. The point is that there’s not much point litigating this now when we can keep our options open and better decouple the projects, since I don’t think we lose much this way.On 24 Mar 2023, at 19:58, David Capwell

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread David Capwell
Assuming we do not release with it, then yes, as we wouldn’t need to maintain. My point for this case was that I don’t feel the time cost is worth it, I am not -1 if someone wants to add, was more saying our time is better off else where. We currently don’t touch Transactional Metadata, we hav

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread Josh McKenzie
> If this is in a release, we then need to maintain that feature, so would be > against it. Isn't the argument that cep-21 provides this so we could just remove the temporary impl and point to the new facility for this generation? On Fri, Mar 24, 2023, at 3:22 PM, David Capwell wrote: >> the que

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread David Capwell
> the question we want to answer is whether or not we build a throwaway patch > for linearizable epochs If this is in a release, we then need to maintain that feature, so would be against it. If this is for testing, then I would argue the current world is “fine”… current world is hard to use a

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread Benedict
I agree, decoupling seems better to me. The fewer inter-effort dependencies, the less foot-treading.As Henrik mentioned, we don’t need a feature complete cep-15 to merge, as its integration is quite lightweight. We can leave some stuff to provide later, and depending on conditions at release time m

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread Josh McKenzie
> FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration risk > while accepting the possible delivery risk. What does the chain of rebases against trunk look like here? cep-21-tcm rebase, then cep-15 on cep-21-tcm, then cep-7 on cep-21-tcm, then a race on whichever of 15 or 7 me

Re: [EXTERNAL] [DISCUSS] Next release date

2023-03-24 Thread Josh McKenzie
> We had far less features in 4.1 and it took us 6 months to release. Definitely don't want to derail discussion but I could use some clarity. My current understanding is that the majority of our delay on 4.1 stabilization was due to ASF CI not being stable and switching to also accepting a comb

Re: [EXTERNAL] [DISCUSS] Next release date

2023-03-24 Thread Caleb Rackliffe
> That does not mean that the code should not be stabilized before the release. We had far less features in 4.1 and it took us 6 months to release. Accepting more features until August will probably result in extending the time needed to stabilize the release. I think what I'm trying to get at is

Re: [EXTERNAL] [DISCUSS] Next release date

2023-03-24 Thread Benjamin Lerer
> > SAI, Accord, UCS, and DDM are all features that can be pretty safely > disabled/not used, and unlike w/ CEP-21, we *should* be able to de-risk > those things much more easily before they merge. That does not mean that the code should not be stabilized before the release. We had far less featu

Re: [EXTERNAL] [DISCUSS] Next release date

2023-03-24 Thread Caleb Rackliffe
> I worry about the labor involved with having very large work like this target a frozen branch and then also needing to pull it up to trunk. That doesn't sound fun. > I for one do not like to have release branches cut months before their expected release. > CEP-15 is mostly “net new stuff” and n

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread Caleb Rackliffe
I actually did a dry run rebase of cep-15-accord on top of cep-21-tcm here: https://github.com/apache/cassandra/pull/2227 It wasn't too terrible, and I was actually able to get the main CQL-based Accord tests working as long as I disabled automatic forwarding of CAS and SERIAL read operations to A

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

2023-03-24 Thread Brandon Williams
On Fri, Mar 24, 2023 at 10:39 AM Jeremiah D Jordan wrote: > > I have concerns with the majority of this being in the sidecar and not in the > database itself. I think it would make sense for the server side of this to > be a new service exposed by the database, not in the sidecar. That way it

Re: [EXTERNAL] [DISCUSS] Next release date

2023-03-24 Thread Benjamin Lerer
> > I’m not sure what freezing early for “extra QA time” really buys us? Tries, SAI, UCS, extended TTLs, Java 17, Dynamic Data Masking... all those changes potentially introduce their set of issues/flakiness or other problems. The freeze will allow us to find those early and facilitate the integr

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

2023-03-24 Thread Jeremiah D Jordan
I have concerns with the majority of this being in the sidecar and not in the database itself. I think it would make sense for the server side of this to be a new service exposed by the database, not in the sidecar. That way it can be able to properly integrate with the authentication and auth

Re: Welcome our next PMC Chair Josh McKenzie

2023-03-24 Thread Dinesh Joshi
Thank you Mick for all the work you did! Welcome Josh and congratulations! On 3/23/23 01:22, Mick Semb Wever wrote: > It is time to pass the baton on, and on behalf of the Apache Cassandra > Project Management Committee (PMC) I would like to welcome and > congratulate our next PMC Chair Josh McKe

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

2023-03-24 Thread Dinesh Joshi
Hi Benjamin, I agree with your concern about long term maintenance of the code. Doug has contributed several patches to Cassandra over the years. Besides him there will be several other maintainers that will take on maintenance of this code including Yifan and myself. Given how closely it is coupl

Re: [EXTERNAL] [DISCUSS] Next release date

2023-03-24 Thread Jeremiah D Jordan
Given the fundamental change to how cluster operations work coming from CEP-21, I’m not sure what freezing early for “extra QA time” really buys us? I wouldn’t trust any multi-node QA done pre commit. What “stabilizing” do we expect to be doing during this time? How much of it do we just have

Re: Welcome our next PMC Chair Josh McKenzie

2023-03-24 Thread Patrick McFadin
Congrats Josh. This is an excellent acknowledgment of your awesome contributions to the Cassandra projects. Mick you left some big shoes to fill. Thank you for your service and for being an endless advocate for the project. Patrick On Fri, Mar 24, 2023 at 8:03 AM Paulo Motta wrote: > Thanks Mi

Re: Welcome our next PMC Chair Josh McKenzie

2023-03-24 Thread Paulo Motta
Thanks Mick and congratulations Josh!! :) On Thu, Mar 23, 2023 at 5:33 PM Erick Ramirez wrote: > Thanks Mick for everything you've done and continue to do for the project! > Congratulations Josh and thanks for stepping up! The community is in good > shape! 🍻 >

Re: [EXTERNAL] Re: [DISCUSS] Next release date

2023-03-24 Thread Josh McKenzie
> I would like to propose a partial freeze of 5.0 in June My .02: +1 to: * partial freeze on an agreed upon date w/agreed upon other things that can optionally go in after * setting a hard limit on when we ship from that frozen branch regardless of whether the features land or not -1 to: * ever

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread Josh McKenzie
> making sure that joining and leaving nodes update some state via Paxos > instead of via gossip What kind of a time delivery risk does coupling CEP-15 with CEP-21 introduce (i.e. unk-unk on CEP-21 leading to delay cascades to CEP-15)? Seems like having a table we CAS state for on epochs wouldn'

Apache TAC: assistance for travel to Berlin Buzzwords

2023-03-24 Thread Josh McKenzie
Cassandra Community! The Travel Assistance Committee with the Apache Foundation is supporting travel to Berlin Buzzwords 2023 (https://2023.berlinbuzzwords.de, 18-20 June 2023) for up to 6 people. This conference has lined up pretty well with our project in the past and would probably be a grea

Re: [EXTERNAL] Re: [DISCUSS] Next release date

2023-03-24 Thread Brandon Williams
I am +1 on a 5.0 branch freeze. Kind Regards, Brandon On Fri, Mar 24, 2023 at 8:54 AM Benjamin Lerer wrote: >> >> Would that be a trunk freeze, or freeze of a cassandra-5.0 branch? > > > I was thinking of a cassandra-5.0 branch freeze. So branching 5.0 and > allowing only CEP-15 and 21 + bug fi

Re: [EXTERNAL] Re: [DISCUSS] Next release date

2023-03-24 Thread Benjamin Lerer
> > Would that be a trunk freeze, or freeze of a cassandra-5.0 branch? I was thinking of a cassandra-5.0 branch freeze. So branching 5.0 and allowing only CEP-15 and 21 + bug fixes there. Le ven. 24 mars 2023 à 13:55, Paulo Motta a écrit : > > I would like to propose a partial freeze of 5.0 in

Re: [EXTERNAL] Re: [DISCUSS] Next release date

2023-03-24 Thread Paulo Motta
> I would like to propose a partial freeze of 5.0 in June. Would that be a trunk freeze, or freeze of a cassandra-5.0 branch? I agree with a branch freeze, but not with trunk freeze. I might work on small features after June and would be happy to delay releasing these on 5.0+, but delaying merge

Re: [EXTERNAL] Re: [DISCUSS] Next release date

2023-03-24 Thread Mick Semb Wever
> I would like to propose a partial freeze of 5.0 in June. > … > This partial freeze will be valid for every new feature except CEP-21 and > CEP-15. > +1 Thanks for summarising the thread this way Benjamin. This addresses my two main concerns: letting the branch/release date slip too much into t

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

2023-03-24 Thread Miklosovic, Stefan
Good point, Benjamin. You wrote: "library could be offered as an open source project outside of the Cassandra project itself". If Cassandra's code makes integrations like these possible (which I guess is the part of CEP), is there any reason this has to live under Cassandra project umbrella in

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

2023-03-24 Thread Benedict
Accord doesn’t have a hard dependency on CEP-21 fwiw, it just needs linearizable epochs. This could be achieved with a much more modest patch, essentially avoiding almost all of the insertion points of cep-21, just making sure that joining and leaving nodes update some state via Paxos instead of vi

Re: [EXTERNAL] Re: [DISCUSS] Next release date

2023-03-24 Thread Benjamin Lerer
Thanks everybody for the feedback. It seems to me that considering that Accord (CEP-15) depends on Transactional Cluster Metadata (CEP-21) and that, according to Sam, CEP-21 could land late July/August, Accord will probably not be fully merged before late August/September. That brings us quite late

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

2023-03-24 Thread Benjamin Lerer
Hi Doug, Outside of the changes to the Cassandra Sidecar that are mentioned, what the CEP proposes is the donation of a library for Spark integration. It seems to me that this library could be offered as an open source project outside of the Cassandra project itself. If we accept Spark Bulk Analyt