What are next steps here? Maybe we collectively put a table together w/the 2 operators and a list of features to compare and contrast? Enumerate the frameworks / dependencies they have to help form a point of view about the strengths and weaknesses of each option?
On Tue, Sep 29, 2020 at 10:22 PM, Christopher Bradford <bradfor...@gmail.com > wrote: > Hello Dev list, > > I'm Chris Bradford a Product Manager at DataStax working with the > cass-operator team. For background, we started down the path of developing > an operator internally to power our C*aaS platform, Astra. Care was taken > from day 1 to keep anything specific to this product at a layer above > cass-operator so it could solely focus on the task of operating Cassandra > clusters. With that being said, every single cluster on Astra is > provisioned and operated by cass-operator. The value of an advanced > operator to Cassandra users is tremendous so we decided to open source the > project (and associated components) with the goal of building a community. > It absolutely makes sense to offer this project and codebase up for > donation as a standard / baseline for running C* on Kubernetes. > > Below you will find a collection of cass-operator features, > differentiators, and roadmap / inflight initiatives. Table-stakes > Must-have functionality for a C* operator > > - > > Datacenter provisioning > - > > Schedule all pods > - > > Bootstrap nodes in the appropriate order > - > > Seeds > - > > Across racks > - > > etc. > - > > Uniform configuration > - > > Scale-up > - > > Add new nodes in a balanced manner across rack > - > > Scale-down > - > > Remove nodes one at a time across racks > - > > Node recovery > - > > Restart process > - > > Reschedule instance (IE replace node) > - Replace instance > - > > Specific workflows for seed node replacements > - > > Multi-DC / Multi-Rack > - > > Multi-Region / Multi-K8s Cluster > - > > Note this requires support at a networking layer for pod to pod IP > connectivity. This may be accomplished within the cluster with CNIs like > Cilium or externally via traditional networking tools. > > Differentiators > > - > > OSS Ecosystem / Components > - > > Cass Config Builder - OSS project extracted from DataStax OpsCenter Life > Cycle Manager to provide automated configuration file rendering > - > > Cass Config Definitions - definitions files for cass-config-builder, > defines all configuration files, their parameters, and templates > - > > Management API for Apache Cassandra (MAAC) > - > > Metrics Collector for Apache Cassandra (MCAC) > - > > Reference Prometheus Operator CRDs > - > > ServiceMonitor > - > > Instance > - > > Reference Grafana Operator CRDs > - > > Instance > - > > Dashboards > - > > Datasource > - > > PodTemplateSpec > - > > Customization of existing pods including support for adding containers, > volumes, etc > - > > Advanced Networking > - > > Node Port > - > > Host Network > - > > Simple security > - > > Management API mTLS support > - > > Automated generation of keystore and truststore for internode and client > to node TLS > - > > Automated superuser account configuration > - > > The default superuser (cassandra/cassandra) is disabled and never > available to clients > - > > Cluster administration account may be automatically (or provided) with > values stored in a k8s secret > - > > Automatic application of NetworkTopologyStrategy with appropriate RF for > system keyspaces > - > > Validating webhook > - > > Invalid changes are rejected with a helpful message > - > > Rolling cluster updates > - > > Change in binary (C* upgrade) > - > > Change in configuration > - > > Canary deployments - single rack application of changes for validation > before broader deployment > - > > Rolling restart > - > > Platform Integration / Testing / Certification > - > > Red Hat Openshift compatible and certified > - > > Secure, Universal Base Image (UBI) foundation images with security > scanning performed by Red Hat > - > > cass-operator > - > > cass-config-builder > - > > apache-cassandra w/ MCAC and MAAC > - > > Integration with Red Hat certification pipeline / marketplace > - > > Presence in Red Hat Operator Hub built into OpenShift interface > - > > VMware Tanzu Kubernetes Grid Integrated Edition compatible and certified > - > > Security scanning for images performed by VMware > - > > Amazon EKS > - > > Google GKE > - > > Azure AKS > - > > Documentation / Reference Implementations > - > > Cloud storage classes > - > > Ingress solutions > - > > Sample connection validation application with reference implementations of > Java Driver client connection parameters > - > > Cluster-level Stop / Resume - stop all running instances while keeping > persistent storage. Allows for scaling compute down to zero. Bringing the > cluster back up follows expected startup procedures > > Road Map / Inflight > > 1. > > Repair > 1. > > Reaper integration > 2. > > Backups > 1. > > Velero integration > 2. Medusa integration > 3. > > Advanced Networking via sidecar > 1. > > Combination of proxy sidecars (a la Envoy) to allow for persistent IP > addresses despite Kubernetes' best efforts to shuffle them. > 4. > > Single pod canary deployments > 5. > > Platform Certification > 1. VMware Project Pacific > > 2. > > Rancher Kubernetes Engine (K3s) > 6. > > Documentation > 1. > > Multi-region > 2. > > Multi-cloud > 3. > > Additional ingress providers > 1. Voyager > 2. HAProxy > 3. Gloo > 4. Ambassdor > 5. Envoy > 6. NGINX Ingress Controller > 4. > > Additional storage class references > 1. > > OpenEBS > 7. > > Cassandra Enhancements > 1. > > [#CASSANDRA-15823] Support for networking via identity instead of IP > - ASF JIRA <https://issues.apache.org/jira/browse/CASSANDRA-15823> > > If there are further questions about the project, codebase, architecture, > etc. the team would be happy to dive in to the details and discuss more. > > Cheers, > ~Chris > > Christopher Bradford > > On Mon, Sep 28, 2020 at 12:19 PM Patrick McFadin <pmcfa...@gmail.com> > wrote: > > I can agree with that Ben. Franck did a good job of outlining CassKop. > Somebody from the cass-operator will be posting something similar and we > can keep it on the mailing list. > > Patrick > > On Sun, Sep 27, 2020 at 2:16 PM Ben Bromhead <b...@instaclustr.com> wrote: > > Thanks Frank and Stefan. > > @Patrick great suggestion and worthwhile getting everything on the table. > > One minor change I would advocate for. The SIG has been great to iterate > and interact on the details, but I really think this conversation given > > the > > nature of the content needs to be on the mailing list. The mailing list > > is > > really our system of record and the most accessible. > > It gives folk time to think and digest, it's asynchronous, easily > searchable and let's be honest, the majority of stakeholders in this are > not US based, so the timing issue then goes away and makes it easier for > people to participate in. I feel like we've made a lot more progress by > simply having this discussion here. > > So instead of a presentation, maybe just an email to the ML addressing > > the > > headings that Patrick identified? > > On Fri, Sep 25, 2020 at 7:55 AM Stefan Miklosovic < stefan.miklosovic@ > instaclustr.com> wrote: > > Hi, > > Patrick's suggestion seems good to me. > > I won't go into specifics here as I need to genuinely prepare for this. It > is quite hard to dig deep into the solutions of others and bring some > constructive criticism because it takes a lot of time to study it and > everybody has some "why's" behind it. > > To summarize my goals and concerns: > > 1) We should be as much "Kubernetes operator idiomatic" as possible. > Industry standards, no custom brain-child of this or that group because > they think it is just cool or they just didn't know any better. I do NOT > say it is like that right now, I just want to be ruthless here as much as > possible when it comes to functionality and why it is done like that. It is > awesome that we have already something latest (thanks to John) and it > adheres to the latest releases. I personally had a hard time to keep up > with all the releases, once I finished something and I aligned it, after a > week or two there was already another one where things were different, it > is a very fast-moving space and I hope that by time we develop something it > will not be obsolete. > > 2) It may be easier said than done but it is guaranteed that people get > emotional, it's their precious etc, so please let's go into this with good > intentions, not trying to push one solution over the other just because > they would like to see it there ... I will have an equally hard time to > comply with this point. My plan is to explain what is _wrong_ with our > solution. Where we made mistakes and what should be done differently but it > is "too late" etc. It is quite hard to describe your work and all effort in > this light but without telling what is wrong we can not decide what is good > imho. > > 3) We should put something together fast enough so we can call it a > release. We can always iterate on it for eternity. But the foundations need > to be there. Here I want to say that I especially like what John did. I > looked through these specs and it was obvious it has been written with care > and attention. It looked _solid_. I am not sure how hard it is to put all > other things on top of that, I truly do not, and here I think we would have > to reinvent that wheel if we want to proceed because I can not imagine what > it would be to retrofit e.g. CassKop on top of John specs, it is just like > putting round pegs into the square holes, maybe some chunks would be reused > easily but otherwise I worry we will be just on square one. > > One specific feeling I have as I read this is that even if there is the > will to create the fourth operator, the respective parties will not be able > to drop their own repository. The whole point behind this effort, to me, is > to have a solid, community driven, stable, modern and feature complete > operator people are truly using. I can see that once this is real, we will > _really_ sunset our operator, redirecting people to the new operator on > main readme doc etc, we truly mean it. Sure, if somebody comes and bug fix > will be needed, we will fix it, but the whole point of doing this is to > stop using what we have currently, over time, otherwise we are just > splitting this space even more. If CassKop is not sure if they will use it > because they do not know if that operator will be "enough" for them, aren't > we just doing it wrong? If I exaggerate, they should be fine with deleting > the whole repository and using just this Cassandra one we are going to make > otherwise I don't see the point to work on this ... > > On Thu, 24 Sep 2020 at 20:45, Joshua McKenzie <jmcken...@apache.org> > wrote: > > - choose cass-operator: it is not on offer right now so let’s see if > > it > > does > > We should all talk a lot more, but this is 100% a mistake - I take > > the > > blame for that. The intention has long been to offer cass-operator > > for > > donation but it slipped through the cracks and your email yesterday > > made > > me > > double-take. > > We have since resolved this misalignment. DataStax would be happy to > > donate > > any and all of cass-operator to the ASF and C* project if it's what > > we > > all > > agree best serves our collective Cassandra users. I'm also cognizant > > that > > an immense amount of effort has gone into CassKop and we seem to have > something of an embarrassment of riches. > > I'm given to understand (haven't dug in personally) that the two > > operators > > express pretty different opinions when it comes to frameworks, > > designs, > > supported versions, etc. I think a discrete enumeration of the > > feature > > set > > and "identities" of both could really help navigate this conversation > > going > > forward. > > Also - thanks for that context Franck. It's always helpful to know > > where > > other people are coming from when we're all working together towards > > a > > common goal. > > On Thu, Sep 24, 2020 at 12:23 PM, <franck.de...@orange.com> wrote: > > I can share Orange’s view of the situation, sorry it is a long > > story! > > We started CassKop at the end of 2018 after betting on K8S which > > was > > not > > so simple as far as C* was concerned. Lack of support for local > > storage, > > IPs that change all the time, different network plugins to try to > > implement > > a non standard K8s way of having nodes see each other from > > different > > dcs… > > We hesitated with Mesos but could not have both and K8S was already > tracting so much you could not not choose it. > > Anyway, we looked around and did not see anyone with such > > requirements > > so > > we said: why not try it ourselves but on github so that we may give > > it > > back > > to the community. We have used C* for quite a few years with great > > success > > on production with massive load and perfect availability. We love > > C* > > @ > > Orange :) Thanks! > > So we started writing support for mono-dc cluster (CassKop) and > > added > > the > > multi dc support with MultiCassKop which is another operator > > included > > in > > the CassKop repo. For more details we tried to document our designs > > as > > much > > as possible here: > > https://orange-opensource.github.io/casskop/docs/ > > 1_concepts/3_design_principes#multi-site-management > > In the middle of last year we had some talks with Datastax about > > working > > together around their new management sidecar. Their position on > > open > > source > > was not clear at that time so we said please come back when you > > have > > decided to go open source with it. Which they did in the beginning > > of > > this > > year. But at that time I guess work had started on cass-operator so > > we > > kept > > our separate ways. > > Since the beginning of the years, we have been working with our OPS > > team > > to have it in production. It is not simple as the team has to learn > > K8S and > > trust a newborn operator. This takes time especially as our > > internal > > cluster has been tweaked for multi-tenancy with obscure options > > being > > set > > by our K8s team… > > We also developed with Instaclustr the Backup & Restore > > functionnality > > (we > > have new CRDs (Custom Resource Definition) for backup and restore > > and a > > reconcile loop that calls out Instaclustr sidecar for these > > operations). We > > now support multiple backups in parallel and can write to s3/ > > google > > or > > azur (but Stefan could give more details here if needed) > > During the SIG calls we mentioned our desire to donate CassKop once > > it > > satisfies our basics requirements (v1 coming just now but I said it > > too > > many times already) I am actually not sure Datastax mentioned their > > desire > > to donate cass-operator but we decided to compare the designs and > > the > > functionalities based on respective CRDs. The CRD is the interface > > with the > > user as it is where you describe the cluster that you want to have. > > These > > talks were very interesting and we found out that the CassKop team > > had > > made > > good choices most of the time but was may be too open. Indeed our > > intention > > was to give all the possibilities for our OPS team to work. This > > includes : > > - very open topology definition using any configuration of labels > > to > > map > > dcs / racks and nodes to labels on clusters (we have labels on dcs > > / > > rooms > > / rows and server racks so we can map C* racks to storage or > > network > > arrays > > internaly) > - possibility to have multiple C* nodes on a single K8S host > > (because > > internal clouds are not really clouds, they have limited resources) > - custom C* image selection, > - custom bootstrap script that lets you configure C* as you want > > using > > ConfigMaps, > - the ability to mount different volumes wherever they wanted, > - the possibility to run any number of sidecars alongside C* for > > custom > > probes in our case > > This makes CassKop quite powerful and flexible. > We made sure that all those options are not enabled by default so > > one > > can > > just pop a simple 3 node cluster quickly > > On the other hand cass-operator had an interesting way of > > configuring > > C* > > just inside the CRD using cass-config. This is simple and elegant > > so > > we are > > implementing it as well for the support of C* 4 > > Now for the future, there are 3 choices in my opinion: > - start from scratch (or John’s repo) by cherry picking bits from > > all > > operators. This is possible but will take some time / effort to > > have > > something usable. And then it will be compared to cass-operator and > CassKop. I don’t see Orange contributing too much here as we > > believe > > CassKop to be a much better starting point > - choose cass-operator: it is not on offer right now so let’s see > > if > > it > > does. I think Orange could contribute some bits inherited from > > CassKop > > if > > it is agreed by the community. Not sure it would be enough for us > > to > > use > > it. > - choose CassKop: we would be delighted to donate it and contribute > > with > > some committers (including the original author who now works for > > AWS). > > It > > would then become the community operator but there would be > > cass-operator > > alongside probably. But Cass-operator is made to make it easier for > Datastax to manage customer clusters by imposing some > > configuration. > > It > > make sense for their needs, so may be 2 operators. We don’t know > > how > > backup/restore will be handled here with medusa being adapted to > > K8s > > Sorry again for being long but 2 years of work deserve some lines > > of > > text > > :) > > I just saw your message Patrick but this was written already so we > > gain a > > week. > > Franck > > On 24 Sep 2020, at 10:08, Benjamin Lerer < > > benjamin.le...@datastax.com > > <mailto:benjamin.le...@datastax.com>> wrote: > > I realise there are meeting logs, but getting a wider discourse > > with > > non-stakeholder input might help to build a community consensus? It > > doesn't > > seem like it can hurt at this point, anyway. > > +1 > > On Wed, Sep 23, 2020 at 9:21 PM Benedict Elliott Smith > > <benedict@apache. > > org<mailto:bened...@apache.org>> wrote: > > Perhaps it helps to widen the field of discussion to the dev list? > > It might help if each of the stakeholder organisations state their > > view on > > the situation, including why they would or would not support a > > given > > approach/operator, and what (preferably specific) circumstances > > might > > lead > > them to change their mind? > > I realise there are meeting logs, but getting a wider discourse > > with > > non-stakeholder input might help to build a community consensus? It > > doesn't > > seem like it can hurt at this point, anyway. > > On 23/09/2020, 17:13, "John Sanda" <john.sa...@gmail.com<mailto: > > john. > > sa...@gmail.com>> wrote: > > I want to point out that pretty much everything being discussed in > > this > > thread has been discussed at length during the SIG meetings. I > > think > > it is > > worth noting because we are pretty much still have the same > > conversation. > > On Wed, Sep 23, 2020 at 12:03 PM Benedict Elliott Smith < > > benedict@apache. > > org<mailto:bened...@apache.org>> wrote: > > I don't think there's anything about a code drop that's not "The > > Apache > > Way" > > If there's a consensus (or even strong majority) amongst invested > > parties, > > I don't see why we could not adopt an operator directly into the > > project. > > It's possible a green field approach might lead to fewer hard > > feelings, as > > everyone is in the same boat. Perhaps all operators are also > > suboptimal > > and > could be improved with a rewrite? But I think coordinating a lot of > different entities around an empty codebase is particularly > > challenging. I > > actually think it could be better for cohesion and collaboration to > > have a > > suboptimal but substantive starting point. > > On 23/09/2020, 16:11, "Stefan Miklosovic" < stefan.miklosovic@ > instaclustr.com<mailto:stefan.mikloso...@instaclustr.com>> wrote: > > I think that from Instaclustr it was stated quite clearly multiple times > that we are "fine to throw it away" if there is something > > better > > and more wide-spread.Indeed, we have invested a lot of time in the > operator but it was not useless at all, we gained a lot of quite > > unique > > knowledge how to put all pieces together. However, I think that this space > is going to be quite fragmented and "balkanized", which > > is > > not always a bad thing, but in a quite narrow area as Kubernetes > > operator > > is, I just do not see how 4 operators are going to be beneficial > > for > > ordinary people ("official" from community, ours, Datastax one and > > CassKop > > (without any significant order)). Sure, innovation and healthy > > competition > > is important but to what extent ... > One can start a Cassandra cluster on Kubernetes just so many times > differently and nobody really likes a vendor lock-in. People > > wanting > > to run a cluster on K8S realise that there are three operators, > > each > > backed by a private business entity, and the community operator is > > not > > there ... Huh, interesting ... One may even start to question what > > is > > wrong with these folks that it takes three companies to build their own > solution. > > Having said that, to my perception, Cassandra community just does > > not > > have enough engineers nor contributors to keep 4 operators alive at the > same time (I wish I was wrong) so the idea of selecting the > > best > > one or to merge obvious things and approaches together is > > understandable, > > even if it meant we eventually sunset ours. In addition, nobody > > from > > big > > players is going to contribute to the code > base of the other one, for obvious reasons, so channeling and > > directing > > this effort into something common for a community seems to be the only > reasonable way of cooperation. > > It is quite hard to bootstrap this if the donation of the code in > > big > > chunks / whole repo is out of question as it is not the "Apache > > way" > > (there was some thread running here about this in more depth a > > while > > ago) and we basically need to start from scratch which is quite > demotivating, we are just inventing the wheel and nobody is up to > > it. > > It is like people are waiting for that to happen so they can jump > > in > > "once it is the thing" but it will never materialise or at least > > the > > hurdle to kick it off is unnecessarily high. Nobody is going to > > invest > > in this heavily if there is already a working operator from > > companies > > mentioned above. As I understood it, one reason of not choosing the way of > donating it all is that "the learning and community building should happen > in organic manner and we just can not accept the > > donation", > > but is not it true that it is easier to build a community around something > which is already there rather than trying to build > > it > > around an idea which is quite hard to dedicate to? > > On Wed, 23 Sep 2020 at 15:28, Joshua McKenzie < > > jmcken...@apache.org > > <mailto:jmcken...@apache.org>> wrote: > > I think there's significant value to the community in trying to coalesce > on a single approach, > I agree. Unfortunately in this case, the parties with a vested > > interest > > and > written operators came to the table and couldn't agree to coalesce on a > single approach. John Sanda attempted to start an initiative to > > write a > > best-of-breed combining choice parts of each operator, but that > > effort > > did > > not gain traction. > > Which is where my hypothesis comes from that if there were a clear > "better > fit" operator to start from we wouldn't be in a deadlock; the > > correct > > choice would be obvious. Reasonably so, every engineer that's > > written > > something is going to want that something to be used and not thrown away > in > favor of another something without strong evidence as to why that's the > better choice. > > As far as I know, nobody has made a clear case as to a more > > compelling > > place to start in terms of an operator donation the project then > collaborates on. There's no mass adoption evidence nor feature > > enumeration > > that I know of for any of the approaches anyone's taken, so the > discussions > remain stalled. > > On Wed, Sep 23, 2020 at 7:18 AM, Benedict Elliott Smith < > > benedict@apache. > > org<mailto:bened...@apache.org> wrote: > > I think there's significant value to the community in trying to coalesce > on a single approach, earlier than later. This is an opportunity to expand > the number of active organisations involved directly in the Apache > Cassandra project, as well as to more quickly expand the project's > functionality into an area we consider urgent and important. I think it > would be a real shame to waste this opportunity. No doubt it will be hard, > as organisations have certain built-in investments in their own > approaches. > > I haven't participated in these calls as I do not consider myself to have > the relevant experience and expertise, and have other focuses on the > project. I just wanted to voice a vote in favour of trying to bring > > the > > different organisations together on a single approach if possible. Is > there > anything the project can do to help this happen? > > On 23/09/2020, 03:04, "Ben Bromhead" <b...@instaclustr.com<mailto: > > ben@ > > instaclustr.com>> wrote: > > I think there is certainly an appetite to donate and standardise on a > given operator (as mentioned in this thread). > > I personally found the SIG hard to participate in due to time zones > > and > > the synchronous nature of it. > > So while it was a great forum to dive into certain details for a subset of > participants and a worthwhile endeavour, I wouldn't paint it as an > accurate > reflection of community intent. > > I don't think that any participants want to continue down the path of "let > a thousand flowers bloom". That's why we are looking towards CasKop > > (as > > well as a number of technical reasons). > > Some of the recorded meetings and outputs can also be found if you are > interested in some primary sources > https://cwiki.apache.org/confluence/display/CASSANDRA/ > Cassandra+Kubernetes+Operator+SIG > . > > From what I understand second-hand from talking to people on the SIG > calls, > > there was a general inability to agree on an existing operator as a > starting point and not much engagement on taking best of breed from the > various to combine them. Seems to leave us in the "let a thousand flowers > bloom" stage of letting operators grow in the ecosystem and seeing which > ones meet the needs of end users before talking about adopting one into > the > foundation. > > Great to hear that you folks are joining forces though! Bodes well for C* > users that are wanting to run things on k8s. > > On Tue, Sep 22, 2020 at 4:26 AM, Ben Bromhead < > > b...@instaclustr.com > > <mailto:b...@instaclustr.com> > > wrote: > > For what it's worth, a quick update from me: > > CassKop now has at least two organisations working on it > > substantially > > (Orange and Instaclustr) as well as the numerous other > > contributors. > > Internally we will also start pointing others towards CasKop once a few > things get merged. While we are not yet sunsetting our operator yet, it > > is > > certainly looking that way. > > I'd love to see the community adopt it as a starting point for working > towards whatever level of functionality is desired. > > Cheers > > Ben > > On Fri, Sep 11, 2020 at 2:37 PM John Sanda < > john.sa...@gmail.com> > wrote: > > On Thu, Sep 10, 2020 at 5:27 PM Josh McKenzie < > > jmcken...@apache.org > > wrote: > > There's basically 1 java driver in the C* ecosystem. We have 3? 4? or > > more > > operators in the ecosystem. Has one of them hit a clear > > supermajority > > of > > adoption that makes it the de facto default and makes sense to pull it > > into > > the project? > > We as a project community were pretty slow to move on building a PoV > > around > > kubernetes so we find ourselves in a situation with a bunch of contenders > for inclusion in the project. It's not clear to me what heuristics we'd > > use > > to gauge which one would be the best fit for inclusion outside letting > community adoption speak. > > --- > Josh McKenzie > > We actually talked a good bit on the SIG call earlier today about > heuristics. We need to document what functionality an operator should > include at level 0, level 1, etc. We did discuss this a good bit during > some of the initial SIG meetings, but I guess it wasn't really a focal > point at the time. I think we should also provide references to existing > operator projects and possibly other related projects. This would benefit > both community users as well as people working on these projects. > > - John > > -- > > Ben Bromhead > > Instaclustr | www.instaclustr.com | @instaclustr > <http://twitter.com/instaclustr> | (650) 284 9692 > > -- > > Ben Bromhead > > Instaclustr | www.instaclustr.com | @instaclustr > <http://twitter.com/instaclustr> | (650) 284 9692 > > --------------------------------------------------------------------- > > To > > unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional > commands, e-mail: dev-h...@cassandra.apache.org > > --------------------------------------------------------------------- > > To > > unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For > > additional > > commands, e-mail: dev-h...@cassandra.apache.org > > --------------------------------------------------------------------- > > To > > unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For > > additional > > commands, e-mail: dev-h...@cassandra.apache.org > > -- > > - John > > --------------------------------------------------------------------- > > To > > unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For > > additional > > commands, e-mail: dev-h...@cassandra.apache.org > > _________________________________________________________________________________________________________________________ > > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc pas etre > > diffuses, > > exploites ou copies sans autorisation. Si vous avez recu ce message > > par > > erreur, veuillez le signaler a l'expediteur et le detruire ainsi > > que > > les > > pieces jointes. Les messages electroniques etant susceptibles > > d'alteration, > > Orange decline toute responsabilite si ce message a ete altere, > > deforme ou > > falsifie. Merci. > > This message and its attachments may contain confidential or > > privileged > > information that may be protected by law; they should not be > > distributed, > > used or copied without authorisation. If you have received this > > email > > in > > error, please notify the sender and delete this message and its > attachments. As emails may be altered, Orange is not liable for > > messages > > that have been modified, changed or falsified. Thank you. > > --------------------------------------------------------------------- To > unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional > commands, e-mail: dev-h...@cassandra.apache.org > > -- > > Ben Bromhead > > Instaclustr | www.instaclustr.com | @instaclustr > <http://twitter.com/instaclustr> | (650) 284 9692 > >
