Re: Roadmap for 4.0

[Ben Bromhead]

o we just review the outstanding improvements and
>>> decide whether it's worth pushing it back or if we've got enough to
>>> release. I suppose keep this time frame in mind when choosing your tickets.
>>>
>>> We can aim for an earlier date (midyear?) but I figure the
>>> testing/validation/bugfixing period prior to release might drag on a bit so
>>> being a bit conservative here.
>>> The main goal would be to not let list 1 grow unless we're well ahead,
>>> and only cull from it if we're heavily over-committed or we decide the
>>> improvement can wait. I assume this all sounds like common sense but
>>> figured it's better to spell it out now.
>>>
>>>
>>> NEXT STEPS
>>> After 2 weeks/whenever the discussion dies off I'll consolidate all the
>>> tickets, relevant comments and follow up with a summary, where we can
>>> discuss/nitpick issues and come up with a final list to go ahead with.
>>>
>>> On a side note, in conjunction with this effort we'll obviously have to
>>> do something about validation and testing. I'll keep that out of this email
>>> for now, but there will be a follow up so that those of us willing to help
>>> validate/test trunk can avoid duplicating effort.
>>>
>>> REVIEW
>>> This is the list of "huge/breaking" tickets that got mentioned in the
>>> last roadmap discussion and their statuses. This is not terribly important
>>> but just so we can keep in mind what we previously talked about. I think we
>>> leave it up to the relevant contributors to decide whether they want to get
>>> the still open tickets into 4.0.
>>>
>>> CASSANDRA-9425 Immutable node-local schema
>>> <https://issues.apache.org/jira/browse/CASSANDRA-9425> - Committed
>>> CASSANDRA-10699 Strongly consistent schema alterations
>>> <https://issues.apache.org/jira/browse/CASSANDRA-10699> - Open, no
>>> discussion in quite some time.
>>> CASSANDRA-12229 NIO streaming
>>> <https://issues.apache.org/jira/browse/CASSANDRA-12229> - Committed
>>> CASSANDRA-8457 NIO messaging
>>> <https://issues.apache.org/jira/browse/CASSANDRA-8457> - Committed
>>> CASSANDRA-12345 Gossip 2.0
>>> <https://issues.apache.org/jira/browse/CASSANDRA-12345> - Open, no sign
>>> of any action.
>>> CASSANDRA-9754 Make index info heap friendly for large CQL partitions
>>> <https://issues.apache.org/jira/browse/CASSANDRA-9754> - In progress
>>> but no update in a long time.
>>> CASSANDRA-11559 enhanced node representation
>>> <https://issues.apache.org/jira/browse/CASSANDRA-11559> - Open, no
>>> change since early 2016.
>>> CASSANDRA-6246 epaxos
>>> <https://issues.apache.org/jira/browse/CASSANDRA-6246> - In progress
>>> but no update since Feb 2017.
>>> CASSANDRA-7544 storage port configurable per node
>>> <https://issues.apache.org/jira/browse/CASSANDRA-7544> - Committed
>>> CASSANDRA-11115 remove thrift support
>>> <https://issues.apache.org/jira/browse/CASSANDRA-5> - Committed
>>> CASSANDRA-10857 dropping compact storage
>>> <https://issues.apache.org/jira/browse/CASSANDRA-10857> - Committed
>>>
>>> To start us off...
>>> And here are my lists to get us started.
>>> 1.
>>> CASSANDRA-8460 - Tiered/Cold storage for TWCS
>>> <https://issues.apache.org/jira/browse/CASSANDRA-8460>
>>> CASSANDRA-12783 - Batchlog redesign
>>> <https://issues.apache.org/jira/browse/CASSANDRA-12783>
>>> CASSANDRA-11559 - Enchance node representation
>>> <https://issues.apache.org/jira/browse/CASSANDRA-11559>
>>> CASSANDRA-12344 - Forward writes to replacement node with same
>>> address <https://issues.apache.org/jira/browse/CASSANDRA-12344>
>>> CASSANDRA-8119 - More expressive Consistency Levels
>>> <https://issues.apache.org/jira/browse/CASSANDRA-8119>
>>> CASSANDRA-14210 - Optimise SSTables upgrade task scheduling
>>> <https://issues.apache.org/jira/browse/CASSANDRA-14210>
>>> CASSANDRA-10540 - RangeAwareCompaction
>>> <https://issues.apache.org/jira/browse/CASSANDRA-10540>
>>>
>>>
>>> 2:
>>> CASSANDRA-10726 - Read repair inserts should not be blocking
>>> <https://issues.apache.org/jira/browse/CASSANDRA-10726>
>>> CASSANDRA-9754 - Make index info heap friendly for large CQL partitions
>>> <https://issues.apache.org/jira/browse/CASSANDRA-9754>
>>> CASSANDRA-12294 - LDAP auth
>>> <https://issues.apache.org/jira/browse/CASSANDRA-12294>
>>> CASSANDRA-12151 - Audit logging
>>> <https://issues.apache.org/jira/browse/CASSANDRA-12151>
>>> CASSANDRA-10495 - Fix streaming with vnodes
>>> <https://issues.apache.org/jira/browse/CASSANDRA-10495>
>>>
>>> Also, here's some handy JQL to start you off:
>>> project = CASSANDRA AND fixVersion in (4.x, 4.0) AND issue in
>>> watchedIssues() AND status != Resolved
>>>
>>>
>>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: datastax cassandra minimum hardware recommendation

[Ben Bromhead]

Also, DS charge by core ;)

Anecdotally, we run a large fleet of Apache C* nodes on AWS with a good
portion of supported instances that run with 16GB of RAM and 4 cores, which
is fine for those workloads.

On Wed, Apr 4, 2018 at 11:08 AM sujeet jog  wrote:

> Thanks Alain
>
> On Wed, Apr 4, 2018 at 3:12 PM, Alain RODRIGUEZ 
> wrote:
>
>> Hello.
>>
>> For questions to Datastax, I recommend you to ask them directly. I often
>> had a quick answer and they probably can answer this better than we do :).
>>
>> Apache Cassandra (and probably DSE-Cassandra) can work with 8 CPU (and
>> less!). I would not go much lower though. I believe the memory amount and
>> good disk throughputs are more important. It also depends on the
>> workload type and intensity, encryption, compression etc.
>>
>> 8 CPUs is probably just fine if well tuned, and here in the mailing list,
>> we 'support' any fancy configuration settings, but with no guarantee on the
>> response time and without taking the responsibility for your cluster :).
>>
>> It reminds me of my own start with Apache Cassandra. I started with
>> t1.micro back then on AWS, and people were still helping me here, of course
>> after a couple of jokes such as 'you should rather try to play a
>> PlayStation 4 game in your Gameboy', that's fair enough I guess :). Well it
>> was working in prod and I learned how to tune Apache Cassandra, I had no
>> other options to have this working.
>>
>> Having more CPU probably improves resiliency to some problems and reduces
>> the importance of having a cluster perfectly tuned.
>>
>> Benchmark your workload, test it. This would be the most accurate answer
>> here given the details we have.
>>
>> C*heers,
>> ---
>> Alain Rodriguez - @arodream - al...@thelastpickle.com
>> France / Spain
>>
>> The Last Pickle - Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>> 2018-04-04 9:44 GMT+01:00 sujeet jog :
>>
>>> the datastax site has a hardware recommendation of 16CPU / 32G RAM for
>>> DSE Enterprise,  Any idea what is the minimum hardware recommendation
>>> supported, can each node be 8CPU and the support covering it ?..
>>>
>>
>>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Sorl/DSE Spark

[Ben Bromhead]

Folks this is the user list for Apache Cassandra. I would suggest
redirecting the question to Datastax the commercial entity that produces
the software.

On Thu, Apr 12, 2018 at 9:51 AM vincent gromakowski <
vincent.gromakow...@gmail.com> wrote:

> Best practise is to use a dedicated DC for analytics separated from the
> hot DC.
>
> Le jeu. 12 avr. 2018 à 15:45, sha p  a écrit :
>
>> Got it.
>> Thank you so for your detailed explanation.
>>
>> Regards,
>> Shyam
>>
>> On Thu, 12 Apr 2018, 17:37 Evelyn Smith,  wrote:
>>
>>> Cassandra tends to be used in a lot of web applications. It’s loads are
>>> more natural and evenly distributed. Like people logging on throughout the
>>> day. And people operating it tend to be latency sensitive.
>>>
>>> Spark on the other hand will try and complete it’s tasks as quickly as
>>> possible. This might mean bulk reading from the Cassandra at 10 times the
>>> usual operations load, but for only say 5 minutes every half hour (however
>>> long it takes to read in the data for a job and whenever that job is run).
>>> In this case during that 5 minutes your normal operations work (customers)
>>> are going to experience a lot of latency.
>>>
>>> This even happens with streaming jobs, every time spark goes to interact
>>> with Cassandra it does so very quickly, hammers it for reads and then does
>>> it’s own stuff until it needs to write things out. This might equate to
>>> intermittent latency spikes.
>>>
>>> In theory, you can throttle your reads and writes but I don’t know much
>>> about this and don’t see people actually doing it.
>>>
>>> Regards,
>>> Evelyn.
>>>
>>> On 12 Apr 2018, at 4:30 pm, sha p  wrote:
>>>
>>> Evelyn,
>>> Can you please elaborate on below
>>> Spark is notorious for causing latency spikes in Cassandra which is not
>>> great if you are are sensitive to that.
>>>
>>>
>>> On Thu, 12 Apr 2018, 10:46 Evelyn Smith,  wrote:
>>>
>>>> Are you building a search engine -> Solr
>>>> Are you building an analytics function -> Spark
>>>>
>>>> I feel they are used in significantly different use cases, what are you
>>>> trying to build?
>>>>
>>>> If it’s an analytics functionality that’s seperate from your operations
>>>> functionality I’d build it in it’s own DC. Spark is notorious for causing
>>>> latency spikes in Cassandra which is not great if you are are sensitive to
>>>> that.
>>>>
>>>> Regards,
>>>> Evelyn.
>>>>
>>>> On 12 Apr 2018, at 6:55 am, kooljava2 
>>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> We are exploring on configuring Sorl/Spark. Wanted to get input on
>>>> this.
>>>> 1) How do we decide which one to use?
>>>> 2) Do we run this on a DC where there is less workload?
>>>>
>>>> Any other suggestion or comments are appreciated.
>>>>
>>>> Thank you.
>>>>
>>>>
>>>>
>>> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Sorl/DSE Spark

[Ben Bromhead]

On Thu, Apr 12, 2018, 21:23 Niclas Hedhman  wrote:

> Ben,
>
> 1. I don't see anything in this thread that is DSE specific, so I think it
> belongs here.
>
DSE is literally in the title.


> 2. Careful when you say that Datastax produces Cassandra. Cassandra is a
> product of Apache Software Foundation, and no one else. You, Ben, should be
> very well aware of this, to avoid further trademark issues between Datastax
> and ASF.
>
Given the context and subject, the software I was referring to is DSE.

Mind you, it would be hilarious if this email caused more trademark issues
with Datastax.



> Cheers
> Niclas Hedhman
> Member of ASF
>
> On Thu, Apr 12, 2018 at 9:57 PM, Ben Bromhead  wrote:
>
>> Folks this is the user list for Apache Cassandra. I would suggest
>> redirecting the question to Datastax the commercial entity that produces
>> the software.
>>
>> On Thu, Apr 12, 2018 at 9:51 AM vincent gromakowski <
>> vincent.gromakow...@gmail.com> wrote:
>>
>>> Best practise is to use a dedicated DC for analytics separated from the
>>> hot DC.
>>>
>>> Le jeu. 12 avr. 2018 à 15:45, sha p  a écrit :
>>>
>>>> Got it.
>>>> Thank you so for your detailed explanation.
>>>>
>>>> Regards,
>>>> Shyam
>>>>
>>>> On Thu, 12 Apr 2018, 17:37 Evelyn Smith,  wrote:
>>>>
>>>>> Cassandra tends to be used in a lot of web applications. It’s loads
>>>>> are more natural and evenly distributed. Like people logging on throughout
>>>>> the day. And people operating it tend to be latency sensitive.
>>>>>
>>>>> Spark on the other hand will try and complete it’s tasks as quickly as
>>>>> possible. This might mean bulk reading from the Cassandra at 10 times the
>>>>> usual operations load, but for only say 5 minutes every half hour (however
>>>>> long it takes to read in the data for a job and whenever that job is run).
>>>>> In this case during that 5 minutes your normal operations work (customers)
>>>>> are going to experience a lot of latency.
>>>>>
>>>>> This even happens with streaming jobs, every time spark goes to
>>>>> interact with Cassandra it does so very quickly, hammers it for reads and
>>>>> then does it’s own stuff until it needs to write things out. This might
>>>>> equate to intermittent latency spikes.
>>>>>
>>>>> In theory, you can throttle your reads and writes but I don’t know
>>>>> much about this and don’t see people actually doing it.
>>>>>
>>>>> Regards,
>>>>> Evelyn.
>>>>>
>>>>> On 12 Apr 2018, at 4:30 pm, sha p  wrote:
>>>>>
>>>>> Evelyn,
>>>>> Can you please elaborate on below
>>>>> Spark is notorious for causing latency spikes in Cassandra which is
>>>>> not great if you are are sensitive to that.
>>>>>
>>>>>
>>>>> On Thu, 12 Apr 2018, 10:46 Evelyn Smith,  wrote:
>>>>>
>>>>>> Are you building a search engine -> Solr
>>>>>> Are you building an analytics function -> Spark
>>>>>>
>>>>>> I feel they are used in significantly different use cases, what are
>>>>>> you trying to build?
>>>>>>
>>>>>> If it’s an analytics functionality that’s seperate from your
>>>>>> operations functionality I’d build it in it’s own DC. Spark is notorious
>>>>>> for causing latency spikes in Cassandra which is not great if you are are
>>>>>> sensitive to that.
>>>>>>
>>>>>> Regards,
>>>>>> Evelyn.
>>>>>>
>>>>>> On 12 Apr 2018, at 6:55 am, kooljava2 
>>>>>> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> We are exploring on configuring Sorl/Spark. Wanted to get input on
>>>>>> this.
>>>>>> 1) How do we decide which one to use?
>>>>>> 2) Do we run this on a DC where there is less workload?
>>>>>>
>>>>>> Any other suggestion or comments are appreciated.
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>> Ben Bromhead
>> CTO | Instaclustr <https://www.instaclustr.com/>
>> +1 650 284 9692
>> Reliability at Scale
>> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>>
>
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Sorl/DSE Spark

[Ben Bromhead]

Thanks Jeff.

On Thu, Apr 12, 2018, 21:37 Jeff Jirsa  wrote:

> Pretty sure Ben meant that datastax produces DSE, not Cassandra, and since
> the questions specifically mentions DSE in the subject (implying that the
> user is going to be running either solr or spark within DSE to talk to
> cassandra), Ben’s recommendation seems quite reasonable to me.
>
>
>
> --
> Jeff Jirsa
>
>
> On Apr 12, 2018, at 6:23 PM, Niclas Hedhman  wrote:
>
> Ben,
>
> 1. I don't see anything in this thread that is DSE specific, so I think it
> belongs here.
>
> 2. Careful when you say that Datastax produces Cassandra. Cassandra is a
> product of Apache Software Foundation, and no one else. You, Ben, should be
> very well aware of this, to avoid further trademark issues between Datastax
> and ASF.
>
> Cheers
> Niclas Hedhman
> Member of ASF
>
> On Thu, Apr 12, 2018 at 9:57 PM, Ben Bromhead  wrote:
>
>> Folks this is the user list for Apache Cassandra. I would suggest
>> redirecting the question to Datastax the commercial entity that produces
>> the software.
>>
>> On Thu, Apr 12, 2018 at 9:51 AM vincent gromakowski <
>> vincent.gromakow...@gmail.com> wrote:
>>
>>> Best practise is to use a dedicated DC for analytics separated from the
>>> hot DC.
>>>
>>> Le jeu. 12 avr. 2018 à 15:45, sha p  a écrit :
>>>
>>>> Got it.
>>>> Thank you so for your detailed explanation.
>>>>
>>>> Regards,
>>>> Shyam
>>>>
>>>> On Thu, 12 Apr 2018, 17:37 Evelyn Smith,  wrote:
>>>>
>>>>> Cassandra tends to be used in a lot of web applications. It’s loads
>>>>> are more natural and evenly distributed. Like people logging on throughout
>>>>> the day. And people operating it tend to be latency sensitive.
>>>>>
>>>>> Spark on the other hand will try and complete it’s tasks as quickly as
>>>>> possible. This might mean bulk reading from the Cassandra at 10 times the
>>>>> usual operations load, but for only say 5 minutes every half hour (however
>>>>> long it takes to read in the data for a job and whenever that job is run).
>>>>> In this case during that 5 minutes your normal operations work (customers)
>>>>> are going to experience a lot of latency.
>>>>>
>>>>> This even happens with streaming jobs, every time spark goes to
>>>>> interact with Cassandra it does so very quickly, hammers it for reads and
>>>>> then does it’s own stuff until it needs to write things out. This might
>>>>> equate to intermittent latency spikes.
>>>>>
>>>>> In theory, you can throttle your reads and writes but I don’t know
>>>>> much about this and don’t see people actually doing it.
>>>>>
>>>>> Regards,
>>>>> Evelyn.
>>>>>
>>>>> On 12 Apr 2018, at 4:30 pm, sha p  wrote:
>>>>>
>>>>> Evelyn,
>>>>> Can you please elaborate on below
>>>>> Spark is notorious for causing latency spikes in Cassandra which is
>>>>> not great if you are are sensitive to that.
>>>>>
>>>>>
>>>>> On Thu, 12 Apr 2018, 10:46 Evelyn Smith,  wrote:
>>>>>
>>>>>> Are you building a search engine -> Solr
>>>>>> Are you building an analytics function -> Spark
>>>>>>
>>>>>> I feel they are used in significantly different use cases, what are
>>>>>> you trying to build?
>>>>>>
>>>>>> If it’s an analytics functionality that’s seperate from your
>>>>>> operations functionality I’d build it in it’s own DC. Spark is notorious
>>>>>> for causing latency spikes in Cassandra which is not great if you are are
>>>>>> sensitive to that.
>>>>>>
>>>>>> Regards,
>>>>>> Evelyn.
>>>>>>
>>>>>> On 12 Apr 2018, at 6:55 am, kooljava2 
>>>>>> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> We are exploring on configuring Sorl/Spark. Wanted to get input on
>>>>>> this.
>>>>>> 1) How do we decide which one to use?
>>>>>> 2) Do we run this on a DC where there is less workload?
>>>>>>
>>>>>> Any other suggestion or comments are appreciated.
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>> Ben Bromhead
>> CTO | Instaclustr <https://www.instaclustr.com/>
>> +1 650 284 9692
>> Reliability at Scale
>> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>>
>
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Cassandra datastax cerrification

[Ben Bromhead]

Certification is only as good as the organizations that recognize it.
Identify what you want to get out of certification, whether that
certificate will get what you want and make a decision based on that. The
delivered content can be the best in the world, but if no one recognizes it
then the certification element becomes meaningless.

I would be asking any certification provider which companies actually
accept it or recognize it as a demonstration of skill. Especially if you
are going to pay money for it.

On the other hand if you are just looking to build knowledge / get some
classroom experience then you can look at the various options with a more
subjective approach.

This is just my 2c based on a past life in the IT Security world where
"certifications" are a massive business and acceptance of certifications
can vary place to place and not a commentary on any of the certification
providers you directly mentioned :)

On Sat, Apr 14, 2018 at 11:09 AM Abdul Patel  wrote:

> Hi All,
>
> I am preparing for cassandra certification(dba) orielly has stopped the
> cassandra cerrification so the best bet is datastax now ..as per my
> knwledge ds201 and 220 should be enough for cerrification and also i am
> reading definitive guide on cassandra ..any other material required ? Any
> practise test websites? As certification is costly and wanna clear in one
> go ...
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Cassandra Summit 2019 / Cassandra Summit 2018

[Ben Bromhead]

Distributed Data Day is shaping up to look like a Cassandra related summit.
See http://distributeddatasummit.com/2018-sf/speakers

You'll most likely see all the same familiar faces and familiar vendors :)

However note it is not the official Apache Cassandra Summit and it is being
run by a for profit group.

On Thu, May 3, 2018 at 5:33 AM Horia Mocioi 
wrote:

> Hello,
>
> Are there any updates on this event?
>
> Regards,
> Horia
>
> On tis, 2018-02-27 at 11:42 +, Carlos Rolo wrote:
>
> Hello all,
>
> I'm interested planning/organizing a small kinda of NGCC in Lisbon,
> Portugal in late May early June. Just waiting for the venue to confirm
> possible dates.
>
> Would be a 1day event kinda last year, is this something people would be
> interested? I can push a google form for accessing the interest today.
>
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +351 918 918 100 <+351%20918%20918%20100>
> www.pythian.com
>
> On Tue, Feb 27, 2018 at 11:39 AM, Kenneth Brotman <
> kenbrot...@yahoo.com.invalid> wrote:
>
> Event planning is fun as long as you can pace it out properly.  Once you
> set a firm date for an event the pressure on you to keep everything on
> track is nerve racking.  To do something on the order of Cassandra Summit
> 2016, I think we are should plan for 2020.  It’s too late for 2018 and even
> trying to meet the timeline for everything that would have to come together
> makes 2019 too nerve racking a target date.  The steps should be:
>
> Form a planning committee
>
> Bring potential sponsors into the planning early
>
> Select an event planning vendor to guide us and to do the
> heavy lifting for us
>
>
>
> In the meantime, we could have a World-wide Distributed Asynchronous
> Cassandra Convention which offers four benefits:
>
> It allows us to address the fact that we are a world-wide
> group that needs a way to reach everyone in a way where no one is
> geographically disadvantaged
>
> No travel time, no travel expenses and no ticket fees
> makes it accessible to a lot of people that otherwise would have to miss out
>
> The lower production costs and simpler administrative workload allows us
> to reach implementation sooner
>
> It’s cutting edge, world class innovation like Cassandra
>
>
>
> Kenneth Brotman
>
>
>
> *From:* Jeff Jirsa [mailto:jji...@gmail.com]
> *Sent:* Monday, February 26, 2018 9:38 PM
> *To:* cassandra
> *Subject:* Re: Cassandra Summit 2019 / Cassandra Summit 2018
>
>
>
> Instaclustr sponsored the 2017 NGCC (Next Gen Cassandra Conference), which
> was developer/development focused (vs user focused).
>
>
>
> For 2018, we're looking at options for both a developer conference and a
> user conference. There's a lot of logistics involved, and I think it's
> fairly obvious that most of the PMC members aren't professional event
> planners, so it's possible that either/both conferences may not happen, but
> we're doing our best to try to put something together.
>
>
>
>
>
> On Mon, Feb 26, 2018 at 3:00 PM, Rahul Singh 
> wrote:
>
> I think some of the Instaclustr folks had done one last year which I
> really wanted to go to.. Distributed / Async both would be easier to get
> people to write papers, make slides, do youtube videos with.. and then we
> could do a virtual web conf of the best submissions.
>
>
> On Feb 26, 2018, 1:04 PM -0600, Kenneth Brotman <
> kenbrot...@yahoo.com.invalid>, wrote:
>
> Is there any planning yet for a Cassandra Summit 2019 or Cassandra Summit
> 2018 (probably too late)?
>
>
>
> Is there a planning committee?
>
>
>
> Who wants there to be a Cassandra Summit 2019 and who thinks there is a
> better way?
>
>
>
> We could try a Cassandra Distributed Summit 2019 where we meet virtually
> and perhaps asynchronously, but there would be a lot more energy and
> bonding if it’s not virtual.  I’m up for any of these.
>
>
>
> Kenneth Brotman
>
>
>
>
>
>
>
>
>
> --
>
>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Using K8s to Manage Cassandra in Production

[Ben Bromhead]

You can get a good way with StatefulSets, but as Tom mentioned there are
still some issues with this, particularly around scaling up and down.

We are working on an Operator for Apache Cassandra, you can find it here
https://github.com/instaclustr/cassandra-operator. This is a joint project
between Instaclustr, Pivotal and a few other folk.

Currently it's a work in progress, but we would love any or all early
feedback/PRs/issues etc. Our first GA release will target the following
capabilities:

   - Safe scaling up and down (including decommissioning)
   - Backup/restore workflow (snapshots only initially)
   - Built in prometheus integration and discovery

Other features like repair, better PV support, maybe even a nice dashboard
will be on the way.


On Wed, May 23, 2018 at 7:35 AM Tom Petracca  wrote:

> Using a statefulset should get you pretty far, though will likely be less
> effective than a coreos-style “operator”. Some random points:
>
>- For scale-up: a node shouldn’t report “ready” until it’s in the
>NORMAL state; this will prevent multiple nodes from bootstrapping at once.
>- For scale-down: as of now there isn’t a mechanism to know if a pod
>is getting decommissioned because you’ve permanently lowered replica count,
>or because it’s just getting bounced/re-scheduled, thus knowing whether or
>not to decommission is basically impossible. Relevant issue:
>kubernetes/kubernetes#1462
><https://github.com/kubernetes/kubernetes/issues/1462>
>
>
>
> *From: *Pradeep Chhetri 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Friday, May 18, 2018 at 10:20 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Using K8s to Manage Cassandra in Production
>
>
>
> Hello Hassaan,
>
>
>
> We use cassandra helm chart[0] for deploying cassandra over kubernetes in
> production. We have around 200GB cas data. It works really well. You can
> scale up nodes easily (I haven't tested scaling down).
>
>
>
> I would say that if you are worried about running cassandra over k8s in
> production, maybe you should first try setting it for your
> staging/preproduction and gain confidence over time.
>
>
>
> I have tested situations where i have killed the host running cassandra
> container and have seen that container moves to a different node and joins
> cluster properly. So from my experience its pretty good. No issues till yet.
>
>
>
> [0]: https://github.com/kubernetes/charts/tree/master/incubator/cassandra
> [github.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_charts_tree_master_incubator_cassandra&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=1oh1YI8i5eJD1DFTwooO7U92fFi2fjan6lqP61yAiyo&m=dupKDpZi0lkjFkqaSd6XaEj5nuY1T5UObgAcXCBqo7A&s=0WTYStEM1zvh2BQKvnVLRpukxgr0aDLyGffyE1V2xik&e=>
>
>
>
>
>
> Regards,
>
> Pradeep
>
>
>
> On Fri, May 18, 2018 at 1:01 PM, Павел Сапежко 
> wrote:
>
> Hi, Hassaan! For example we are using C* in k8s in production for our
> video surveillance system. Moreover, we are using Ceph RBD as our storage
> for cassandra. Today we have 8 C* nodes each manages 2Tb of data.
>
>
>
> On Fri, May 18, 2018 at 9:27 AM Hassaan Pasha  wrote:
>
> Hi,
>
>
>
> I am trying to craft a deployment strategy for deploying and maintaining a
> C* cluster. I was wondering if there are actual production deployments of
> C* using K8s as the orchestration layer.
>
>
>
> I have been given the impression that K8s managing a C* cluster can be a
> recipe for disaster, especially if you aren't well versed with the
> intricacies of a scale-up/down event. I know use cases where people are
> using Mesos or a custom tool built with terraform/chef etc to run their
> production clusters but have yet to find a real K8s use case.
>
>
>
> *Questions?*
>
> Is K8s a reasonable choice for managing a production C* cluster?
>
> Are there documented use cases for this?
>
>
>
> Any help would be greatly appreciated.
>
>
>
> --
>
> Regards,
>
>
>
> *Hassaan Pasha*
>
> --
>
> Regrads,
>
> Pavel Sapezhko
>
>
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Using K8s to Manage Cassandra in Production

[Ben Bromhead]

The official Kubernetes Java driver is actually pretty feature complete, if
not exactly idiomatic Java...  it's only missing full examples to get it to
GOLD compatibility levels iirc.

A few reasons we went down the Java path:

   - Cassandra community engagement was the primary concern. If you are a
   developer in the Cassandra community you have a base level of Java
   knowledge, so it means if you want to work on the Kubernetes operator you
   only have to learn 1 thing, Kubernetes. If the operator was in Go, you
   would then have two things to learn, Go and Kubernetes :)
   - We actually wrote an initial PoC in Go (based off the etcd operator,
   you can find it here
   https://github.com/benbromhead/cassandra-operator-old ), but because it
   was in Go we ended up making architectural decisions simply because
   Go doesn't do JMX, so it felt like we were just fighting different
   ecosystems just to be part of the cool group.

Some other less important points weighed the decision in Java's favour:

   - The folk at Instaclustr all know Java, and are productive in it from
   day 1. Go is fun and relatively simple, but not our forte.
   -  Mature package management, Generics/inability to write DRY
   code, a million if err statements  (:
   - Some other awesome operators/controllers are written in JVM based
   languages. The sparkKubernetes resource manager (which is a k8s controller)
   is written in Scala.


On Wed, May 23, 2018 at 10:04 AM vincent gromakowski <
vincent.gromakow...@gmail.com> wrote:

> Why did you choose java for the operator implementation when everybody
> seems to use the go client (probably for greater functionalities) ?
>
> 2018-05-23 15:39 GMT+02:00 Ben Bromhead :
>
>> You can get a good way with StatefulSets, but as Tom mentioned there are
>> still some issues with this, particularly around scaling up and down.
>>
>> We are working on an Operator for Apache Cassandra, you can find it here
>> https://github.com/instaclustr/cassandra-operator. This is a joint
>> project between Instaclustr, Pivotal and a few other folk.
>>
>> Currently it's a work in progress, but we would love any or all early
>> feedback/PRs/issues etc. Our first GA release will target the following
>> capabilities:
>>
>>- Safe scaling up and down (including decommissioning)
>>- Backup/restore workflow (snapshots only initially)
>>- Built in prometheus integration and discovery
>>
>> Other features like repair, better PV support, maybe even a nice
>> dashboard will be on the way.
>>
>>
>> On Wed, May 23, 2018 at 7:35 AM Tom Petracca 
>> wrote:
>>
>>> Using a statefulset should get you pretty far, though will likely be
>>> less effective than a coreos-style “operator”. Some random points:
>>>
>>>- For scale-up: a node shouldn’t report “ready” until it’s in the
>>>NORMAL state; this will prevent multiple nodes from bootstrapping at 
>>> once.
>>>- For scale-down: as of now there isn’t a mechanism to know if a pod
>>>is getting decommissioned because you’ve permanently lowered replica 
>>> count,
>>>or because it’s just getting bounced/re-scheduled, thus knowing whether 
>>> or
>>>not to decommission is basically impossible. Relevant issue:
>>>kubernetes/kubernetes#1462
>>><https://github.com/kubernetes/kubernetes/issues/1462>
>>>
>>>
>>>
>>> *From: *Pradeep Chhetri 
>>> *Reply-To: *"user@cassandra.apache.org" 
>>> *Date: *Friday, May 18, 2018 at 10:20 AM
>>> *To: *"user@cassandra.apache.org" 
>>> *Subject: *Re: Using K8s to Manage Cassandra in Production
>>>
>>>
>>>
>>> Hello Hassaan,
>>>
>>>
>>>
>>> We use cassandra helm chart[0] for deploying cassandra over kubernetes
>>> in production. We have around 200GB cas data. It works really well. You can
>>> scale up nodes easily (I haven't tested scaling down).
>>>
>>>
>>>
>>> I would say that if you are worried about running cassandra over k8s in
>>> production, maybe you should first try setting it for your
>>> staging/preproduction and gain confidence over time.
>>>
>>>
>>>
>>> I have tested situations where i have killed the host running cassandra
>>> container and have seen that container moves to a different node and joins
>>> cluster properly. So from my experience its pretty good. No issues till yet.
>>>
>>>
>>>
>>> [0]: https://github.com/kubernetes/charts/tree/master/incubator/cassandra
&

Re: Using K8s to Manage Cassandra in Production

[Ben Bromhead]

Here is the expectations around compatibility levels
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/csi-new-client-library-procedure.md#client-capabilities
Though
references to gold, silver, bronze etc seem to have largely gone away...
not sure what's going on there?

For a full reference just browser through the repo,
https://github.com/kubernetes-client/java/blob/master/kubernetes/README.md is
a good place to start as is
https://github.com/kubernetes-client/java/tree/master/examples

The Java driver doesn't have as much of the nice things in
https://github.com/kubernetes/client-go/tree/master/tools but it does have
some good helper classes in the util package so I guess we spent a little
more time wiring things together?

Code Generation is done via the jsonschema2pojo maven plugin and we also
just have the raw CRD definition in a resource directory.

On Wed, May 23, 2018 at 11:23 AM vincent gromakowski <
vincent.gromakow...@gmail.com> wrote:

> Thanks ! Do you have some pointers on the available features ? I am more
> afraid of the lack of custom controller integration, for instance the code
> generator...
>
> 2018-05-23 17:17 GMT+02:00 Ben Bromhead :
>
>> The official Kubernetes Java driver is actually pretty feature complete,
>> if not exactly idiomatic Java...  it's only missing full examples to get it
>> to GOLD compatibility levels iirc.
>>
>> A few reasons we went down the Java path:
>>
>>- Cassandra community engagement was the primary concern. If you are
>>a developer in the Cassandra community you have a base level of Java
>>knowledge, so it means if you want to work on the Kubernetes operator you
>>only have to learn 1 thing, Kubernetes. If the operator was in Go,
>>you would then have two things to learn, Go and Kubernetes :)
>>- We actually wrote an initial PoC in Go (based off the etcd
>>operator, you can find it here
>>https://github.com/benbromhead/cassandra-operator-old ), but because
>>it was in Go we ended up making architectural decisions simply because
>>Go doesn't do JMX, so it felt like we were just fighting different
>>ecosystems just to be part of the cool group.
>>
>> Some other less important points weighed the decision in Java's favour:
>>
>>- The folk at Instaclustr all know Java, and are productive in it
>>from day 1. Go is fun and relatively simple, but not our forte.
>>-  Mature package management, Generics/inability to write DRY
>>code, a million if err statements  (:
>>- Some other awesome operators/controllers are written in JVM based
>>languages. The sparkKubernetes resource manager (which is a k8s 
>> controller)
>>is written in Scala.
>>
>>
>> On Wed, May 23, 2018 at 10:04 AM vincent gromakowski <
>> vincent.gromakow...@gmail.com> wrote:
>>
>>> Why did you choose java for the operator implementation when everybody
>>> seems to use the go client (probably for greater functionalities) ?
>>>
>>> 2018-05-23 15:39 GMT+02:00 Ben Bromhead :
>>>
>>>> You can get a good way with StatefulSets, but as Tom mentioned there
>>>> are still some issues with this, particularly around scaling up and down.
>>>>
>>>> We are working on an Operator for Apache Cassandra, you can find it
>>>> here https://github.com/instaclustr/cassandra-operator. This is a
>>>> joint project between Instaclustr, Pivotal and a few other folk.
>>>>
>>>> Currently it's a work in progress, but we would love any or all early
>>>> feedback/PRs/issues etc. Our first GA release will target the following
>>>> capabilities:
>>>>
>>>>- Safe scaling up and down (including decommissioning)
>>>>- Backup/restore workflow (snapshots only initially)
>>>>- Built in prometheus integration and discovery
>>>>
>>>> Other features like repair, better PV support, maybe even a nice
>>>> dashboard will be on the way.
>>>>
>>>>
>>>> On Wed, May 23, 2018 at 7:35 AM Tom Petracca 
>>>> wrote:
>>>>
>>>>> Using a statefulset should get you pretty far, though will likely be
>>>>> less effective than a coreos-style “operator”. Some random points:
>>>>>
>>>>>- For scale-up: a node shouldn’t report “ready” until it’s in the
>>>>>NORMAL state; this will prevent multiple nodes from bootstrapping at 
>>>>> once.
>>>>>- For scale-down: as 

Re: Jmx_exporter CPU spike

[Ben Bromhead]

Hi Rajpal

I'd invite you to have a look at
https://github.com/zegelin/cassandra-exporter

Significantly faster (bypasses JMX rpc stuff, 10ms to collect metrics for
300 tables vs 2-3 seconds via JMX), plus the naming/tagging fits far better
into the Prometheus world. Still missing a few stats like GC etc, but feel
free to submit a PR!

Ben



On Mon, Jul 9, 2018 at 12:03 AM Rahul Singh 
wrote:

> How often are you polling the JMX? How much of a spike are you seeing in
> CPU?
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
> On Jul 5, 2018, 2:45 PM -0500, rajpal reddy ,
> wrote:
>
>
> we have Qualys security scan running causing the cpu spike. We are seeing
> the CPU spike only when Jmx metrics are exposed using Jmx_exporter. tried
> setting up imx authentication still see cpu spike. if i stop using jmx
> exporter we don’t see any cpu spike. is there any thing we have to tune to
> make work with Jmx_exporter?
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Advantage over Cassandra in Kubernetes

[Ben Bromhead]

This is a fairly high-level question which could end up going quite deep,
but below is a quick summary off the top of my head.

You can get a few advantages when running Cassandra in Kubernetes,
particularly:

   - Easy discovery and network connectivity with other services running on
   K8s
   - Reproducible, repeatable operations and deployments
   - A cloud-independent approach to container orchestration, that is
   supported by all major cloud providers.
   - Easy backups, deployments, scaling etc via statefulsets or an operator
   (see https://github.com/instaclustr/cassandra-operator).

There are also some rough edges with running Cassandra on Kubernetes:

   - Failure domain placement with statefulsets is still challenging (v1.12
   goes a long way to fixing this -
   
https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/
   )
   - Getting resource constraints correct and working out scheduling in
   constrained environments can be maddening.
   - Only a few, small deployments (that I'm aware of) are running
   Cassandra in Kubernetes in production. So you will be breaking new ground
   and encounter problems that haven't been solved before.
   - The Cassandra examples in the official Kubernetes documentation is not
   something you want to take into production.

Cheers

Ben

On Thu, Oct 11, 2018 at 6:50 PM Goutham reddy 
wrote:

> Hi,
> We are in the process of setting up Cassandra cluster with high
> availability. So the debate is installing Cassandra in Kubernetes cluster.
> Can someone  throw some light, what advantages can I get when created
> Cassandra cluster inside Kubernetes cluster. Any comments are highly
> appreciated:)
>
> Thanks and Regards,
> Goutham Reddy Aenugu.
> --
> Regards
> Goutham Reddy
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: Cassandra DCOS | monitoring connection and user activity

[Ben Bromhead]

To dig a little deeper.

It sounds like JMX is just enabled for loopback, if JMX was
disabled/unreachable entirely, nodetool wouldn't work at all.

You will likely need to customize your Cassandra config to allow remote JMX
connections (though this does increase your attack surface and risk from a
security perspective). Alternatively, whatever monitoring solution you land
on will likely need a local agent to connect to JMX via loopback.

To answer your specific monitoring questions, iirc JMX will expose the
number of active connections per node, however, it does not show what
queries are being executed.

Cassandra does support slow query logging (later versions), or you can use
query tracing (probabilistic and at the session level) to see what is
happening in real time. Though query tracing is fairly heavy from a
performance perspective.

I generally don't recommend thinking about tracking bad query issues in
Cassandra the same way you would with a relational DB. Cassandra does not
have long running queries that get stuck, hold resources, hold locks etc.

Most bad performance smells from bad queries (as opposed to just an
increase in load) will show up in the logs e.g. large batches, queries
timing out, slow queries (when slow query logger is enabled),
excessive/long GC pauses, etc

However, you mention you get timeouts when you increase the load on
Cassandra. If you had some monitoring in place to graph reads and writes
per second against timeouts you might discover you simply need to add more
nodes or look at trying to squeeze more performance out of what you have.

Refer to Anups previous thoughts on where to get started with getting
operational data out of Cassandra

On Wed, Oct 31, 2018 at 8:21 PM Anup Shirolkar <
anup.shirol...@instaclustr.com> wrote:

> Hi,
>
> Looks like you need a monitoring for Cassandra but without using JMX.
> It is possible to use metric reporting libraries in Cassandra:
> https://wiki.apache.org/cassandra/Metrics#Reporting
>
> I do not have specific experience with using Cassandra on DCOS but,
> monitoring with libraries and tools should not be any different.
>
> There are various options available to establish good monitoring
> (Graphite, Prometheus, Grafana)
>
> Helpful links:
>
>
> https://blog.pythian.com/monitoring-apache-cassandra-metrics-graphite-grafana/
> https://github.com/instaclustr/cassandra-exporter
> https://prometheus.io/docs/instrumenting/exporters/
> https://grafana.com/dashboards/5408
>
> http://thelastpickle.com/blog/2017/04/05/cassandra-graphite-measurements-filtered.html
>
> Regards,
>
> Anup Shirolkar
>
>
>
>
> On Wed, 31 Oct 2018 at 18:41, Caesar, Maik  wrote:
>
>> Hello All,
>>
>> have someone experience with monitoring cassandra in DCOS?
>>
>> If we increase the load to the Casandra in DCOS, the application get
>> timeouts and loose the connection and I do not have any information about
>> what happened.
>>
>> Is there a way to get information about the amount of current connection
>> and which queries are executed? Cassandra in DCOS has disabled the JMX
>> interface and I think the noodetool do not provide such information.
>>
>>
>>
>> Regards
>>
>> Maik
>>
>>
>>
>> DXC Technology Company -- This message is transmitted to you by or on
>> behalf of DXC Technology Company or one of its affiliates. It is intended
>> exclusively for the addressee. The substance of this message, along with
>> any attachments, may contain proprietary, confidential or privileged
>> information or information that is otherwise legally exempt from
>> disclosure. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient of this message, you are
>> not authorized to read, print, retain, copy or disseminate any part of this
>> message. If you have received this message in error, please destroy and
>> delete all copies and notify the sender by return e-mail. Regardless of
>> content, this e-mail shall not operate to bind DXC Technology Company or
>> any of its affiliates to any order or other contract unless pursuant to
>> explicit written agreement or government initiative expressly permitting
>> the use of e-mail for such purpose. --.
>>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: CassKop : a Cassandra operator for Kubernetes developped by Orange

[Ben Bromhead]

This all looks really awesome, while i wish there wasn't as much duplicated
effort across the different projects it's great to see how different teams
are solving this very real problem!

Also Attila we do have our cloud offering as you mentioned, but we also
have our own k8s operator which is Apache licensed.

So awesome work to the teams from Orange, Datastax and Sky and welcome to
the fun of building on a hugely popular and constantly changing project :)

Most folk don't realize you can't just run a stateful set!


On Sat, May 25, 2019, 05:54 Attila Wind  wrote:

> Maybe my understanding is wrong and I am not really a "deployment guru"
> but it looks like to me that
>
> Orange (https://github.com/Orange-OpenSource/cassandra-k8s-operator, 1
> contributor and 1 commit for now on 2019-05-24)
> and sky-uk/cassandra-operator (
> https://github.com/sky-uk/cassandra-operator , it's in alpha phase and
> not recommended in production, 3 contributors, 24 commits btw
> 2019.03.25-2019.05.21, 32 Issues)
> are developing something I could use in my OWN(!) Kubernetes based
> solution (even on premise if I want or whatever)
> They are both open source. Right?
>
> While
> Datastax and Instaclustr are commercial players and offer the solution in
> a tightly-coupled way with Cloud only
> (I just took a quick look on Instaclustr but could not even figure out
> pricing info for this service... probably I am lame... or not? :-))
>
> So this looks to me a nice competition...
> What do I miss?
>
> ps.: maybe Orange and sky-uk/cassandra-operator guys should cooperate..??
> Others are clearly building business around it
>
> cheers
> Attila Wind
>
> http://www.linkedin.com/in/attilaw
> Mobile: +36 31 7811355
>
>
> On 2019. 05. 24. 20:36, John Sanda wrote:
>
> There is also
> https://github.com/sky-uk/cassandra-operator
>
> On Fri, May 24, 2019 at 2:34 PM Rahul Singh 
> wrote:
>
>> Fantastic! Now there are three teams making k8s operators for C*:
>> Datastax, Instaclustr, and now Orange.
>>
>> rahul.xavier.si...@gmail.com
>>
>> http://cassandra.link
>>
>> I'm speaking at #DataStaxAccelerate, the world’s premiere
>> #ApacheCassandra conference, and I want to see you there! Use my code
>> Singh50 for 50% off your registration. www.datastax.com/accelerate
>>
>>
>> On Fri, May 24, 2019 at 9:07 AM Jean-Armel Luce 
>> wrote:
>>
>>> Hi folks,
>>>
>>> We are excited to announce that CassKop, a Cassandra operator for
>>> Kubernetes developped by Orange teams, is now ready for Beta testing.
>>>
>>> CassKop works as a usual K8S controller (reconcile the real state with a
>>> desired state) and automates the Cassandra operations through JMX. All the
>>> operations are launched by calling standard K8S APIs (kubectl apply …) or
>>> by using a K8S plugin (kubectl casskop …).
>>>
>>> CassKop is developed in GO, based on CoreOS operator-sdk framework.
>>> Main features already available :
>>> - deploying a rack aware cluster (or AZ aware cluster)
>>> - scaling up & down (including cleanups)
>>> - setting and modifying configuration parameters (C* and JVM parameters)
>>> - adding / removing a datacenter in Cassandra (all datacenters must be
>>> in the same region)
>>> - rebuilding nodes
>>> - removing node or replacing node (in case of hardware failure)
>>> - upgrading C* or Java versions (including upgradesstables)
>>> - monitoring (using Prometheus/Grafana)
>>> - ...
>>>
>>> By using local and persistent volumes, it is possible to handle failures
>>> or stop/start nodes for maintenance operations with no transfer of data
>>> between nodes.
>>> Moreover, we can deploy cassandra-reaper in K8S and use it for
>>> scheduling repair sessions.
>>> For now, we can deploy a C* cluster only as a mono-region cluster. We
>>> will work during the next weeks to be able to deploy a C* cluster as a
>>> multi regions cluster.
>>>
>>> Still in the roadmap :
>>> - Network encryption
>>> - Monitoring (exporting logs and metrics)
>>> - backup & restore
>>> - multi-regions support
>>>
>>> We'd be interested to hear you try this and let us know what you think!
>>>
>>> Please read the description and installation instructions on
>>> https://github.com/Orange-OpenSource/cassandra-k8s-operator.
>>> For a quick start, you can also follow this step by step guide :
>>> https://orange-opensource.github.io/cassandra-k8s-operator/index.html?slides=Slides-CassKop-demo.md#1
>>>
>>>
>>> The CassKop Team
>>>
>> --
>
> - John
>
>

Re: SS Tables Files Streaming

[Ben Bromhead]

Yup, with repair and particularly bootstrap is there is a decent amount of
"over streaming" of data due to the fact it's just sending an sstable.

On Fri, 6 May 2016 at 14:49 Anubhav Kale  wrote:

> Does repair really send SS Table files as is ? Wouldn’t data for tokens be
> distributed across SS Tables ?
>
>
>
> *From:* Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com]
> *Sent:* Friday, May 6, 2016 2:12 PM
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: SS Tables Files Streaming
>
>
>
> Also probably sstableloader / bulk loading interface
>
>
>
>
>
>
>
>
>
> (I don’t think any of these necessarily stream “as-is”, but that’s a
> different conversation I suspect)
>
>
>
>
>
> *From: *Jonathan Haddad
> *Reply-To: *"user@cassandra.apache.org"
> *Date: *Friday, May 6, 2016 at 1:52 PM
> *To: *"user@cassandra.apache.org"
> *Subject: *Re: SS Tables Files Streaming
>
>
>
> Repairs, bootstamp, decommission.
>
>
>
> On Fri, May 6, 2016 at 1:16 PM Anubhav Kale 
> wrote:
>
> Hello,
>
>
>
> In what scenarios can SS Table files on disk from Node 1 go to Node 2 as
> is ?  I’m aware this happens in *nodetool rebuild* and I am assuming this
> does *not* happen in repairs. Can someone confirm ?
>
>
>
> The reason I ask is I am working on a solution for backup / restore and I
> need to be sure if I boot a node, start copying over backed up files then
> those files won’t get overwritten by something coming from other nodes.
>
>
>
> Thanks !
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: SS Tables Files Streaming

[Ben Bromhead]

Note that incremental repair strategies (2.1+) run anti-compaction against
sstables in the range being repaired, so this will prevent overstreaming
based on the ranges in the repair session.

On Mon, 9 May 2016 at 10:31 Ben Bromhead  wrote:

> Yup, with repair and particularly bootstrap is there is a decent amount of
> "over streaming" of data due to the fact it's just sending an sstable.
>
> On Fri, 6 May 2016 at 14:49 Anubhav Kale 
> wrote:
>
>> Does repair really send SS Table files as is ? Wouldn’t data for tokens
>> be distributed across SS Tables ?
>>
>>
>>
>> *From:* Jeff Jirsa [mailto:jeff.ji...@crowdstrike.com]
>> *Sent:* Friday, May 6, 2016 2:12 PM
>>
>>
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: SS Tables Files Streaming
>>
>>
>>
>> Also probably sstableloader / bulk loading interface
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> (I don’t think any of these necessarily stream “as-is”, but that’s a
>> different conversation I suspect)
>>
>>
>>
>>
>>
>> *From: *Jonathan Haddad
>> *Reply-To: *"user@cassandra.apache.org"
>> *Date: *Friday, May 6, 2016 at 1:52 PM
>> *To: *"user@cassandra.apache.org"
>> *Subject: *Re: SS Tables Files Streaming
>>
>>
>>
>> Repairs, bootstamp, decommission.
>>
>>
>>
>> On Fri, May 6, 2016 at 1:16 PM Anubhav Kale 
>> wrote:
>>
>> Hello,
>>
>>
>>
>> In what scenarios can SS Table files on disk from Node 1 go to Node 2 as
>> is ?  I’m aware this happens in *nodetool rebuild* and I am assuming
>> this does *not* happen in repairs. Can someone confirm ?
>>
>>
>>
>> The reason I ask is I am working on a solution for backup / restore and I
>> need to be sure if I boot a node, start copying over backed up files then
>> those files won’t get overwritten by something coming from other nodes.
>>
>>
>>
>> Thanks !
>>
>> --
> Ben Bromhead
> CTO | Instaclustr <https://www.instaclustr.com/>
> +1 650 284 9692
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: How Fast Does Information Spread With Gossip?

[Ben Bromhead]

Gossip propagation is generally best modelled by epidemic algorithms.

Luckily for us Cassandra's gossip protocol is fairly simply.

Cassandra will perform one Gossip Task every second. Within each gossip
task it will randomly gossip with another available node in the cluster, it
will also possibly attempt to gossip with a down node (based on a random
chance that increases as the number of down nodes increases) and if it
hasn't gossiped with seed that round it may also attempt to gossip with a
defined seed. So Cassandra can do up to 3 rounds per second, however these
extra rounds are supposed to be optimizations for improving average case
convergence and recovering from split brain scenarios quicker than would
normally occur.

Assuming just one gossip round per second, for a new piece of information
to spread to all members of the cluster via gossip, you would see a worst
case performance of O(n) gossip rounds where n is the number of nodes in
the cluster. This is because each Cassandra node can gossip to any other
node irrespective of topology (a fully connected mesh).

There is some ongoing discussion about expanding gossip to utilise partial
views of the cluster and exchanging those, or using spanning/broadcast
trees to speed up convergence and reduce workload in large clusters (1000+)
nodes, see https://issues.apache.org/jira/browse/CASSANDRA-12345 for
details.



On Fri, 16 Sep 2016 at 01:01 Jens Rantil  wrote:

> > Is a minute a reasonable upper bound for most clusters?
>
> I have no numbers and I'm sure this differs depending on how large your
> cluster is. We have a small cluster of around 12 nodes and I statuses
> generally propagate in under 5 seconds for sure. So, it will definitely be
> less than 1 minute.
>
> Cheers,
> Jens
>
> On Wed, Sep 14, 2016 at 8:49 PM jerome  wrote:
>
>> Hi,
>>
>>
>> I was curious if anyone had any kind of statistics or ballpark figures on
>> how long it takes information to propagate through a cluster with Gossip?
>> I'm particularly interested in how fast information about the liveness of a
>> node spreads. For example, in an n-node cluster the median amount of time
>> it takes for all nodes to learn that a node went down is f(n) seconds. Is a
>> minute a reasonable upper bound for most clusters? Too high, too low?
>>
>>
>> Thanks,
>>
>> Jerome
>>
> --
>
> Jens Rantil
> Backend Developer @ Tink
>
> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> For urgent matters you can reach me at +46-708-84 18 32.
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Is there any way to throttle the memtable flushing throughput?

[Ben Bromhead]

A few thoughts on the larger problem at hand.

The AWS instance type you are using is not appropriate for a production
workload. Also with memtable flushes that cause spiky write throughput it
sounds like your commitlog is on the same disk as your data directory,
combined with the use of non-SSD EBS I'm not surprised this is happening.
The small amount of memory on the node could also mean your flush writers
are getting backed up (blocked), possibly causing JVM heap pressure and
other fun things (you can check this with nodetool tpstats).

Before you get into tuning memtable flushing I would do the following:

   - Reset your commitlog_sync settings back to default
   - Use an EC2 instance type with at least 15GB of memory, 4 cores and is
   EBS optimized (dedicated EBS bandwidth)
   - Use gp2 or io2 EBS volumes
   - Put your commitlog on a separate EBS volume.
   - Make sure your memtable_flush_writers are not being blocked, if so
   increase the number of flush writers (no more than # of cores)
   - Optimize your read_ahead_kb size and compression_chunk_length to keep
   those EBS reads as small as possible.

Once you have fixed the above, memtable flushing should not be an issue.
Even if you can't/don't want to upgrade the instance type, the other steps
will help things.

Ben

On Tue, 11 Oct 2016 at 10:23 Satoshi Hikida  wrote:

> Hi,
>
> I'm investigating the read/write performance of the C* (Ver. 2.2.8).
> However, I have an issue about memtable flushing which forces the spiky
> write throughput. And then it affects the latency of the client's requests.
>
> So I want to know the answers for the following questions.
>
> 1. Is there any way that throttling the write throughput of the memtable
> flushing? If it exists, how can I do that?
> 2. Is there any way to reduce the spike of the write bandwidth during the
> memtable flushing?
>(I'm in trouble because the delay of the request increases when the
> spike of the write bandwidth occurred)
>
> I'm using one C* node for this investigation. And C* runs on an EC2
> instance (2vCPU, 4GB memory), In addition, I attach two magnetic disks to
> the instance, one stores system data(root file system.(/)), the other
> stores C* data (data files and commit logs).
>
> I also changed a few configurations.
> - commitlog_sync: batch
> - commitlog_sync_batch_window_in_ms: 2
> (Using default value for the other configurations)
>
>
> Regards,
> Satoshi
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: New node overstreaming data ?

[Ben Bromhead]

Over streaming is pretty common, especially with vnodes in 2.x. When
Cassandra streams data to a bootstrapping node it sends the entire SSTable
that contains the data the new node requires even if that table might only
have 1 row out of thousands. This can be exacerbated by STCS with large
SSTables.

Generally reducing the network streaming throughput and increasing
concurrent_compactors (and un-throttling compaction throughput) is the way
to go. If you are in the cloud (e.g. AWS) you can also attach large block
store volumes (EBS) to the instance to act as overflow.

There is some ongoing work in Cassandra 3.x that will make streaming more
efficient and allow Cassandra to only stream portions of the SSTable that
are required.


On Thu, 13 Oct 2016 at 07:57 Anubhav Kale 
wrote:

> Hello,
>
>
>
> We run 2.1.13 and seeing an odd issue. A node went down, and stayed down
> for a while so it went out of gossip. When we try to bootstrap it again (as
> a new node), it overstreams from other nodes, eventually disk becomes full
> and crashes. This repeated 3 times.
>
>
>
> Does anyone have any insights on what to try next (both in terms of root
> causing, and working around) ? To work around, we tried increasing
> #compactors and reducing stream throughput so that at least incoming
> #SSTables would be controlled.
>
>
>
> This has happened to us few times in the past too, so I am wondering if
> this is a known problem (I couldn’t find any JIRAs).
>
>
>
> Thanks !
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Does increment/decrement by 0 generate any commits ?

[Ben Bromhead]

According to https://issues.apache.org/jira/browse/CASSANDRA-7304 unset
values in a prepared statement for a counter does not change the value of
the counter. This applies for versions of Cassandra 2.2 and above.

I would also look to verify the claimed behaviour myself.

On Tue, 11 Oct 2016 at 09:49 Dorian Hoxha  wrote:

> I just have a bunch of counters in 1 row, and I want to selectively update
> them. And I want to keep prepared queries. But I don't want to keep 30
> prepared queries (1 for each counter column, but keep only 1). So in most
> cases, I will increment 1 column by positive integer and the others by 0.
>
> Makes sense ?
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Adding disk capacity to a running node

[Ben Bromhead]

s will use new disk to distribute both
> new and existing data.
>
>
>
> Best regards, Vladimir Yudovin,
>
>
> *Winguzone
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com-3Ffrom-3Dlist&d=DQMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=ixOxpX-xpw1dJZNpaMT3mepToWX8gzmsVaXFizQLzoU&s=4q7P9fddEYpXwPR-h9yA_tk5JwR8l6c7cKJ-LQTVcGM&e=>
> - Hosted Cloud Cassandra on Azure and SoftLayer.Launch your cluster in
> minutes.*
>
>
>
>
>
>  On Mon, 17 Oct 2016 11:43:27 -0400*Seth Edwards  >* wrote 
>
>
>
> We have a few nodes that are running out of disk capacity at the moment
> and instead of adding more nodes to the cluster, we would like to add
> another disk to the server and add it to the list of data directories. My
> question, is, will Cassandra use the new disk for compactions on sstables
> that already exist in the primary directory?
>
>
>
>
>
>
>
> Thanks!
>
>
>
>
>
>
> 
> CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and
> may be legally privileged. If you are not the intended recipient, do not
> disclose, copy, distribute, or use this email or any attachments. If you
> have received this in error please let the sender know and then delete the
> email and all attachments.
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Adding disk capacity to a running node

[Ben Bromhead]

yup you would need to copy the files across to the new volume from the dir
you wanted to give additional space to. Rough steps would look like:

   1. Create EBS volume (make it big... like 3TB)
   2. Attach to instance
   3. Mount/format EBS volume
   4. Stop C*
   5. Copy full/troublesome directory to the EBS volume
   6. Remove copied files (using rsync for the copy / remove step can be a
   good idea)
   7. bind mount EBS volume with the same path as the troublesome directory
   8. Start C* back up
   9. Let it finish compacting / streaming etc
   10. Stop C*
   11. remove bind mount
   12. copy files back to ephemeral
   13. start C* back up
   14. repeat on other nodes
   15. run repair

You can use this process if you somehow end up in a full disk situation. If
you end up in a low disk situation you'll have other issues (like corrupt /
half written SSTable components), but it's better than nothing

Also to maintain your read throughput during this whole thing, double check
the EBS volumes read_ahead_kb setting on the block volume and reduce it to
something sane like 0 or 16.



On Mon, 17 Oct 2016 at 13:42 Seth Edwards  wrote:

> @Ben
>
> Interesting idea, is this also an option for situations where the disk is
> completely full and Cassandra has stopped? (Not that I want to go there).
>
> If this was the route taken, and we did
>
> mount --bind   /mnt/path/to/large/sstable   /mnt/newebs
>
> We would still need to do some manual copying of files? such as
>
> mv /mnt/path/to/large/sstable.sd /mnt/newebs ?
>
> Thanks!
>
> On Mon, Oct 17, 2016 at 12:59 PM, Ben Bromhead 
> wrote:
>
> Yup as everyone has mentioned ephemeral are fine if you run in multiple
> AZs... which is pretty much mandatory for any production deployment in AWS
> (and other cloud providers) . i2.2xls are generally your best bet for high
> read throughput applications on AWS.
>
> Also on AWS ephemeral storage will generally survive a user initiated
> restart. For the times that AWS retires an instance, you get plenty of
> notice and it's generally pretty rare. We run over 1000 instances on AWS
> and see one forced retirement a month if that. We've never had an instance
> pulled from under our feet without warning.
>
> To add another option for the original question, one thing you can do is
> to attach a large EBS drive to the instance and bind mount it to the
> directory for the table that has the very large SSTables. You will need to
> copy data across to the EBS volume. Let everything compact and then copy
> everything back and detach EBS. Latency may be higher than normal on the
> node you are doing this on (especially if you are used to i2.2xl
> performance).
>
> This is something we often have to do, when we encounter pathological
> compaction situations associated with bootstrapping, adding new DCs or STCS
> with a dominant table or people ignore high disk usage warnings :)
>
> On Mon, 17 Oct 2016 at 12:43 Jeff Jirsa 
> wrote:
>
> Ephemeral is fine, you just need to have enough replicas (in enough AZs
> and enough regions) to tolerate instances being terminated.
>
>
>
>
>
>
>
> *From: *Vladimir Yudovin 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Monday, October 17, 2016 at 11:48 AM
> *To: *user 
>
>
> *Subject: *Re: Adding disk capacity to a running node
>
>
>
> It's extremely unreliable to use ephemeral (local) disks. Even if you
> don't stop instance by yourself, it can be restarted on different server in
> case of some hardware failure or AWS initiated update. So all node data
> will be lost.
>
>
>
> Best regards, Vladimir Yudovin,
>
>
> *Winguzone
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com-3Ffrom-3Dlist&d=DQMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=ixOxpX-xpw1dJZNpaMT3mepToWX8gzmsVaXFizQLzoU&s=4q7P9fddEYpXwPR-h9yA_tk5JwR8l6c7cKJ-LQTVcGM&e=>
> - Hosted Cloud Cassandra on Azure and SoftLayer.Launch your cluster in
> minutes.*
>
>
>
>
>
>  On Mon, 17 Oct 2016 14:45:00 -0400*Seth Edwards  >* wrote 
>
>
>
> These are i2.2xlarge instances so the disks currently configured as
> ephemeral dedicated disks.
>
>
>
> On Mon, Oct 17, 2016 at 11:34 AM, Laing, Michael <
> michael.la...@nytimes.com> wrote:
>
>
>
> You could just expand the size of your ebs volume and extend the file
> system. No data is lost - assuming you are running Linux.
>
>
>
>
>
> On Monday, October 17, 2016, Seth Edwards  wrote:
>
> We're running 2.0.16. We're migrating to a new data model but we've had an
> unexpected increase in write traffic that has caus

Introducing Cassandra 3.7 LTS

[Ben Bromhead]

Hi All

I am proud to announce we are making available our production build of
Cassandra 3.7 that we run at Instaclustr (both for ourselves and our
customers). Our release of Cassandra 3.7 includes a number of backported
patches from later versions of Cassandra e.g. 3.8 and 3.9 but doesn't
include the new features of these releases.

You can find our release of Cassandra 3.7 LTS on github here (
https://github.com/instaclustr/cassandra). You can read more of our
thinking and how this applies to our managed service here (
https://www.instaclustr.com/blog/2016/10/19/patched-cassandra-3-7/).

We also have an expanded FAQ about why and how we are approaching 3.x in
this manner (https://github.com/instaclustr/cassandra#cassandra-37-lts),
however I've included the top few question and answers below:

*Is this a fork?*
No, This is just Cassandra with a different release cadence for those who
want 3.x features but are slightly more risk averse than the current
schedule allows.

*Why not just use the official release?*
With the 3.x tick-tock branch we have encountered more instability than
with the previous release cadence. We feel that releasing new features
every other release makes it very hard for operators to stabilize their
production environment without bringing in brand new features that are not
battle tested. With the release of Cassandra 3.8 and 3.9 simultaneously the
bug fix branch included new and real-world untested features, specifically
CDC. We have decided to stick with Cassandra 3.7 and instead backport
critical issues and maintain it ourselves rather than trying to stick with
the current Apache Cassandra release cadence.

*Why backport?*
At Instaclustr we support and run a number of different versions of Apache
Cassandra on behalf of our customers. Over the course of managing Cassandra
for our customers we often encounter bugs. There are existing patches for
some of them, others we patch ourselves. Generally, if we can, we try to
wait for the next official Apache Cassandra release, however in the need to
ensure our customers remain stable and running we will sometimes backport
bugs and write our own hotfixes (which are also submitted back to the
community).

*Why release it?*
A number of our customers and people in the community have asked if we
would make this available, which we are more than happy to do so. This
repository represents what Instaclustr runs in production for Cassandra 3.7
and this is our way of helping the community get a similar level of
stability as what you would get from our managed service.

Cheers

Ben



-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Handle Leap Seconds with Cassandra

[Ben Bromhead]

http://www.datastax.com/dev/blog/preparing-for-the-leap-second gives a
pretty good overview

If you are using a timestamp as part of your primary key, this is the
situation where you could end up overwriting data. I would suggest using
timeuuid instead which will ensure that you get different primary keys even
for data inserted at the exact same timestamp.

The blog post also suggests using certain monotonic timestamp classes in
Java however these will not help you if you have multiple clients that may
overwrite data.

As for the interleaving or out of order problem, this is hard to address in
Cassandra without resorting to external coordination or LWTs. If you are
relying on a wall clock to guarantee order in a distributed system you will
get yourself into trouble even without leap seconds (clock drift, NTP
inaccuracy etc).

On Thu, 20 Oct 2016 at 10:30 Anuj Wadehra  wrote:

> Hi,
>
> I would like to know how you guys handle leap seconds with Cassandra.
>
> I am not bothered about the livelock issue as we are using appropriate
> versions of Linux and Java. I am more interested in finding an optimum
> answer for the following question:
>
> How do you handle wrong ordering of multiple writes (on same row and
> column) during the leap second? You may overwrite the new value with old
> one (disaster).
>
> And Downtime is no option :)
>
> I can see that CASSANDRA-9131 is still open..
>
> FYI..we are on 2.0.14 ..
>
>
> Thanks
> Anuj
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Introducing Cassandra 3.7 LTS

[Ben Bromhead]

Thanks Sankalp, we are also reviewing our internal 2.1 list against what
you published (though we are trying to upgrade everyone to later versions
e.g. 2.2). It's great to compare notes.

On Thu, 20 Oct 2016 at 16:19 sankalp kohli  wrote:

> This is awesome. I have send out the patches which we back ported into 2.1
> on the dev list.
>
> On Wed, Oct 19, 2016 at 4:33 PM, kurt Greaves 
> wrote:
>
>
> On 19 October 2016 at 21:07, sfesc...@gmail.com 
> wrote:
>
> Wow, thank you for doing this. This sentiment regarding stability seems to
> be widespread. Is the team reconsidering the whole tick-tock cadence? If
> not, I would add my voice to those asking that it is revisited.
>
>
> There has certainly been discussion regarding the tick-tock cadence, and
> it seems safe to say it will change. There hasn't been any official
> announcement yet, however.
>
> Kurt Greaves
> k...@instaclustr.com
> www.instaclustr.com
>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Handle Leap Seconds with Cassandra

[Ben Bromhead]

If you need guaranteed strict ordering in a distributed system, I would not
use Cassandra, Cassandra does not provide this out of the box. I would look
to a system that uses lamport or vector clocks. Based on your description
of how your systems runs at the moment (and how close your updates are
together), you have either already experienced out of order updates or
there is a real possibility you will in the future.

Sorry to be so dire, but if you do require causal consistency / strict
ordering, you are not getting it at the moment. Distributed systems theory
is really tricky, even for people that are "experts" on distributed systems
over unreliable networks (I would certainly not put myself in that
category). People have made a very good name for themselves by showing that
the vast majority of distributed databases have had bugs when it comes to
their various consistency models and the claims these databases make.

So make sure you really do need guaranteed causal consistency/strict
ordering or if you can design around it (e.g. using conflict free
replicated data types) or choose a system that is designed to provide it.

Having said that... here are some hacky things you could do in Cassandra to
try and get this behaviour, which I in no way endorse doing :)

   - Cassandra counters do leverage a logical clock per shard and you could
   hack something together with counters and lightweight transactions, but you
   would want to do your homework on counters accuracy during before diving
   into it... as I don't know if the implementation is safe in the context of
   your question. Also this would probably require a significant rework of
   your application plus a significant performance hit. I would invite a
   counter guru to jump in here...


   - You can leverage the fact that timestamps are monotonic if you isolate
   writes to a single node for a single shared... but you then loose
   Cassandra's availability guarantees, e.g. a keyspace with an RF of 1 and a
   CL of > ONE will get monotonic timestamps (if generated on the server
   side).


   - Continuing down the path of isolating writes to a single node for a
   given shard you could also isolate writes to the primary replica using your
   client driver during the leap second (make it a minute either side of the
   leap), but again you lose out on availability and you are probably already
   experiencing out of ordered writes given how close your writes and updates
   are.


A note on NTP: NTP is generally fine if you use it to keep the clocks
synced between the Cassandra nodes. If you are interested in how we have
implemented NTP at Instaclustr, see our blogpost on it
https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/
.



Ben


On Thu, 27 Oct 2016 at 10:18 Anuj Wadehra  wrote:

> Hi Ben,
>
> Thanks for your reply. We dont use timestamps in primary key. We rely on
> server side timestamps generated by coordinator. So, no functions at
> client side would help.
>
> Yes, drifts can create problems too. But even if you ensure that nodes are
> perfectly synced with NTP, you will surely mess up the order of updates
> during the leap second(interleaving). Some applications update same column
> of same row quickly (within a second ) and reversing the order would
> corrupt the data.
>
> I am interested in learning how people relying on strict order of updates
> handle leap second scenario when clock goes back one second(same second is
> repeated). What kind of tricks people use  to ensure that server side
> timestamps are monotonic ?
>
> As per my understanding NTP slew mode may not be suitable for Cassandra as
> it may cause unpredictable drift amongst the Cassandra nodes. Ideas ??
>
>
> Thanks
> Anuj
>
>
>
> Sent from Yahoo Mail on Android
> <https://overview.mail.yahoo.com/mobile/?.src=Android>
>
> On Thu, 20 Oct, 2016 at 11:25 PM, Ben Bromhead
>
>  wrote:
> http://www.datastax.com/dev/blog/preparing-for-the-leap-second gives a
> pretty good overview
>
> If you are using a timestamp as part of your primary key, this is the
> situation where you could end up overwriting data. I would suggest using
> timeuuid instead which will ensure that you get different primary keys even
> for data inserted at the exact same timestamp.
>
> The blog post also suggests using certain monotonic timestamp classes in
> Java however these will not help you if you have multiple clients that may
> overwrite data.
>
> As for the interleaving or out of order problem, this is hard to address
> in Cassandra without resorting to external coordination or LWTs. If you are
> relying on a wall clock to guarantee order in a distributed system you will
> get yourself into trouble even without leap seconds (clock drift, NTP
> inaccuracy etc).
>
> On Thu, 20 Oct 2016 at 10:30 Anuj W

Re: Introducing Cassandra 3.7 LTS

[Ben Bromhead]

We are not publishing the build artefacts for our LTS at the moment as we
don't test them on the different distros (debian/ubuntu, centos etc). If
anyone wishes to do so feel free to create a PR and submit them!

On Wed, 2 Nov 2016 at 11:37 Jesse Hodges  wrote:

> awesome, thanks for the tip!
>
> -Jesse
>
> On Wed, Nov 2, 2016 at 12:39 PM, Benjamin Roth 
> wrote:
>
> You can build one on your own very easily. Just check out the desired git
> repo and do this:
>
>
> http://stackoverflow.com/questions/8989192/how-to-package-the-cassandra-source-code-into-debian-package
>
> 2016-11-02 17:35 GMT+01:00 Jesse Hodges :
>
> Just curious, has anybody created a debian package for this?
>
> Thanks, Jesse
>
> On Sat, Oct 22, 2016 at 7:45 PM, Kai Wang  wrote:
>
> This is awesome! Stability is the king.
>
> Thank you so much!
>
> On Oct 19, 2016 2:56 PM, "Ben Bromhead"  wrote:
>
> Hi All
>
> I am proud to announce we are making available our production build of
> Cassandra 3.7 that we run at Instaclustr (both for ourselves and our
> customers). Our release of Cassandra 3.7 includes a number of backported
> patches from later versions of Cassandra e.g. 3.8 and 3.9 but doesn't
> include the new features of these releases.
>
> You can find our release of Cassandra 3.7 LTS on github here (
> https://github.com/instaclustr/cassandra). You can read more of our
> thinking and how this applies to our managed service here (
> https://www.instaclustr.com/blog/2016/10/19/patched-cassandra-3-7/).
>
> We also have an expanded FAQ about why and how we are approaching 3.x in
> this manner (https://github.com/instaclustr/cassandra#cassandra-37-lts),
> however I've included the top few question and answers below:
>
> *Is this a fork?*
> No, This is just Cassandra with a different release cadence for those who
> want 3.x features but are slightly more risk averse than the current
> schedule allows.
>
> *Why not just use the official release?*
> With the 3.x tick-tock branch we have encountered more instability than
> with the previous release cadence. We feel that releasing new features
> every other release makes it very hard for operators to stabilize their
> production environment without bringing in brand new features that are not
> battle tested. With the release of Cassandra 3.8 and 3.9 simultaneously the
> bug fix branch included new and real-world untested features, specifically
> CDC. We have decided to stick with Cassandra 3.7 and instead backport
> critical issues and maintain it ourselves rather than trying to stick with
> the current Apache Cassandra release cadence.
>
> *Why backport?*
> At Instaclustr we support and run a number of different versions of Apache
> Cassandra on behalf of our customers. Over the course of managing Cassandra
> for our customers we often encounter bugs. There are existing patches for
> some of them, others we patch ourselves. Generally, if we can, we try to
> wait for the next official Apache Cassandra release, however in the need to
> ensure our customers remain stable and running we will sometimes backport
> bugs and write our own hotfixes (which are also submitted back to the
> community).
>
> *Why release it?*
> A number of our customers and people in the community have asked if we
> would make this available, which we are more than happy to do so. This
> repository represents what Instaclustr runs in production for Cassandra 3.7
> and this is our way of helping the community get a similar level of
> stability as what you would get from our managed service.
>
> Cheers
>
> Ben
>
>
>
> --
> Ben Bromhead
> CTO | Instaclustr <https://www.instaclustr.com/>
> +1 650 284 9692
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>
>
>
>
>
> --
> Benjamin Roth
> Prokurist
>
> Jaumo GmbH · www.jaumo.com
> Wehrstraße 46 · 73035 Göppingen · Germany
> Phone +49 7161 304880-6 · Fax +49 7161 304880-1
> AG Ulm · HRB 731058 · Managing Director: Jens Kammerer
>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Handle Leap Seconds with Cassandra

[Ben Bromhead]

Based on most of what I've said previously pretty much most ways of
avoiding your ordering issue of the leap second is going to be a "hack" and
there will be some amount of hope involved.

If the updates occur more than 300ms apart and you are confident your nodes
have clocks that are within 150ms of each other, then I'd close my eyes and
hope they all leap second at the same time within that 150ms.

If they are less then 300ms (I'm guessing you meant less 300ms), then I
would look to figure out what the smallest gap is between those two updates
and make sure your nodes clocks are close enough in that gap that the leap
second will occur on all nodes within that gap.

If that's not good enough, you could just halt those scenarios for 2
seconds over the leap second and then resume them once you've confirmed all
clocks have skipped.


On Wed, 2 Nov 2016 at 18:13 Anuj Wadehra  wrote:

> Thanks Ben for taking out time for the detailed reply !!
>
> We dont need strict ordering for all operations but we are looking for
> scenarios where 2 quick updates to same column of same row are possible. By
> quick updates, I mean >300 ms. Configuring NTP properly (as mentioned in
> some blogs in your link) should give fair relative accuracy between the
> Cassandra nodes. But leap second takes the clock back for an ENTIRE one
> sec (huge) and the probability of old write overwriting the new one
> increases drastically. So, we want to be proactive with things.
>
> I agree that you should avoid such scebaruos with design (if possible).
>
> Good to know that you guys have setup your own NTP servers as per the
> recommendation. Curious..Do you also do some monitoring around NTP?
>
>
>
> Thanks
> Anuj
>
> On Fri, 28 Oct, 2016 at 12:25 AM, Ben Bromhead
>
>  wrote:
> If you need guaranteed strict ordering in a distributed system, I would
> not use Cassandra, Cassandra does not provide this out of the box. I would
> look to a system that uses lamport or vector clocks. Based on your
> description of how your systems runs at the moment (and how close your
> updates are together), you have either already experienced out of order
> updates or there is a real possibility you will in the future.
>
> Sorry to be so dire, but if you do require causal consistency / strict
> ordering, you are not getting it at the moment. Distributed systems theory
> is really tricky, even for people that are "experts" on distributed systems
> over unreliable networks (I would certainly not put myself in that
> category). People have made a very good name for themselves by showing that
> the vast majority of distributed databases have had bugs when it comes to
> their various consistency models and the claims these databases make.
>
> So make sure you really do need guaranteed causal consistency/strict
> ordering or if you can design around it (e.g. using conflict free
> replicated data types) or choose a system that is designed to provide it.
>
> Having said that... here are some hacky things you could do in Cassandra
> to try and get this behaviour, which I in no way endorse doing :)
>
>- Cassandra counters do leverage a logical clock per shard and you
>could hack something together with counters and lightweight transactions,
>but you would want to do your homework on counters accuracy during before
>diving into it... as I don't know if the implementation is safe in the
>context of your question. Also this would probably require a significant
>rework of your application plus a significant performance hit. I would
>invite a counter guru to jump in here...
>
>
>- You can leverage the fact that timestamps are monotonic if you
>isolate writes to a single node for a single shared... but you then loose
>Cassandra's availability guarantees, e.g. a keyspace with an RF of 1 and a
>CL of > ONE will get monotonic timestamps (if generated on the server
>side).
>
>
>- Continuing down the path of isolating writes to a single node for a
>given shard you could also isolate writes to the primary replica using your
>client driver during the leap second (make it a minute either side of the
>leap), but again you lose out on availability and you are probably already
>experiencing out of ordered writes given how close your writes and updates
>are.
>
>
> A note on NTP: NTP is generally fine if you use it to keep the clocks
> synced between the Cassandra nodes. If you are interested in how we have
> implemented NTP at Instaclustr, see our blogpost on it
> https://www.instaclustr.com/blog/2015/11/05/apache-cassandra-synchronization/
> .
>
>
>
> Ben
>
>
> On Thu, 27 Oct 2016 at 10:18 Anuj Wadehra  wrote:
>

Re: Are Cassandra writes are faster than reads?

[Ben Bromhead]

To add some flavor as to how the commitlog implementation is so quick.

It only flushes to disk every 10s by default. So writes are effectively
done to memory and then to disk asynchronously later on. This is generally
accepted to be OK, as the write is also going to other nodes.

You can of course change this behavior to flush on each write or to skip
the commitlog altogether (danger!). This however will change how "safe"
things are from a durability perspective.

On Sun, Nov 6, 2016, 12:51 Jeff Jirsa  wrote:

> Cassandra writes are particularly fast, for a few reasons:
>
>
>
> 1)   Most writes go to a commitlog (append-only file, written
> linearly, so particularly fast in terms of disk operations) and then pushed
> to the memTable. Memtable is flushed in batches to the permanent data
> files, so it buffers many mutations and then does a sequential write to
> persist that data to disk.
>
> 2)   Reads may have to merge data from many data tables on disk.
> Because the writes (described very briefly in step 1) write to immutable
> files, updates/deletes have to be merged on read – this is extra effort for
> the read path.
>
>
>
> If you don’t do much in terms of overwrites/deletes, and your partitions
> are particularly small, and your data fits in RAM (probably mmap/page cache
> of data files, unless you’re using the row cache), reads may be very fast
> for you. Certainly individual reads on low-merge workloads can be < 0.1ms.
>
>
>
> -  Jeff
>
>
>
> *From: *Vikas Jaiman 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Sunday, November 6, 2016 at 12:42 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Are Cassandra writes are faster than reads?
>
>
>
> Hi all,
>
>
>
> Are Cassandra writes are faster than reads ?? If yes, why is this so? I am
> using consistency 1 and data is in memory.
>
>
>
> Vikas
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Are Cassandra writes are faster than reads?

[Ben Bromhead]

Check out https://wiki.apache.org/cassandra/WritePathForUsers for the full
gory details.

On Sun, 6 Nov 2016 at 21:09 Ali Akhtar  wrote:

> How long does it take for updates to get merged / compacted into the main
> data file?
>
> On Mon, Nov 7, 2016 at 5:31 AM, Ben Bromhead  wrote:
>
> To add some flavor as to how the commitlog implementation is so quick.
>
> It only flushes to disk every 10s by default. So writes are effectively
> done to memory and then to disk asynchronously later on. This is generally
> accepted to be OK, as the write is also going to other nodes.
>
> You can of course change this behavior to flush on each write or to skip
> the commitlog altogether (danger!). This however will change how "safe"
> things are from a durability perspective.
>
> On Sun, Nov 6, 2016, 12:51 Jeff Jirsa  wrote:
>
> Cassandra writes are particularly fast, for a few reasons:
>
>
>
> 1)   Most writes go to a commitlog (append-only file, written
> linearly, so particularly fast in terms of disk operations) and then pushed
> to the memTable. Memtable is flushed in batches to the permanent data
> files, so it buffers many mutations and then does a sequential write to
> persist that data to disk.
>
> 2)   Reads may have to merge data from many data tables on disk.
> Because the writes (described very briefly in step 1) write to immutable
> files, updates/deletes have to be merged on read – this is extra effort for
> the read path.
>
>
>
> If you don’t do much in terms of overwrites/deletes, and your partitions
> are particularly small, and your data fits in RAM (probably mmap/page cache
> of data files, unless you’re using the row cache), reads may be very fast
> for you. Certainly individual reads on low-merge workloads can be < 0.1ms.
>
>
>
> -  Jeff
>
>
>
> *From: *Vikas Jaiman 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Sunday, November 6, 2016 at 12:42 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Are Cassandra writes are faster than reads?
>
>
>
> Hi all,
>
>
>
> Are Cassandra writes are faster than reads ?? If yes, why is this so? I am
> using consistency 1 and data is in memory.
>
>
>
> Vikas
>
> --
> Ben Bromhead
> CTO | Instaclustr <https://www.instaclustr.com/>
> +1 650 284 9692
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Are Cassandra writes are faster than reads?

[Ben Bromhead]

They can be and it depends on your compaction strategy :)

On Sun, 6 Nov 2016 at 21:24 Ali Akhtar  wrote:

> tl;dr? I just want to know if updates are bad for performance, and if so,
> for how long.
>
> On Mon, Nov 7, 2016 at 10:23 AM, Ben Bromhead  wrote:
>
> Check out https://wiki.apache.org/cassandra/WritePathForUsers for the
> full gory details.
>
> On Sun, 6 Nov 2016 at 21:09 Ali Akhtar  wrote:
>
> How long does it take for updates to get merged / compacted into the main
> data file?
>
> On Mon, Nov 7, 2016 at 5:31 AM, Ben Bromhead  wrote:
>
> To add some flavor as to how the commitlog implementation is so quick.
>
> It only flushes to disk every 10s by default. So writes are effectively
> done to memory and then to disk asynchronously later on. This is generally
> accepted to be OK, as the write is also going to other nodes.
>
> You can of course change this behavior to flush on each write or to skip
> the commitlog altogether (danger!). This however will change how "safe"
> things are from a durability perspective.
>
> On Sun, Nov 6, 2016, 12:51 Jeff Jirsa  wrote:
>
> Cassandra writes are particularly fast, for a few reasons:
>
>
>
> 1)   Most writes go to a commitlog (append-only file, written
> linearly, so particularly fast in terms of disk operations) and then pushed
> to the memTable. Memtable is flushed in batches to the permanent data
> files, so it buffers many mutations and then does a sequential write to
> persist that data to disk.
>
> 2)   Reads may have to merge data from many data tables on disk.
> Because the writes (described very briefly in step 1) write to immutable
> files, updates/deletes have to be merged on read – this is extra effort for
> the read path.
>
>
>
> If you don’t do much in terms of overwrites/deletes, and your partitions
> are particularly small, and your data fits in RAM (probably mmap/page cache
> of data files, unless you’re using the row cache), reads may be very fast
> for you. Certainly individual reads on low-merge workloads can be < 0.1ms.
>
>
>
> -  Jeff
>
>
>
> *From: *Vikas Jaiman 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Sunday, November 6, 2016 at 12:42 PM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Are Cassandra writes are faster than reads?
>
>
>
> Hi all,
>
>
>
> Are Cassandra writes are faster than reads ?? If yes, why is this so? I am
> using consistency 1 and data is in memory.
>
>
>
> Vikas
>
> --
> Ben Bromhead
> CTO | Instaclustr <https://www.instaclustr.com/>
> +1 650 284 9692
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>
>
> --
> Ben Bromhead
> CTO | Instaclustr <https://www.instaclustr.com/>
> +1 650 284 9692
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Are Cassandra writes are faster than reads?

[Ben Bromhead]

Awesome! For a full explanation of what you are seeing (we call it micro
batching) check out Adam Zegelins talk on it
https://www.youtube.com/watch?v=wF3Ec1rdWgc

On Tue, 8 Nov 2016 at 02:21 Rajesh Radhakrishnan <
rajesh.radhakrish...@phe.gov.uk> wrote:

>
> Hi,
>
> Just found that reducing the batch size below 20 also increases the
> writing speed and reduction in memory usage(especially for Python driver).
>
> Kind regards,
> Rajesh R
>
> ------
> *From:* Ben Bromhead [b...@instaclustr.com]
> *Sent:* 07 November 2016 05:44
> *To:* user@cassandra.apache.org
> *Subject:* Re: Are Cassandra writes are faster than reads?
>
> They can be and it depends on your compaction strategy :)
>
> On Sun, 6 Nov 2016 at 21:24 Ali Akhtar  <http://redir.aspx?REF=KvuN_F91CkILmAKkPOD8RLOkpaObm4vWZ4CTx2PNAjG8Cvd6wAfUCAFtYWlsdG86YWxpLnJhYzIwMEBnbWFpbC5jb20.>>
> wrote:
>
> tl;dr? I just want to know if updates are bad for performance, and if so,
> for how long.
>
> On Mon, Nov 7, 2016 at 10:23 AM, Ben Bromhead  <http://redir.aspx?REF=bOLz-2Z_cjZ-R5mW4ySFRmRgIvYoWF43pRrpxxUsOOC8Cvd6wAfUCAFtYWlsdG86YmVuQGluc3RhY2x1c3RyLmNvbQ..>
> > wrote:
>
> Check out https://wiki.apache.org/cassandra/WritePathForUsers
> <http://redir.aspx?REF=z6gebtTM9Bi4b1ZEZqnpcgJOwnifCWloccEOX28F8UC8Cvd6wAfUCAFodHRwczovL3dpa2kuYXBhY2hlLm9yZy9jYXNzYW5kcmEvV3JpdGVQYXRoRm9yVXNlcnM.>
>  for
> the full gory details.
>
> On Sun, 6 Nov 2016 at 21:09 Ali Akhtar  <http://redir.aspx?REF=KvuN_F91CkILmAKkPOD8RLOkpaObm4vWZ4CTx2PNAjG8Cvd6wAfUCAFtYWlsdG86YWxpLnJhYzIwMEBnbWFpbC5jb20.>>
> wrote:
>
> How long does it take for updates to get merged / compacted into the main
> data file?
>
> On Mon, Nov 7, 2016 at 5:31 AM, Ben Bromhead  <http://redir.aspx?REF=bOLz-2Z_cjZ-R5mW4ySFRmRgIvYoWF43pRrpxxUsOOC8Cvd6wAfUCAFtYWlsdG86YmVuQGluc3RhY2x1c3RyLmNvbQ..>
> > wrote:
>
> To add some flavor as to how the commitlog implementation is so quick.
>
> It only flushes to disk every 10s by default. So writes are effectively
> done to memory and then to disk asynchronously later on. This is generally
> accepted to be OK, as the write is also going to other nodes.
>
> You can of course change this behavior to flush on each write or to skip
> the commitlog altogether (danger!). This however will change how "safe"
> things are from a durability perspective.
>
> On Sun, Nov 6, 2016, 12:51 Jeff Jirsa  <http://redir.aspx?REF=CSJmlUdwjTSoe3NQdZNlO6pFPeaI_KxNpZweB-GbDYO8Cvd6wAfUCAFtYWlsdG86amVmZi5qaXJzYUBjcm93ZHN0cmlrZS5jb20.>>
> wrote:
>
> Cassandra writes are particularly fast, for a few reasons:
>
>
>
> 1)   Most writes go to a commitlog (append-only file, written
> linearly, so particularly fast in terms of disk operations) and then pushed
> to the memTable. Memtable is flushed in batches to the permanent data
> files, so it buffers many mutations and then does a sequential write to
> persist that data to disk.
>
> 2)   Reads may have to merge data from many data tables on disk.
> Because the writes (described very briefly in step 1) write to immutable
> files, updates/deletes have to be merged on read – this is extra effort for
> the read path.
>
>
>
> If you don’t do much in terms of overwrites/deletes, and your partitions
> are particularly small, and your data fits in RAM (probably mmap/page cache
> of data files, unless you’re using the row cache), reads may be very fast
> for you. Certainly individual reads on low-merge workloads can be < 0.1ms.
>
>
>
> -  Jeff
>
>
>
> *From: *Vikas Jaiman  <http://redir.aspx?REF=VgqqnBUEzP6sLWofnDxFp3iyHQ4TGCTJL8MbqH0NOUK8Cvd6wAfUCAFtYWlsdG86ZXIudmlrYXNqYWltYW5AZ21haWwuY29t>
> >
> *Reply-To: *"user@cassandra.apache.org
> <http://redir.aspx?REF=yxCMb2E-WgRKlJCeCUpFf-0-Th-NE4pZJyZdWo0SRMS8Cvd6wAfUCAFtYWlsdG86dXNlckBjYXNzYW5kcmEuYXBhY2hlLm9yZw..>"
>  <http://redir.aspx?REF=yxCMb2E-WgRKlJCeCUpFf-0-Th-NE4pZJyZdWo0SRMS8Cvd6wAfUCAFtYWlsdG86dXNlckBjYXNzYW5kcmEuYXBhY2hlLm9yZw..>
> >
> *Date: *Sunday, November 6, 2016 at 12:42 PM
> *To: *"user@cassandra.apache.org
> <http://redir.aspx?REF=yxCMb2E-WgRKlJCeCUpFf-0-Th-NE4pZJyZdWo0SRMS8Cvd6wAfUCAFtYWlsdG86dXNlckBjYXNzYW5kcmEuYXBhY2hlLm9yZw..>"
>  <http://redir.aspx?REF=yxCMb2E-WgRKlJCeCUpFf-0-Th-NE4pZJyZdWo0SRMS8Cvd6wAfUCAFtYWlsdG86dXNlckBjYXNzYW5kcmEuYXBhY2hlLm9yZw..>
> >
> *Subject: *Are Cassandra writes are faster than reads?
>
>
>
> Hi all,
>
>
>
> Are Cassandra writes are faster than reads ?? If yes, why is this so? I am
> using consistency 1 and data is in memory.
>
>
>
> Vikas
>
> --
> Ben Bromhead
> CTO | Instaclustr

Re: Priority for cassandra nodes in cluster

[Ben Bromhead]

+1 w/ Benjamin.

However if you wish to make use of spare hardware capacity, look to
something like mesos DC/OS or kubernetes. You can run multiple services
across a fleet of hardware, but provision equal resources to Cassandra and
have somewhat reliable hardware sharing mechanisms.

On Sat, 12 Nov 2016 at 14:12 Jon Haddad  wrote:

> Agreed w/ Benjamin.  Trying to diagnose issues in prod will be a
> nightmare.  Keep your DB servers homogeneous.
>
> On Nov 12, 2016, at 1:52 PM, Benjamin Roth 
> wrote:
>
> 1. From a 15 year experience of running distributed Services: dont Mix
> Services on machines if you don't have to. Dedicate each server to a single
> task if you can afford it. It is easier to manage and reduces risks in case
> of overload or failure
> 2. You can assign a different number of tokens for each node by setting
> this in Cassandra.yaml before you bootstrap that node
>
> Am 12.11.2016 22:48 schrieb "sat" :
>
> Hi,
>
> We are planning to install 3 node cluster in production environment. Is it
> possible to provide weightage or priority to the nodes in cluster.
>
> Eg., We want more more records to be written to first 2 nodes and less to
> the 3rd node. We are thinking of this approach because we want to install
> other IO intensive messaging server in the 3rd node, in order to reduce the
> load we are requesting for this approach.
>
>
> Thanks and Regards
> A.SathishKumar
>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: High system CPU during high write workload

[Ben Bromhead]

Hi Abhishek

The article with the futex bug description lists the solution, which is to
upgrade to a version of RHEL or CentOS that have the specified patch.

What help do you specifically need? If you need help upgrading the OS I
would look at the documentation for RHEL or CentOS.

Ben

On Mon, 14 Nov 2016 at 22:48 Abhishek Gupta 
wrote:

Hi,

We are seeing an issue where the system CPU is shooting off to a figure or
> 90% when the cluster is subjected to a relatively high write workload i.e
4k wreq/secs.

2016-11-14T13:27:47.900+0530 Process summary
  process cpu=695.61%
  application cpu=676.11% (*user=200.63% sys=475.49%) **<== Very High
System CPU *
  other: cpu=19.49%
  heap allocation rate *403mb*/s
[000533] user= 1.43% sys= 6.91% alloc= 2216kb/s - SharedPool-Worker-129
[000274] user= 0.38% sys= 7.78% alloc= 2415kb/s - SharedPool-Worker-34
[000292] user= 1.24% sys= 6.77% alloc= 2196kb/s - SharedPool-Worker-56
[000487] user= 1.24% sys= 6.69% alloc= 2260kb/s - SharedPool-Worker-79
[000488] user= 1.24% sys= 6.56% alloc= 2064kb/s - SharedPool-Worker-78
[000258] user= 1.05% sys= 6.66% alloc= 2250kb/s - SharedPool-Worker-41

On doing strace it was found that the following system call is consuming
all the system CPU
 timeout 10s strace -f -p 5954 -c -q
% time seconds  usecs/call callserrors syscall
-- --- --- - - 

*88.33 1712.798399   16674102723 22191 futex* 3.98   77.098730
   4356 17700   read
 3.27   63.474795  394253   16129 restart_syscall
 3.23   62.601530   29768  2103   epoll_wait

On searching we found the following bug with the RHEL 6.6, CentOS 6.6
kernel seems to be a probable cause for the issue:

https://docs.datastax.com/en/landing_page/doc/landing_page/troubleshooting/cassandra/fetuxWaitBug.html

The patch fix mentioned in the doc is also not present in our kernel.

sudo rpm -q --changelog kernel-`uname -r` | grep futex | grep ref
- [kernel] futex_lock_pi() key refcnt fix (Danny Feng) [566347]
{CVE-2010-0623}

Can some who has faced and resolved this issue help us here.

Thanks,
Abhishek


-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Any Bulk Load on Large Data Set Advice?

[Ben Bromhead]

+1 on parquet and S3.

Combined with spark running on spot instances your grant money will go much
further!

On Thu, 17 Nov 2016 at 07:21 Jonathan Haddad  wrote:

> If you're only doing this for spark, you'll be much better off using
> parquet and HDFS or S3. While you *can* do analytics with cassandra, it's
> not all that great at it.
> On Thu, Nov 17, 2016 at 6:05 AM Joe Olson  wrote:
>
> I received a grant to do some analysis on netflow data (Local IP address,
> Local Port, Remote IP address, Remote Port, time, # of packets, etc) using
> Cassandra and Spark. The de-normalized data set is about 13TB out the door.
> I plan on using 9 Cassandra nodes (replication factor=3) to store the data,
> with Spark doing the aggregation.
>
> Data set will be immutable once loaded, and am using the replication
> factor = 3 to somewhat simulate the real world. Most of the analysis will
> be of the sort "Give me all the remote ip addresses for source IP 'X'
> between time t1 and t2"
>
> I built and tested a bulk loader following this example in GitHub:
> https://github.com/yukim/cassandra-bulkload-example to generate the
> SSTables, but I have not executed it on the entire data set yet.
>
> Any advice on how to execute the bulk load under this configuration?  Can
> I generate the SSTables in parallel? Once generated, can I write the
> SSTables to all nodes simultaneously? Should I be doing any kind of sorting
> by the partition key?
>
> This is a lot of data, so I figured I'd ask before I pulled the trigger.
> Thanks in advance!
>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Is it *safe* to issue multiple replace-node at the same time?

[Ben Bromhead]

Same rack and no range movements, my first instinct is to say yes it is
safe (I like to treat racks as one giant meta node). However I would want
to have a read through the replace code.

On Mon, Nov 21, 2016, 07:22 Dikang Gu  wrote:

> Hi guys,
>
> Sometimes we need to replace multiple hosts in the same rack, is it safe
> to replace them in parallel, using the replace-node command?
>
> Will it cause any data inconsistency if we do so?
>
> Thanks
> Dikang.
>
> --
> Dikang
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Clarify Support for 2.2 on Download Page

[Ben Bromhead]

Hi Derek

You should subscribe and post this question to the Dev list, they will be
able to get you sorted quickly!

Normally you can edit documentation directly via github (e.g.
https://github.com/apache/cassandra/tree/trunk/doc/source), however the
download source appears to be outside the Cassandra repo.

Ben



On Wed, 16 Nov 2016 at 13:08 Derek Burdick  wrote:

> Hi, is it possible to update the language on the Apache Cassandra Download
> page to reflect that version 2.2 will enter Critical Fix Only support after
> November 21st?
>
> The current language creates quite a bit of confusion in the community
> with how long 2.2 and 2.1 will receive fixes from the community.
>
> http://cassandra.apache.org/download/
>
> Specifically these three lines:
>
>- Apache Cassandra 3.0 is supported until May 2017. The latest release
>is 3.0.9
>
> <http://www.apache.org/dyn/closer.lua/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz>
> (pgp
>
> <http://www.apache.org/dist/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz.asc>
>, md5
>
> <http://www.apache.org/dist/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz.md5>
> and sha1
>
> <http://www.apache.org/dist/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz.sha1>),
>released on 2016-09-20.
>- Apache Cassandra 2.2 is supported until November 2016. The latest
>release is 2.2.8
>
> <http://www.apache.org/dyn/closer.lua/cassandra/2.2.8/apache-cassandra-2.2.8-bin.tar.gz>
> (pgp
>
> <http://www.apache.org/dist/cassandra/2.2.8/apache-cassandra-2.2.8-bin.tar.gz.asc>
>, md5
>
> <http://www.apache.org/dist/cassandra/2.2.8/apache-cassandra-2.2.8-bin.tar.gz.md5>
> and sha1
>
> <http://www.apache.org/dist/cassandra/2.2.8/apache-cassandra-2.2.8-bin.tar.gz.sha1>),
>released on 2016-09-28.
>- Apache Cassandra 2.1 is supported until November 2016 with critical
>fixes only. The latest release is 2.1.16
>
> <http://www.apache.org/dyn/closer.lua/cassandra/2.1.16/apache-cassandra-2.1.16-bin.tar.gz>
> (pgp
>
> <http://www.apache.org/dist/cassandra/2.1.16/apache-cassandra-2.1.16-bin.tar.gz.asc>
>, md5
>
> <http://www.apache.org/dist/cassandra/2.1.16/apache-cassandra-2.1.16-bin.tar.gz.md5>
> and sha1
>
> <http://www.apache.org/dist/cassandra/2.1.16/apache-cassandra-2.1.16-bin.tar.gz.sha1>),
>released on 2016-10-10.
>
>
> What would be the best approach to help get this changed?
>
> -Derek
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Why does `now()` produce different times within the same query?

[Ben Bromhead]

ock skew on the thousands of
> sensors that write to our cluster.
>
> Thanks,
> Cody
>
> On Wed, Nov 30, 2016 at 5:33 PM, Robert Wille  wrote:
>
> In my opinion, this is not broken and “fixing” it would break existing
> code. Consider a batch that includes multiple inserts, each of which
> inserts the value returned by now(). Getting the same UUID for each insert
> would be a major problem.
>
> Cheers
>
> Robert
>
>
> On Nov 30, 2016, at 4:46 PM, Todd Fast  wrote:
>
> FWIW I'd suggest opening a bug--this behavior is certainly quite
> unexpected and more than just a documentation issue. In general I can't
> imagine any desirable properties of the current implementation, and there
> are likely a bunch of latent bugs sitting out there, so it should be fixed.
>
> Todd
>
> On Wed, Nov 30, 2016 at 12:37 PM Terry Liu  wrote:
>
> Sorry for my typo. Obviously, I meant:
> "It appears that a single query that calls Cassandra's`now()` time
> function *multiple times *may actually cause a query to write or return
> different times."
>
> Less of a surprise now that I realize more about the implementation, but I
> agree that more explicit documentation around when exactly the "execution"
> of each now() statement happens and what implications it has for the
> resulting timestamps would be helpful when running into this.
>
> Thanks for the quick responses!
>
> -Terry
>
>
>
> On Tue, Nov 29, 2016 at 2:45 PM, Marko Švaljek  wrote:
>
> every now() call in statement is under the hood "replaced" with newly
> generated uuid.
>
> It can happen that they belong to  different milliseconds in time.
>
> If you need to have same timestamps you need to set them on the client
> side.
>
>
> @msvaljek <https://twitter.com/msvaljek>
>
> 2016-11-29 22:49 GMT+01:00 Terry Liu :
>
> It appears that a single query that calls Cassandra's `now()` time
> function may actually cause a query to write or return different times.
>
> Is this the expected or defined behavior, and if so, why does it behave
> like this rather than evaluating `now()` once across an entire statement?
>
> This really affects UPDATE statements but to test it more easily, you
> could try something like:
>
> SELECT toTimestamp(now()) as a, toTimestamp(now()) as b
> FROM keyspace.table
> LIMIT 100;
>
> If you run that a few times, you should eventually see that the timestamp
> returned moves onto the next millisecond mid-query.
>
> --
> *Software Engineer*
> Turnitin - http://www.turnitin.com
> t...@turnitin.com
>
>
>
>
>
> --
> *Software Engineer*
> Turnitin - http://www.turnitin.com
> t...@turnitin.com
>
>
>
>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Why does `now()` produce different times within the same query?

[Ben Bromhead]

>
>
>
> I will note that Ben seems to suggest keeping the return of now() unique
> across
> call while keeping the time component equals, thus varying the rest of the
> uuid
> bytes. However:
>  - I'm starting to wonder what this would buy us. Why would someone be
> super
>confused by the time changing across calls (in a single
> statement/batch), but
>be totally not confused by the actual full return to not be equal?
>
Given that a common way of interacting with timeuuids is with toTimestamp I
can see the confusion and assumptions around behaviour.

And how is
>that actually useful: you're having different result anyway and you're
>letting the server pick the timestamp in the first place, so you're
> probably
>not caring about milliseconds precision of that timestamp in the first
> place.
>
If you want consistency of timestamps within your query as OP did I can see
how this is useful. Postgres claims this is a "feature".

 - This would basically be a violation of the timeuuid spec
>

Not quite... Type 1 uuids let you swap out the low 47 bits of the node
component with other randomly generated bits (
https://www.ietf.org/rfc/rfc4122.txt)

 - This would be a big pain in the code and make of now() a special case
> among functions. I'm unconvinced special cases are making things easier
> in general.
>

On reflection, I have to agree here, now() has been around for ever and
this is the first anecdote I've seen of someone getting caught out.

However with my user advocate hat on I think it would be worth
investigating further beyond a documentation update if others found it a
sticking point in Cassandra adoption.


> So I'm all for improving the documentation if this confuses users due to
> expectations (mistakenly) carried from prior experiences, and please
> feel free to open a JIRA for that. I'm a lot less in agreement that there
> is
> something wrong with the way the function behave in principle.
>


> > I can see why this issue has been largely ignored and hasn't had a
> chance for
> > the behaviour to be formally defined
>
> Don't make too much assumptions. The behavior is perfectly well defined:
> now()
> is a "normal" function and is evaluated whenever it's called according to
> the
> timeuuid spec (or as close to it as we can make it).
>
Maybe formally defined is the wrong term... Formally documented?

>
> On Thu, Dec 1, 2016 at 7:25 AM, Benjamin Roth 
> wrote:
>
> Great comment. +1
>
> Am 01.12.2016 06:29 schrieb "Ben Bromhead" :
>
> tl;dr +1 yup raise a jira to discuss how now() should behave in a single
> statement (and possible extend to batch statements).
>
> The values of now should be the same if you assume that now() works like
> it does in relational databases such as postgres or mysql, however at the
> moment it instead works like sysdate() in mysql. Given that CQL is supposed
> to be SQL like, I think the assumption around the behaviour of now() was a
> fair one to make.
>
> I definitely agree that raising a jira ticket would be a great place to
> discuss what the behaviour of now() should be for Cassandra. Personally I
> would be in favour of seeing the deterministic component (the actual time
> part) being the same across multiple calls in the one statement or multiple
> statements in a batch.
>
> Cassandra documentation does not make any claims as to how now() works
> within a single statement and reading the code it shows the intent is to
> work like sysdate() from MySQL rather than now(). One of the identified
> dangers of making cql similar to sql is that, while yes it aids adoption,
> users will find that SQL like things don't behave as expected. Of course as
> a user, one shouldn't have to read the source code to determine correct
> behaviour.
>
> Given that a timeuuid is made up of deterministic and (pseudo)
> non-deterministic components I can see why this issue has been largely
> ignored and hasn't had a chance for the behaviour to be formally defined
> (you would expect now to return the same time in the one statement despite
> multiple calls, but you wouldn't expect the same behaviour for say a call
> to rand()).
>
>
>
>
>
>
>
> On Wed, 30 Nov 2016 at 19:54 Cody Yancey  wrote:
>
> This is not a bug, and in fact changing it would be a serious bug.
>
> False. Absolutely no consumer would be broken by a change to guarantee an
> identical time component that isn't broken already, for the simple reason
> your code already has to handle that case, as it is in fact the majority
> case RIGHT NOW. Users can hit this bug, in production, because unit tests
> might not 

Re: [Announcement] BigGraphite

[Ben Bromhead]

This looks super cool

On Tue, 6 Dec 2016 at 01:49 DuyHai Doan  wrote:

> Hello Cassandra users
>
> On behalf of Corentin Chary from Criteo, I'm happy to announce the release
> of BigGraphite, an OSS Graphite plugin  dedicated for time series based on
> Apache Cassandra. BigGraphite is leveraging latest features as SASI index &
> Back Pressure.
>
> More info here:
>
> https://github.com/criteo/biggraphite/wiki/BigGraphite-Announcement
> http://labs.criteo.com/blog/
>
> Regards
>
> Duy Hai DOAN
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Cassandra Authentication

[Ben Bromhead]

We have a process that syncs and manages RF==N and we also control and
manage users, however that entails it's own set of challenges and
maintenance.

For most users I would suggest 3 < RF <=5 is sufficient. Also make sure you
don't use the user "Cassandra" in production as authentication queries are
done at QUORUM.

On Wed, 18 Jan 2017 at 13:41 Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Hello,
>
> When enabling Authentication on cassandra, is it required to set the RF
> same as the no.of nodes(
> https://docs.datastax.com/en/cql/3.1/cql/cql_using/update_ks_rf_t.html)?
> or can I live with RF of 3 in each DC (other KS are using 3)
>
> If it has to be equal to the number of nodes then, every time adding or
> removing a node requires update of RF.
>
> Thanks in advance.
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Cassandra Authentication

[Ben Bromhead]

the volume of data is pretty low + you still want to be able to
authenticate even if you have more nodes down than the RF for other
keyspaces. Essentially you don't want auth to be the thing that stops you
serving requests.

On Wed, 18 Jan 2017 at 14:57 Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Thanks Ben,
>
> RF 3 isn't sufficient for system_auth? as we are using 3 RF for other
> production KS, do you see any challenges?
>
> On Wed, Jan 18, 2017 at 2:39 PM, Ben Bromhead  wrote:
>
> We have a process that syncs and manages RF==N and we also control and
> manage users, however that entails it's own set of challenges and
> maintenance.
>
> For most users I would suggest 3 < RF <=5 is sufficient. Also make sure
> you don't use the user "Cassandra" in production as authentication queries
> are done at QUORUM.
>
> On Wed, 18 Jan 2017 at 13:41 Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
> Hello,
>
> When enabling Authentication on cassandra, is it required to set the RF
> same as the no.of nodes(
> https://docs.datastax.com/en/cql/3.1/cql/cql_using/update_ks_rf_t.html)?
> or can I live with RF of 3 in each DC (other KS are using 3)
>
> If it has to be equal to the number of nodes then, every time adding or
> removing a node requires update of RF.
>
> Thanks in advance.
>
> --
> Ben Bromhead
> CTO | Instaclustr <https://www.instaclustr.com/>
> +1 650 284 9692 <+1%20650-284-9692>
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Authentication with Java driver

[Ben Bromhead]

What are you specifically trying to achieve? Are you trying to authenticate
multiple Cassandra users from a single application instance? Or will your
have lot's of application instances connecting to Cassandra using the same
set of credentials? Or a combination of both? Multiple application
instances with different credentials?

On Tue, 7 Feb 2017 at 06:19 Andrew Tolbert 
wrote:

> Hello,
>
> The API seems kind of not correct because credentials should be
> usually set with a session but actually they are set with a cluster.
>
>
> With the datastax driver, Session is what manages connection pools to
> each node.  Cluster manages configuration and a separate connection
> ('control connection') to subscribe to state changes (schema changes, node
> topology changes, node up/down events).
>
>
> So, if there are 1000 clients, then with this API it has to create
> 1000 cluster instances ?
>
>
> I'm unsure how common it is for per-user authentication to be done when
> connecting to the database.  I think an application would normally
> authenticate with one set of credentials instead of multiple.  The protocol
> Cassandra uses does authentication at the connection level instead of at
> the request level, so that is currently a limitation to support something
> like reusing Sessions for authenticating multiple users.
>
> Thanks,
> Andy
>
>
> On Tue, Feb 7, 2017 at 7:19 AM Hiroyuki Yamada  wrote:
>
> Hi,
>
> The API seems kind of not correct because credentials should be
> usually set with a session but actually they are set with a cluster.
>
> So, if there are 1000 clients, then with this API it has to create
> 1000 cluster instances ?
> 1000 clients seems usual if there are many nodes (say 20) and each
> node has some concurrency (say 50),
> but 1000 cluster instances seems too many.
>
> Is this an expected way to do this ? or
> Is there any way to authenticate per session ?
>
> Thanks,
> Hiro
>
> On Tue, Feb 7, 2017 at 11:38 AM, Yuji Ito  wrote:
> > Hi all,
> >
> > I want to know how to authenticate Cassandra users for multiple instances
> > with Java driver.
> > For instance, each thread creates a instance to access Cassandra with
> > authentication.
> >
> > As the implementation example, only the first constructor builds a
> cluster
> > and a session.
> > Other constructors use them.
> > This example is implemented according to the datastax document:
> "Basically
> > you will want to share the same cluster and session instances across your
> > application".
> >
> http://www.datastax.com/dev/blog/4-simple-rules-when-using-the-datastax-drivers-for-cassandra
> >
> > However, other constructors don't authenticate the user and the password.
> > That's because they don't need to build a cluster and a session.
> >
> > So, should I create a cluster and a session per instance for the
> > authentication?
> > If yes, can I create a lot of instances(clusters and sessions) to access
> C*
> > concurrently?
> >
> > == example ==
> > public class A {
> >   private static Cluster cluster = null;
> >   private static Map sessions = null;
> >   private Session session;
> >
> >   public A (String keyspace, String user, String password) {
> > if (cluster == null) {
> > builder = Cluster.builder();
> > ...
> > builder = builder.withCredentials(user, password);
> >     cluster = builder.build();
> > }
> > session = sessions.get(keyspace);
> > if (session == null) {
> > session = cluster.connection(keyspace);
> > sessions.put(keyspace, session)
> > }
> > ...
> >   }
> >   ...
> >   public ResultSet update(...) {
> >   ...
> >   public ResultSet get(...) {
> >   ...
> > }
> > == example ==
> >
> > Thanks,
> > Yuji
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Authentication with Java driver

[Ben Bromhead]

On Tue, 7 Feb 2017 at 17:52 Yuji Ito  wrote:

Thanks Andrew, Ben,

My application creates a lot of instances connecting to Cassandra with
basically the same set of credentials.

Do you mean lots of instances of the process or lots of instances of the
cluster/session object?


After an instance connects to Cassandra with the credentials, can any
instance connect to Cassandra without credentials?

As long as you don't share the session or cluster objects. Each new
cluster/session will need to reauthenticate.


== example ==
A first = new A("database", "user", "password");  // proper credentials
r = first.get();
...
A other = new A("database", "user", "pass"); // wrong password
r = other.get();
== example ==

I want to refuse the `other` instance with improper credentials.


This looks like you are creating new cluster/session objects (filling in
the blanks for your pseudocode here). So "other" will not authenticate to
Cassandra.

This brings up a wider point of why you are doing this? Generally most
applications will create a single longed lived session object that lasts
the life of the application process.

I would not rely on Cassandra auth to authenticate downstream actors, not
because it's bad, just its generally inefficient to create lots of session
objects. The session object maintains a connection pool, pipelines
requests, is thread safe and generally pretty solid.




Yuji


On Wed, Feb 8, 2017 at 4:11 AM, Ben Bromhead  wrote:

What are you specifically trying to achieve? Are you trying to authenticate
multiple Cassandra users from a single application instance? Or will your
have lot's of application instances connecting to Cassandra using the same
set of credentials? Or a combination of both? Multiple application
instances with different credentials?

On Tue, 7 Feb 2017 at 06:19 Andrew Tolbert 
wrote:

Hello,

The API seems kind of not correct because credentials should be
usually set with a session but actually they are set with a cluster.


With the datastax driver, Session is what manages connection pools to each
node.  Cluster manages configuration and a separate connection ('control
connection') to subscribe to state changes (schema changes, node topology
changes, node up/down events).


So, if there are 1000 clients, then with this API it has to create
1000 cluster instances ?


I'm unsure how common it is for per-user authentication to be done when
connecting to the database.  I think an application would normally
authenticate with one set of credentials instead of multiple.  The protocol
Cassandra uses does authentication at the connection level instead of at
the request level, so that is currently a limitation to support something
like reusing Sessions for authenticating multiple users.

Thanks,
Andy


On Tue, Feb 7, 2017 at 7:19 AM Hiroyuki Yamada  wrote:

Hi,

The API seems kind of not correct because credentials should be
usually set with a session but actually they are set with a cluster.

So, if there are 1000 clients, then with this API it has to create
1000 cluster instances ?
1000 clients seems usual if there are many nodes (say 20) and each
node has some concurrency (say 50),
but 1000 cluster instances seems too many.

Is this an expected way to do this ? or
Is there any way to authenticate per session ?

Thanks,
Hiro

On Tue, Feb 7, 2017 at 11:38 AM, Yuji Ito  wrote:
> Hi all,
>
> I want to know how to authenticate Cassandra users for multiple instances
> with Java driver.
> For instance, each thread creates a instance to access Cassandra with
> authentication.
>
> As the implementation example, only the first constructor builds a cluster
> and a session.
> Other constructors use them.
> This example is implemented according to the datastax document: "Basically
> you will want to share the same cluster and session instances across your
> application".
>
http://www.datastax.com/dev/blog/4-simple-rules-when-using-the-datastax-drivers-for-cassandra
>
> However, other constructors don't authenticate the user and the password.
> That's because they don't need to build a cluster and a session.
>
> So, should I create a cluster and a session per instance for the
> authentication?
> If yes, can I create a lot of instances(clusters and sessions) to access
C*
> concurrently?
>
> == example ==
> public class A {
>   private static Cluster cluster = null;
>   private static Map sessions = null;
>   private Session session;
>
>   public A (String keyspace, String user, String password) {
> if (cluster == null) {
> builder = Cluster.builder();
> ...
> builder = builder.withCredentials(user, password);
> cluster = builder.build();
> }
> session = sessions.get(keyspace);
> if (session == null) {
> se

Instaclustr Masters scholarship

[Ben Bromhead]

As part of our commitment to contributing back to the Apache Cassandra open
source project and the wider community we are always looking for ways we
can foster knowledge sharing and improve usability of Cassandra itself. One
of the ways we have done so previously was to open up our internal builds
and versions of Cassandra (https://github.com/instaclustr/cassandra).

We have also been looking at a few novel or outside the box ways we can
further contribute back to the community. As such, we are sponsoring a
masters project in conjunction with the Australian based University of
Canberra. Instaclustr’s staff will be available to provide advice and
feedback to the successful candidate.

*Scope*
Distributed database systems are relatively new technology compared to
traditional relational databases. Distributed advantages provide
significant advantages in terms of reliability and scalability but often at
a cost of increased complexity. This complexity presents challenges for
testing of these systems to prove correct operation across all possible
system states. The scope of this masters scholarship is to use the Apache
Cassandra repair process as an example to consider and improve available
approaches to distributed database systems testing.

The repair process in Cassandra is a scheduled process that runs to ensure
the multiple copies of each piece of data that is maintained by Cassandra
are kept synchronised. Correct operation of repairs has been an ongoing
challenge for the Cassandra project partly due to the difficulty in
designing and developing  comprehensive automated tests for this
functionality.

The expected scope of this project is to:

   - survey and understand the existing testing framework available as part
   of the Cassandra project, particularly as it pertains to testing repairs
   - consider, research and develop enhanced approaches to testing of
   repairs
   - submit any successful approaches to the Apache Cassandra project for
   feedback and inclusion in the project code base

Australia is a pretty great place to advance your education and is
welcoming of foreign students.

We are also open to sponsoring a PhD project with a more in depth focus for
the right candidate.

For more details please don't hesitate to get in touch with myself or reach
out to i...@instaclustr.com.

Cheers

Ben
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Authentication with Java driver

[Ben Bromhead]

If the processes are launched separately or you fork before setting up the
cluster object it won't share credentials.

On Wed, Feb 8, 2017, 02:33 Yuji Ito  wrote:

> Thanks Ben,
>
> Do you mean lots of instances of the process or lots of instances of the
> cluster/session object?
>
>
> Lots of instances of the process are generated.
> I wanted to confirm that `other` doesn't authenticate.
>
> If I want to avoid that, my application has to create new cluster/session
> objects per instance.
> But it is inefficient and uncommon.
> So, we aren't sure that the application works when a lot of
> cluster/session objects are created.
> Is it correct?
>
> Thank you,
> Yuji
>
>
>
> On Wed, Feb 8, 2017 at 12:01 PM, Ben Bromhead  wrote:
>
> On Tue, 7 Feb 2017 at 17:52 Yuji Ito  wrote:
>
> Thanks Andrew, Ben,
>
> My application creates a lot of instances connecting to Cassandra with
> basically the same set of credentials.
>
> Do you mean lots of instances of the process or lots of instances of the
> cluster/session object?
>
>
> After an instance connects to Cassandra with the credentials, can any
> instance connect to Cassandra without credentials?
>
> As long as you don't share the session or cluster objects. Each new
> cluster/session will need to reauthenticate.
>
>
> == example ==
> A first = new A("database", "user", "password");  // proper credentials
> r = first.get();
> ...
> A other = new A("database", "user", "pass"); // wrong password
> r = other.get();
> == example ==
>
> I want to refuse the `other` instance with improper credentials.
>
>
> This looks like you are creating new cluster/session objects (filling in
> the blanks for your pseudocode here). So "other" will not authenticate to
> Cassandra.
>
> This brings up a wider point of why you are doing this? Generally most
> applications will create a single longed lived session object that lasts
> the life of the application process.
>
> I would not rely on Cassandra auth to authenticate downstream actors, not
> because it's bad, just its generally inefficient to create lots of session
> objects. The session object maintains a connection pool, pipelines
> requests, is thread safe and generally pretty solid.
>
>
>
>
> Yuji
>
>
> On Wed, Feb 8, 2017 at 4:11 AM, Ben Bromhead  wrote:
>
> What are you specifically trying to achieve? Are you trying to
> authenticate multiple Cassandra users from a single application instance?
> Or will your have lot's of application instances connecting to Cassandra
> using the same set of credentials? Or a combination of both? Multiple
> application instances with different credentials?
>
> On Tue, 7 Feb 2017 at 06:19 Andrew Tolbert 
> wrote:
>
> Hello,
>
> The API seems kind of not correct because credentials should be
> usually set with a session but actually they are set with a cluster.
>
>
> With the datastax driver, Session is what manages connection pools to
> each node.  Cluster manages configuration and a separate connection
> ('control connection') to subscribe to state changes (schema changes, node
> topology changes, node up/down events).
>
>
> So, if there are 1000 clients, then with this API it has to create
> 1000 cluster instances ?
>
>
> I'm unsure how common it is for per-user authentication to be done when
> connecting to the database.  I think an application would normally
> authenticate with one set of credentials instead of multiple.  The protocol
> Cassandra uses does authentication at the connection level instead of at
> the request level, so that is currently a limitation to support something
> like reusing Sessions for authenticating multiple users.
>
> Thanks,
> Andy
>
>
> On Tue, Feb 7, 2017 at 7:19 AM Hiroyuki Yamada  wrote:
>
> Hi,
>
> The API seems kind of not correct because credentials should be
> usually set with a session but actually they are set with a cluster.
>
> So, if there are 1000 clients, then with this API it has to create
> 1000 cluster instances ?
> 1000 clients seems usual if there are many nodes (say 20) and each
> node has some concurrency (say 50),
> but 1000 cluster instances seems too many.
>
> Is this an expected way to do this ? or
> Is there any way to authenticate per session ?
>
> Thanks,
> Hiro
>
> On Tue, Feb 7, 2017 at 11:38 AM, Yuji Ito  wrote:
> > Hi all,
> >
> > I want to know how to authenticate Cassandra users for multiple instances
> > with Java driver.
> > For instance, each thread creates a instance to access Cassandra with
> &g

Re: compaction falling behind

[Ben Bromhead]

You can do so in two ways:

1) direct observation:
You can keep an eye on the number of pending compactions. This will
fluctuate with load, compaction strategy, ongoing repairs and nodes
bootstrapping but generally the pattern is it should trend towards 0.

There have been a number of bugs in past versions of Cassandra whereby the
number of pending compactions is not reported correctly, so depending on
what version of Cassandra you run this could impact you.

2) Indirect observation
You can keep an eye on metrics that healthy compaction will directly
contribute to. These include the number of sstables per read histogram,
estimated droppable tombstones, tombstones per read etc. You should keep an
eye on these things anyway as they can often show you areas where you can
fine tune compaction or your data model.

Everything exposed by nodetool is consumable via JMX which is great to plug
into your metrics/monitoring/observability system :)

On Mon, 13 Feb 2017 at 13:23 John Sanda  wrote:

> What is a good way to determine whether or not compaction is falling
> behind? I read a couple things earlier that suggest nodetool
> compactionstats might not be the most reliable thing to use.
>
>
>
> - John
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Re : Possibility of using 2 different snitches in the Multi_DC cluster

[Ben Bromhead]

Also you may want to run multiple data centres in the one AWS region (load
segmentation, spark etc). +1 GPFS for everything

On Wed, 3 Feb 2016 at 07:42 sai krishnam raju potturi 
wrote:

> thanks a lot Robert. Greatly appreciate it.
>
> thanks
> Sai
>
> On Tue, Feb 2, 2016 at 6:19 PM, Robert Coli  wrote:
>
>> On Tue, Feb 2, 2016 at 1:23 PM, sai krishnam raju potturi <
>> pskraj...@gmail.com> wrote:
>>
>>> What is the possibility of using GossipingPropertFileSnitch on
>>> datacenters in our private cloud, and Ec2MultiRegionSnitch in AWS?
>>>
>>
>> You should just use GPFS everywhere.
>>
>> This is also the reason why you should not use EC2MRS if you might ever
>> have a DC that is outside of AWS. Just use GPFS.
>>
>> =Rob
>> PS - To answer your actual question... one "can" use different snitches
>> on a per node basis, but ONE REALLY REALLY SHOULDN'T CONSIDER THIS A VALID
>> APPROACH AND IF ONE TRIES AND FAILS I WILL POINT AND LAUGH AND NOT HELP
>> THEM :D
>>
>
> --
Ben Bromhead
CTO | Instaclustr
+1 650 284 9692

Re: Any tips on how to track down why Cassandra won't cluster?

[Ben Bromhead]

Check network connectivity. If you are using public addresses as the
broadcast, make sure you can telnet from one node to the other nodes public
address using the internode port.

Last time I looked into something like this, for some reason if you only
add a security group id to the allowed traffic in a security group you
still need to add public IP addresses for each node in a security groups
allowed inbound traffic as well.

On Wed, 3 Feb 2016 at 11:49 Richard L. Burton III 
wrote:

> I'm deploying 2 nodes at the moment using cassandra-dse on Amazon. I
> configured it to use EC2Snitch and configured rackdc to use us-east with
> rack "1".
>
> The second node points to the first node as the seed e.g., "seeds":
> ["54.*.*.*"] and all of the ports are open.
>
> Any suggestions on how to track down what might trigger this problem? I'm
> not receiving any exceptions.
>
>
> --
> -Richard L. Burton III
> @rburton
>
-- 
Ben Bromhead
CTO | Instaclustr
+1 650 284 9692

Re: EC2 storage options for C*

[Ben Bromhead]

t;>>>>>> only worry I have is EBS outages, which have happened.
>>>>>>>
>>>>>>> On Sunday, January 31, 2016, Jeff Jirsa 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Also in that video - it's long but worth watching
>>>>>>>>
>>>>>>>> We tested up to 1M reads/second as well, blowing out page cache to
>>>>>>>> ensure we weren't "just" reading from memory
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jeff Jirsa
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 31, 2016, at 9:52 AM, Jack Krupansky <
>>>>>>>> jack.krupan...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> How about reads? Any differences between read-intensive and
>>>>>>>> write-intensive workloads?
>>>>>>>>
>>>>>>>> -- Jack Krupansky
>>>>>>>>
>>>>>>>> On Sun, Jan 31, 2016 at 3:13 AM, Jeff Jirsa <
>>>>>>>> jeff.ji...@crowdstrike.com> wrote:
>>>>>>>>
>>>>>>>>> Hi John,
>>>>>>>>>
>>>>>>>>> We run using 4T GP2 volumes, which guarantee 10k iops. Even at 1M
>>>>>>>>> writes per second on 60 nodes, we didn’t come close to hitting even 
>>>>>>>>> 50%
>>>>>>>>> utilization (10k is more than enough for most workloads). PIOPS is not
>>>>>>>>> necessary.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> From: John Wong
>>>>>>>>> Reply-To: "user@cassandra.apache.org"
>>>>>>>>> Date: Saturday, January 30, 2016 at 3:07 PM
>>>>>>>>> To: "user@cassandra.apache.org"
>>>>>>>>> Subject: Re: EC2 storage options for C*
>>>>>>>>>
>>>>>>>>> For production I'd stick with ephemeral disks (aka instance
>>>>>>>>> storage) if you have running a lot of transaction.
>>>>>>>>> However, for regular small testing/qa cluster, or something you
>>>>>>>>> know you want to reload often, EBS is definitely good enough and we 
>>>>>>>>> haven't
>>>>>>>>> had issues 99%. The 1% is kind of anomaly where we have flush blocked.
>>>>>>>>>
>>>>>>>>> But Jeff, kudo that you are able to use EBS. I didn't go through
>>>>>>>>> the video, do you actually use PIOPS or just standard GP2 in your
>>>>>>>>> production cluster?
>>>>>>>>>
>>>>>>>>> On Sat, Jan 30, 2016 at 1:28 PM, Bryan Cheng <
>>>>>>>>> br...@blockcypher.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yep, that motivated my question "Do you have any idea what kind
>>>>>>>>>> of disk performance you need?". If you need the performance, its 
>>>>>>>>>> hard to
>>>>>>>>>> beat ephemeral SSD in RAID 0 on EC2, and its a solid, battle tested
>>>>>>>>>> configuration. If you don't, though, EBS GP2 will save a _lot_ of 
>>>>>>>>>> headache.
>>>>>>>>>>
>>>>>>>>>> Personally, on small clusters like ours (12 nodes), we've found
>>>>>>>>>> our choice of instance dictated much more by the balance of price, 
>>>>>>>>>> CPU, and
>>>>>>>>>> memory. We're using GP2 SSD and we find that for our patterns the 
>>>>>>>>>> disk is
>>>>>>>>>> rarely the bottleneck. YMMV, of course.
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 29, 2016 at 7:32 PM, Jeff Jirsa <
>>>>>>>>>> jeff.ji...@crowdstrike.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> If you have to ask that question, I strongly recommend m4 or c4
>>>>>>>>>>> instances with GP2 EBS.  When you don’t care about replacing a node 
>>>>>>>>>>> because
>>>>>>>>>>> of an instance failure, go with i2+ephemerals. Until then, GP2 EBS 
>>>>>>>>>>> is
>>>>>>>>>>> capable of amazing things, and greatly simplifies life.
>>>>>>>>>>>
>>>>>>>>>>> We gave a talk on this topic at both Cassandra Summit and AWS
>>>>>>>>>>> re:Invent: https://www.youtube.com/watch?v=1R-mgOcOSd4 It’s
>>>>>>>>>>> very much a viable option, despite any old documents online that say
>>>>>>>>>>> otherwise.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> From: Eric Plowe
>>>>>>>>>>> Reply-To: "user@cassandra.apache.org"
>>>>>>>>>>> Date: Friday, January 29, 2016 at 4:33 PM
>>>>>>>>>>> To: "user@cassandra.apache.org"
>>>>>>>>>>> Subject: EC2 storage options for C*
>>>>>>>>>>>
>>>>>>>>>>> My company is planning on rolling out a C* cluster in EC2. We
>>>>>>>>>>> are thinking about going with ephemeral SSDs. The question is this: 
>>>>>>>>>>> Should
>>>>>>>>>>> we put two in RAID 0 or just go with one? We currently run a 
>>>>>>>>>>> cluster in our
>>>>>>>>>>> data center with 2 250gig Samsung 850 EVO's in RAID 0 and we are 
>>>>>>>>>>> happy with
>>>>>>>>>>> the performance we are seeing thus far.
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> Eric
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Steve Robenalt
>>> Software Architect
>>> sroben...@highwire.org 
>>> (office/cell): 916-505-1785
>>>
>>> HighWire Press, Inc.
>>> 425 Broadway St, Redwood City, CA 94063
>>> www.highwire.org
>>>
>>> Technology for Scholarly Communication
>>>
>>
>>
> --
Ben Bromhead
CTO | Instaclustr
+1 650 284 9692

Re: Back to the futex()? :(

[Ben Bromhead]

m/kuhBmHXG> is
>> my cassandra.yaml and here <http://pastebin.com/fyXeTfRa> are my JVM
>> args. I realized I neglected to adjust memtable_flush_writers as I was
>> writing this--so I'll get on that. Aside from that, I'm not sure what to
>> do. (Thanks, again, for reading.)
>>
>> * They were batched for consistency--I'm hoping to return to using them
>> when I'm back at normal load, which is tiny compared to backloading, but
>> the impact on performance was eye-opening.
>> ___
>> Will Hayworth
>> Developer, Engagement Engine
>> Atlassian
>>
>> My pronoun is "they". <http://pronoun.is/they>
>>
>>
>>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Re : decommissioned nodes shows up in "nodetool describecluster" as UNREACHABLE in 2.1.12 version

[Ben Bromhead]

I'm not sure what version of Cassandra you are running so here is some
general advice:

   - Gossip entries for decommissioned nodes will hang around for a few
   days to help catch up nodes in the case of a partition. This is why you see
   the decommissioned nodes listed as LEFT. This is intentional
   - If you keep seeing those entries in your logs and you are on 2.0.x,
   you might be impacted by
   https://issues.apache.org/jira/browse/CASSANDRA-10371. In this case
   upgrade to 2.1 or you can try the work arounds listed in the ticket.

Ben

On Tue, 16 Feb 2016 at 11:09 sai krishnam raju potturi 
wrote:

> hi;
> we have a 12 node cluster across 2 datacenters. We are currently using
> cassandra 2.1.12 version.
>
> SNITCH : GossipingPropertyFileSnitch
>
> When we decommissioned few nodes in a particular datacenter and observed
> the following :
>
> nodetool status shows only the live nodes in the cluster.
>
> nodetool describecluster shows the decommissioned nodes as UNREACHABLE.
>
> nodetool gossipinfo shows the decommissioned nodes as "LEFT"
>
>
> When the live nodes were restarted, "nodetool describecluster" shows only
> the live nodes, which is expected.
>
> Purging the gossip info too did not help.
>
> INFO  17:27:07 InetAddress /X.X.X.X is now DOWN
> INFO  17:27:07 Removing tokens [125897680671740685543105407593050165202,
> 140213388002871593911508364312533329916,
>  98576967436431350637134234839492449485] for /X.X.X.X
> INFO  17:27:07 InetAddress /X.X.X.X is now DOWN
> INFO  17:27:07 Removing tokens [6977666116265389022494863106850615,
> 111270759969411259938117902792984586225,
> 138611464975439236357814418845450428175] for /X.X.X.X
>
> Has anybody experienced similar behaviour. Restarting the entire cluster,
>  everytime a node is decommissioned does not seem right. Thanks in advance
> for the help.
>
>
> thanks
> Sai
>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Tuning chunk_length_kb in cassandra 2.1.12

[Ben Bromhead]

You will need to experiment with chunk_length based on your dataset. At the
end of the day its about finding the sweetspot as chunk_length needs to be
big enough such that you can get a decent compression rate (large chunks
increases the likelihood of a better compression ratio, which means you
will read less from disk) but you also want it to be small so that you are
not reading unrelated data from disk.

But... before you go down the chunk_length testing rabbit hole. Make sure
you are using a sane read_ahead value on the block device your data
directory sits on. For example if you are on AWS and using a raid device
built with mdadm the read_ahead value for the block device can be as high
as 128kb by default. If you are on SSDs you can safely drop it to 8 or 16
(or even 0) and see a big uptick in read performance.

For lots of juicy low level disk tuning and further details see Al Tobey's
guide https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html

On Fri, 29 Jan 2016 at 08:26 Jean Carlo  wrote:

> Hi guys
>
> I want to set the param chunk_length_kb in order to improve the read
> latency of my cassandra_stress's test.
>
> This is the table
>
> CREATE TABLE "Keyspace1".standard1 (
> key blob PRIMARY KEY,
> "C0" blob,
> "C1" blob,
> "C2" blob,
> "C3" blob,
> "C4" blob
> ) WITH bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'sstable_size_in_mb': '160', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
>
> I have 6 columns of type blob. This table is filled by cassandra_stres
>
> admin@cqlsh:Keyspace1> select * from standard1 limit 2;
>
>  key|
> C0 |
> C1 |
> C2 |
> C3 | C4
>
> +++++
>  0x4b343050393536353531 |
> 0xe0e3d68ed1536e4d994aa74860270ac91cf7941acb5eefd925815481298f0d558d4f |
> 0xa43f78202576f1ccbdf50657792fac06f0ca7c9416ee68a08125c8dce4dfd085131d |
> 0xab12b06bf64c73e708d1b96fea9badc678303906e3d5f5f96fae7d8092ee0df0c54c |
> 0x428a157cb598487a1b938bdb6c45b09fad3b6408fddc290a6b332b91426b00ddaeb2 |
> 0x0583038d881ab25be72155bc3aa5cb9ec3aab8e795601abe63a2b35f48ce1e359f5e
>
> I am having a read latency of  ~500 microseconds, I think it takes to much
> time comparing to the write latency of ~30 microseconds.
>
> My first clue is to fix the  chunk_length_kb to a value close to the size
> of the rows in kb
>
> Am I in the right direction? If it is true, how can I compute the size of
> a row?
>
> Other question, the value of "Compacted partition" of the command nodetool
> cfstats migth give me a value close to the chunk_length_kb ?
>
> Best regards
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Sudden disk usage

[Ben Bromhead]

+1 to checking for snapshots. Cassandra by default will automatically
snapshot tables before destructive actions like drop or truncate.

Some general advice regarding cleanup. Cleanup will result in a temporary
increase in both disk I/O load and disk space usage (especially with STCS).
It should only be used as part of a planned increase in capacity when you
still have plenty of disk space left on your existing nodes.

If you are running Cassandra in the cloud (AWS, Azure etc) you can add an
EBS volume, copy your sstables to it then bind mount it to the troubled CF
directory. This will give you some emergency disk space to let compaction
and cleanup do its thing safely.

On Tue, 16 Feb 2016 at 10:57 Robert Coli  wrote:

> On Sat, Feb 13, 2016 at 4:30 PM, Branton Davis  > wrote:
>
>> We use SizeTieredCompaction.  The nodes were about 67% full and we were
>> planning on adding new nodes (doubling the cluster to 6) soon.
>>
>
> Be sure to add those new nodes one at a time.
>
> Have you checked for, and cleared, old snapshots? Snapshots are
> automatically taken at various times and have the unusual property of
> growing larger over time. This is because they are hard links of data files
> and do not take up disk space of their own until the files they link to are
> compacted into new files.
>
> =Rob
>
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Cassandra nodes reduce disks per node

[Ben Bromhead]

you can do this in a "rolling" fashion (one node at a time).

On Wed, 17 Feb 2016 at 14:03 Branton Davis 
wrote:

> We're about to do the same thing.  It shouldn't be necessary to shut down
> the entire cluster, right?
>
> On Wed, Feb 17, 2016 at 12:45 PM, Robert Coli 
> wrote:
>
>>
>>
>> On Tue, Feb 16, 2016 at 11:29 PM, Anishek Agarwal 
>> wrote:
>>>
>>> To accomplish this can I just copy the data from disk1 to disk2 with in
>>> the relevant cassandra home location folders, change the cassanda.yaml
>>> configuration and restart the node. before starting i will shutdown the
>>> cluster.
>>>
>>
>> Yes.
>>
>> =Rob
>>
>>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: „Using Timestamp“ Feature

[Ben Bromhead]

When using client supplied timestamps you need to ensure the clock on the
client is in sync with the nodes in the cluster otherwise behaviour will be
unpredictable.

On Thu, 18 Feb 2016 at 08:50 Tyler Hobbs  wrote:

> 2016-02-18 2:00 GMT-06:00 Matthias Niehoff <
> matthias.nieh...@codecentric.de>:
>
>>
>> * is the 'using timestamp' feature (and providing statement timestamps)
>> sufficiently robust and mature to build an application on?
>>
>
> Yes.  It's been there since the start of CQL3.
>
>
>> * In a BatchedStatement, can different statements have different
>> (explicitly provided) timestamps, or is the BatchedStatement's timestamp
>> used for them all? Is this specified / stable behaviour?
>>
>
> Yes, you can separate timestamps per statement.  And, in fact, if you
> potentially mix inserts and deletes on the same rows, you *should *use
> explicit timestamps with different values.  See the timestamp notes here:
> http://cassandra.apache.org/doc/cql3/CQL.html#batchStmt
>
>
>> * cqhsh reports a syntax error when I use 'using timestamp' with an
>> update statement (works with 'insert'). Is there a good reason for this, or
>> is it a bug?
>>
>
> The "USING TIMESTAMP" goes in a different place in update statements.  It
> should be something like:
>
> UPDATE mytable USING TIMESTAMP ? SET col = ? WHERE key = ?
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: How can I make Cassandra stable in a 2GB RAM node environment ?

[Ben Bromhead]

+1 for
http://opensourceconnections.com/blog/2013/08/31/building-
the-perfect-cassandra-test-environment/
<http://opensourceconnections.com/blog/2013/08/31/building-the-perfect-cassandra-test-environment/>


We also run Cassandra on t2.mediums for our Developer clusters. You can
force Cassandra to do most "memory" things by hitting the disk instead (on
disk compaction passes, flush immediately to disk) and by throttling client
connections. In fact on the t2 series memory is not the biggest concern,
but rather the CPU credit issue.

On Mon, 7 Mar 2016 at 11:53 Robert Coli  wrote:

> On Fri, Mar 4, 2016 at 8:27 PM, Jack Krupansky 
> wrote:
>
>> Please review the minimum hardware requirements as clearly documented:
>>
>> http://docs.datastax.com/en/cassandra/3.x/cassandra/planning/planPlanningHardware.html
>>
>
> That is a document for Datastax Cassandra, not Apache Cassandra. It's
> wonderful that Datastax provides docs, but Datastax Cassandra is a superset
> of Apache Cassandra. Presuming that the requirements of one are exactly
> equivalent to the requirements of the other is not necessarily reasonable.
>
> Please adjust your hardware usage to at least meet the clearly documented
>> minimum requirements. If you continue to encounter problems once you have
>> corrected your configuration error, please resubmit the details with
>> updated hardware configuration details.
>>
>
> Disagree. OP specifically stated that they knew this was not a recommended
> practice. It does not seem unlikely that they are constrained to use this
> hardware for reasons outside of their control.
>
>
>> Just to be clear, development on less than 4 GB is not supported and
>> production on less than 8 GB is not supported. Those are not suggestions or
>> guidelines or recommendations, they are absolute requirements.
>>
>
> What does "supported" mean here? That Datastax will not provide support if
> you do not follow the above recommendations? Because it certainly is
> "supported" in the sense of "it can be made to work" ... ?
>
> The premise of a minimum RAM level seems meaningless without context. How
> much data are you serving from your 2GB RAM node? What is the rate of
> client requests?
>
> To be clear, I don't recommend trying to run production Cassandra with
> under 8GB of RAM on your node, but "absolute requirement" is a serious
> overstatement.
>
>
> http://opensourceconnections.com/blog/2013/08/31/building-the-perfect-cassandra-test-environment/
>
> Has some good discussion of how to run Cassandra in a low memory
> environment. Maybe someone should tell John that his 64MB of JVM heap for a
> test node is 62x too small to be "supported"? :D
>
> =Rob
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Optional TLS CQL Encryption

[Ben Bromhead]

Hi Jason

If you enable encryption it will be always on. Optional encryption is
generally a bad idea (tm). Also always creating a new session every query
is also a bad idea (tm) even without the minimal overhead of encryption.

If you are really hell bent on doing this you could have a node that is
part of the cluster but has -Dcassandra.join_ring=false set in jvm options
in cassandra-env.sh so it does not get any data and configure that to have
no encryption enabled. This is known as a fat client. Then connect to that
specific node whenever you want to do terrible non encrypted things.

Having said all that, please don't do this.

Cheers

On Tue, 19 Apr 2016 at 15:32 Jason J. W. Williams 
wrote:

> Hey Guys,
>
> Is there a way to make TLS encryption optional for the CQL listener? We'd
> like to be able to use for remote management connections but not for same
> datacenter usage (since the build/up  tear down cost is too high for things
> that don't use pools).
>
> Right now it appears if we enable encryption it requires it for all
> connections, which definitely is not what we want.
>
> -J
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Changing Racks of Nodes

[Ben Bromhead]

If the rack as defined in Cassandra stays the same (e.g.
cassandra-rackdc.properties), things will keep working as expected...
except when the actual rack (or fault domain) goes down and you are likely
to lose more nodes than expected.

If you change the rack as defined in Cassandra, the node will start
handling queries it does not have data for.

The best way to change the move racks is to decommission the node, then
bootstrap it with the new rack settings.

On Wed, 20 Apr 2016 at 15:49 Anubhav Kale 
wrote:

> Hello,
>
>
>
> If a running node moves around and changes its rack in the process, when
> its back in the cluster (through ignore-rack property), is it a correct
> statement that queries will not see some data residing on this node until a
> repair is run ?
>
>
>
> Or, is it more like the node may get requests for the data it does not own
> (meaning data will never “disappear”) ?
>
>
>
> I’d appreciate some details on this topic from experts !
>
>
>
> Thanks !
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: trouble setting up initial cluster: Host ID collision between active endpoint

[Ben Bromhead]

Hi Tim

If you want to check out Cassandra on AWS you should also have a look
www.instaclustr.com.

We are still very much in Beta (so if you come across anything, please let
us know), but if you have a few minutes and want to deploy a cluster in
just a few clicks I highly recommend trying Instaclustr out.

Cheers

Ben Bromhead
*Instaclustr*

On Fri, Jan 25, 2013 at 12:35 AM, Tim Dunphy  wrote:

> Cool Thanks for the advice Aaron. I actually did get this working before I
> read your reply. The trick apparently for me was to use the IP for the
> first node in the seeds setting of each successive node. But I like the
> idea of using larges for an hour or so and terminating them for some basic
> experimentation.  Also, thanks for pointing me to the Datastax AMIs I'll be
> sure to check them out.
>
> Tim
>
>
> On Thu, Jan 24, 2013 at 3:45 AM, aaron morton wrote:
>
>> They both have 0 for their token, and this is stored in their System
>> keyspace.
>> Scrub them and start again.
>>
>> But I found that the tokens that were being generated would require way
>> too much memory
>>
>> Token assignments have nothing to do with memory usage.
>>
>>  m1.micro instances
>>
>> You are better off using your laptop than micro instances.
>> For playing around try m1.large and terminate them when not in use.
>> To make life easier use this to make the cluster for you
>> http://www.datastax.com/docs/1.2/install/install_ami
>>
>> Cheers
>>
>>-
>> Aaron Morton
>> Freelance Cassandra Developer
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 24/01/2013, at 5:17 AM, Tim Dunphy  wrote:
>>
>> Hello list,
>>
>>  I really do appreciate the advice I've gotten here as I start building
>> familiarity with Cassandra. Aside from the single node instance I setup for
>> a developer friend, I've just been playing with a single node in a VM on my
>> laptop and playing around with the cassandra-cli and PHP.
>>
>> Well I've decided to setup my first cluster on my amazon ec2 account and
>> I'm running into an issue getting the nodes to gossip.
>>
>> I've set the IP's of 'node01' and 'node02' ec2 instances in their
>> respective listen_address, rpc_address and made sure that the
>> 'cluster_name' on both was in agreement.
>>
>>  I believe the problem may be in one of two places: either the seeds or
>> the initial_token setting.
>>
>> For the seeds I have it setup as such. I put the IPs for both machines in
>> the 'seeds' settings for each, thinking this would be how each node would
>> discover each other:
>>
>>  - seeds: "10.xxx.xxx.248,10.xxx.xxx.123"
>>
>> Initially I tried the tokengen script that I found in the documentation.
>> But I found that the tokens that were being generated would require way too
>> much memory for the m1.micro instances that I'm experimenting with on the
>> Amazon free tier. And according to the docs in the config it is in some
>> cases ok to leave that field blank. So that's what I did on both instances.
>>
>> Not sure how much/if this matters but I am using the setting -
>> endpoint_snitch: Ec2Snitch
>>
>> Finally, when I start up the first node all goes well.
>>
>> But when I startup the second node I see this exception on both hosts:
>>
>> node1
>>
>> INFO 11:02:32,231 Listening for thrift clients...
>>  INFO 11:02:59,262 Node /10.xxx.xxx.123 is now part of the cluster
>>  INFO 11:02:59,268 InetAddress /10.xxx.xxx.123 is now UP
>> ERROR 11:02:59,270 Exception in thread Thread[GossipStage:1,5,main]
>> java.lang.RuntimeException: Host ID collision between active endpoint
>> /10..xxx.248 and /10.xxx.xxx.123
>> (id=54ce7ccd-1b1d-418e-9861-1c281c078b8f)
>> at
>> org.apache.cassandra.locator.TokenMetadata.updateHostId(TokenMetadata.java:227)
>> at
>> org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1296)
>> at
>> org.apache.cassandra.service.StorageService.onChange(StorageService.java:1157)
>> at
>> org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1895)
>> at
>> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:805)
>> at
>> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:883)
>> at
>> org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.ja

Re: Write performance expectations...

[Ben Bromhead]

If you are using CPython (most likely) remember to use the multiprocessing
interface rather than multithreading to avoid the global interpreter lock.

Cheers

Ben

On Thu, Feb 14, 2013 at 4:35 AM,  wrote:

> I'm not using multi-threads/processes. I'll try multi-threading to see if
> I get a boost.
>
> Thanks.
>
> Ken
>
>
> --
> *From: *"Tyler Hobbs" 
> *To: *user@cassandra.apache.org
> *Sent: *Wednesday, February 13, 2013 11:06:30 AM
> *Subject: *Re: Write performance expectations...
>
>
> 2500 inserts per second is about what a single python thread using pycassa
> can do against a local node.  Are you using multiple threads for the
> inserts? Multiple processes?
>
>
> On Wed, Feb 13, 2013 at 8:21 AM, Alain RODRIGUEZ wrote:
>
>> Is there a particular reason for you to use EBS ? Instance Store
>> are recommended because they improve performances by reducing the I/O
>> throttling.
>>
>> An other thing you should be aware of is that replicating the data to all
>> node reduce your performance, it is more or less like if you had only one
>> node (at performance level I mean).
>>
>> Also, writing to different datacenters probably induce some network
>> latency.
>>
>> You should give the EC2 instance type (m1.xlarge / m1.large / ...) if you
>> want some feedback about the 2500 w/s, and also give the mean size of your
>> rows.
>>
>> Alain
>>
>>
>> 2013/2/13 
>>
>> Hello,
>>>  New member here, and I have (yet another) question on write
>>> performance.
>>>
>>> I'm using Apache Cassandra version 1.1, Python 2.7 and Pycassa 1.7.
>>>
>>> I have a cluster of 2 datacenters, each with 3 nodes, on AWS EC2 using
>>> EBS and the RandomPartioner. I'm writing to a column family in a keyspace
>>> that's replicated to all nodes in both datacenters, with a consistency
>>> level of LOCAL_QUORUM.
>>>
>>> I'm seeing write performance of around 2500 rows per second.
>>>
>>> Is this in the ballpark for this kind of configuration?
>>>
>>> Thanks in advance.
>>>
>>> Ken
>>>
>>>
>>
>
>
> --
> Tyler Hobbs
> DataStax 
>

Re: no other nodes seen on priam cluster

[Ben Bromhead]

Hi Marcelo

A few questions:

Have your added the priam java agent to cassandras JVM argurments (e.g.
-javaagent:$CASS_HOME/lib/priam-cass-extensions-1.1.15.jar)  and does the
web container running priam have permissions to write to the cassandra
config directory? Also what do the priam logs say?

If you want to get up and running quickly with cassandra, AWS and priam
quickly check out
www.instaclustr.comyou.
We deploy Cassandra under your AWS account and you have full root access to
the nodes if you want to explore and play around + there is a free tier
which is great for experimenting and trying Cassandra out.

Cheers

Ben

On Wed, Feb 27, 2013 at 6:09 AM, Marcelo Elias Del Valle  wrote:

> Hello,
>
>  I am using cassandra 1.2.1 and I am trying to set up a Priam cluster
> on AWS with two nodes. However, I can't get both nodes up and running
> because of a weird error (at least to me).
>  When I start both nodes, they are both able to connect to each other
> and do some communication. However, after some seconds, I just see
> "Java.lang.RuntimeException: No other nodes seen! ", so they disconnect and
> die. I tried to test all ports (7000, 9160 and  7199) between both nodes
> and there is no firewall. On the second node, before the above exception, I
> get a broken pipe, as shown bellow.
>   Any hint?
>
> DEBUG 18:54:31,776 attempting to connect to /10.224.238.170
> DEBUG 18:54:32,402 Reseting version for /10.224.238.170
> DEBUG 18:54:32,778 Connection version 6 from /10.224.238.170
> DEBUG 18:54:32,779 Upgrading incoming connection to be compressed
> DEBUG 18:54:32,779 Max version for /10.224.238.170 is 6
> DEBUG 18:54:32,779 Setting version 6 for /10.224.238.170
> DEBUG 18:54:32,780 set version for /10.224.238.170 to 6
> DEBUG 18:54:33,455 Disseminating load info ...
> DEBUG 18:54:59,082 Reseting version for /10.224.238.170
> DEBUG 18:55:00,405 error writing to /10.224.238.170
> java.io.IOException: Broken pipe
>  at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>  at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)
> at sun.nio.ch.IOUtil.write(IOUtil.java:43)
>  at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at java.nio.channels.Channels.writeFullyImpl(Channels.java:59)
>  at java.nio.channels.Channels.writeFully(Channels.java:81)
> at java.nio.channels.Channels.access$000(Channels.java:47)
>  at java.nio.channels.Channels$1.write(Channels.java:155)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>  at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at org.xerial.snappy.SnappyOutputStream.flush(SnappyOutputStream.java:272)
>  at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:189)
>  at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:143)
> DEBUG 18:55:01,405 attempting to connect to /10.224.238.170
> DEBUG 18:55:01,461 Started replayAllFailedBatches
> DEBUG 18:55:01,462 forceFlush requested but everything is clean in batchlog
> DEBUG 18:55:01,463 Finished replayAllFailedBatches
>  INFO 18:55:01,472 JOINING: schema complete, ready to bootstrap
> DEBUG 18:55:01,473 ... got ring + schema info
>  INFO 18:55:01,473 JOINING: getting bootstrap token
> ERROR 18:55:01,475 Exception encountered during startup
> java.lang.RuntimeException: No other nodes seen!  Unable to bootstrap.If
> you intended to start a single-node cluster, you should make sure your
> broadcast_address (or listen_address) is listed as a seed.  Otherwise, you
> need to determine why the seed being contacted has no knowledge of the rest
> of the cluster.  Usually, this can be solved by giving all nodes the same
> seed list.
>
>
> and on the first node:
>
> DEBUG 18:54:30,833 Disseminating load info ...
> DEBUG 18:54:31,532 Connection version 6 from /10.242.139.159
> DEBUG 18:54:31,533 Upgrading incoming connection to be compressed
> DEBUG 18:54:31,534 Max version for /10.242.139.159 is 6
> DEBUG 18:54:31,534 Setting version 6 for /10.242.139.159
> DEBUG 18:54:31,534 set version for /10.242.139.159 to 6
> DEBUG 18:54:31,542 Reseting version for /10.242.139.159
> DEBUG 18:54:31,791 Connection version 6 from /10.242.139.159
> DEBUG 18:54:31,792 Upgrading incoming connection to be compressed
> DEBUG 18:54:31,792 Max version for /10.242.139.159 is 6
> DEBUG 18:54:31,792 Setting version 6 for /10.242.139.159
> DEBUG 18:54:31,793 set version for /10.242.139.159 to 6
>  INFO 18:54:32,414 Node /10.242.139.159 is now part of the cluster
> DEBUG 18:54:32,415 Resetting pool for /10.242.139.159
> DEBUG 18:54:32,415 removing expire time for endpoint : /10.242.139.159
>  INFO 18:54:32,415 InetAddress /10.242.139.159 is now UP
> DEBUG 18:54:32,789 attempting to connect to
> ec2-75-101-233-115.compute-1.amazonaws.com/10.242.139.

Re: no other nodes seen on priam cluster

[Ben Bromhead]

Off the top of my head I would check to make sure the Autoscaling Group you
created is restricted to a single Availability Zone, also Priam sets the
number of EC2 instances it expects based on the maximum instance count you
set on your scaling group (it did this last time i checked a few months
ago, it's behaviour may have changed).

So I would make your desired, min and max instances for your scaling group
are all the same, make sure your ASG is restricted to a
single availability zone (e.g. us-east-1b) and then (if you are able to and
there is no data in your cluster) delete all the SimpleDB entries Priam has
created and then also possibly clear out the cassandra data directory.

Other than that I see you've raised it as an issue on the Priam project
page , so see what they say ;)

Cheers

Ben

On Thu, Feb 28, 2013 at 3:40 AM, Marcelo Elias Del Valle  wrote:

> One additional important info, I checked here and the seeds seems really
> different on each node. The command
> echo `curl 
> http://127.0.0.1:8080/Priam/REST/v1/cassconfig/get_seeds`<http://127.0.0.1:8080/Priam/REST/v1/cassconfig/get_seeds>
> returns ip2 on first node and ip1,ip1 on second node.
> Any idea why? It's probably what is causing cassandra to die, right?
>
>
> 2013/2/27 Marcelo Elias Del Valle 
>
>> Hello Ben, Thanks for the willingness to help,
>>
>> 2013/2/27 Ben Bromhead 
>>>
>>> Have your added the priam java agent to cassandras JVM argurments (e.g.
>>> -javaagent:$CASS_HOME/lib/priam-cass-extensions-1.1.15.jar)  and does
>>> the web container running priam have permissions to write to the cassandra
>>> config directory? Also what do the priam logs say?
>>>
>>
>> I put the priam log of the first node bellow. Yes, I have added
>> priam-cass-extensions to java args and Priam IS actually writting to
>> cassandra dir.
>>
>>
>>> If you want to get up and running quickly with cassandra, AWS and priam
>>> quickly check out 
>>> www.instaclustr.com<http://www.instaclustr.com/?cid=cass-list>you.
>>> We deploy Cassandra under your AWS account and you have full root access
>>> to the nodes if you want to explore and play around + there is a free tier
>>> which is great for experimenting and trying Cassandra out.
>>>
>>
>> That sounded really great. I am not sure if it would apply to our case
>> (will consider it though), but some partners would have a great benefit
>> from it, for sure! I will send your link to them.
>>
>> What priam says:
>>
>> 2013-02-27 14:14:58.0614 INFO pool-2-thread-1
>> com.netflix.priam.utils.SystemUtils Calling URL API:
>> http://169.254.169.254/latest/meta-data/public-hostname returns:
>> ec2-174-129-59-107.compute-1.amazon
>> aws.com
>> 2013-02-27 14:14:58.0615 INFO pool-2-thread-1
>> com.netflix.priam.utils.SystemUtils Calling URL API:
>> http://169.254.169.254/latest/meta-data/public-ipv4 returns:
>> 174.129.59.107
>> 2013-02-27 14:14:58.0618 INFO pool-2-thread-1
>> com.netflix.priam.utils.SystemUtils Calling URL API:
>> http://169.254.169.254/latest/meta-data/instance-id returns: i-88b32bfb
>> 2013-02-27 14:14:58.0618 INFO pool-2-thread-1
>> com.netflix.priam.utils.SystemUtils Calling URL API:
>> http://169.254.169.254/latest/meta-data/instance-type returns: c1.medium
>> 2013-02-27 14:14:59.0614 INFO pool-2-thread-1
>> com.netflix.priam.defaultimpl.PriamConfiguration REGION set to us-east-1,
>> ASG Name set to dmp_cluster-useast1b
>> 2013-02-27 14:14:59.0746 INFO pool-2-thread-1
>> com.netflix.priam.defaultimpl.PriamConfiguration appid used to fetch
>> properties is: dmp_cluster
>> 2013-02-27 14:14:59.0843 INFO pool-2-thread-1
>> org.quartz.simpl.SimpleThreadPool Job execution threads will use class
>> loader of thread: pool-2-thread-1
>> 2013-02-27 14:14:59.0861 INFO pool-2-thread-1
>> org.quartz.core.SchedulerSignalerImpl Initialized Scheduler Signaller of
>> type: class org.quartz.core.SchedulerSignalerImpl
>> 2013-02-27 14:14:59.0862 INFO pool-2-thread-1
>> org.quartz.core.QuartzScheduler Quartz Scheduler v.1.7.3 created.
>> 2013-02-27 14:14:59.0864 INFO pool-2-thread-1
>> org.quartz.simpl.RAMJobStore RAMJobStore initialized.
>> 2013-02-27 14:14:59.0864 INFO pool-2-thread-1
>> org.quartz.impl.StdSchedulerFactory Quartz scheduler
>> 'DefaultQuartzScheduler' initialized from default resource file in Quartz
>> package: 'quartz.propertie
>> s'
>> 2013-02-27 14:14:59.0864 INFO pool-2-thread-1
>> org.quartz.impl.StdSchedulerFactory Quartz scheduler version: 1.7.3
>> 2013-02-27 14:14:59.0864 INFO p

Re: no other nodes seen on priam cluster

[Ben Bromhead]

Glad you got it going!

There is a REST call you can make to priam telling it to double the cluster 
size (/v1/cassconfig/double_ring), it will pre fill all SimpleDB entries for 
when the nodes come online, you then change the number of nodes on the 
autoscale group. Now that Priam supports C* 1.2 with Vnodes, increasing the 
cluster size in an ad-hoc manner might be just around the corner.

Instacluster has some predefined cluster sizes (Free, Basic, Professional and 
Enterprise), these are loosely based on the estimated performance and storage 
capacity. 

You can also create a custom cluster where you define the number of nodes 
(minimum of 4) and the Instance type according to your requirements. For 
pricing on those check out https://www.instaclustr.com/pricing/per-instance, we 
base our pricing on estimated support and throughput requirements.

Cheers

Ben
Instaclustr | www.instaclustr.com | @instaclustr



On 02/03/2013, at 3:59 AM, Marcelo Elias Del Valle  wrote:

> Thanks a lot Ben, actually I managed to make it work erasing the SimpleDB 
> Priam uses to keeps instances... I would pulled the last commit from the 
> repo, not sure if it helped or not.
> 
> But you message made me curious about something...  How do you do to add more 
> Cassandra nodes on the fly? Just update the autoscale properties? I saw 
> instaclustr.com changes the instance type as the number of nodes increase 
> (not sure why the price also becomes higher per instance in this case), I am 
> guessing priam use the data backed up to S3 to restore a node data in another 
> instance, right?
> 
> []s
> 
> 
> 
> 2013/2/28 Ben Bromhead 
> Off the top of my head I would check to make sure the Autoscaling Group you 
> created is restricted to a single Availability Zone, also Priam sets the 
> number of EC2 instances it expects based on the maximum instance count you 
> set on your scaling group (it did this last time i checked a few months ago, 
> it's behaviour may have changed). 
> 
> So I would make your desired, min and max instances for your scaling group 
> are all the same, make sure your ASG is restricted to a single availability 
> zone (e.g. us-east-1b) and then (if you are able to and there is no data in 
> your cluster) delete all the SimpleDB entries Priam has created and then also 
> possibly clear out the cassandra data directory. 
> 
> Other than that I see you've raised it as an issue on the Priam project page 
> , so see what they say ;)
> 
> Cheers
> 
> Ben
> 
> On Thu, Feb 28, 2013 at 3:40 AM, Marcelo Elias Del Valle  
> wrote:
> One additional important info, I checked here and the seeds seems really 
> different on each node. The command
> echo `curl http://127.0.0.1:8080/Priam/REST/v1/cassconfig/get_seeds`
> returns ip2 on first node and ip1,ip1 on second node.
> Any idea why? It's probably what is causing cassandra to die, right?
> 
> 
> 2013/2/27 Marcelo Elias Del Valle 
> Hello Ben, Thanks for the willingness to help, 
> 
> 2013/2/27 Ben Bromhead 
> Have your added the priam java agent to cassandras JVM argurments (e.g. 
> -javaagent:$CASS_HOME/lib/priam-cass-extensions-1.1.15.jar)  and does the web 
> container running priam have permissions to write to the cassandra config 
> directory? Also what do the priam logs say?
> 
> I put the priam log of the first node bellow. Yes, I have added 
> priam-cass-extensions to java args and Priam IS actually writting to 
> cassandra dir.
>  
> If you want to get up and running quickly with cassandra, AWS and priam 
> quickly check out www.instaclustr.com you. 
> We deploy Cassandra under your AWS account and you have full root access to 
> the nodes if you want to explore and play around + there is a free tier which 
> is great for experimenting and trying Cassandra out.
> 
> That sounded really great. I am not sure if it would apply to our case (will 
> consider it though), but some partners would have a great benefit from it, 
> for sure! I will send your link to them.
> 
> What priam says:
> 
> 2013-02-27 14:14:58.0614 INFO pool-2-thread-1 
> com.netflix.priam.utils.SystemUtils Calling URL API: 
> http://169.254.169.254/latest/meta-data/public-hostname returns: 
> ec2-174-129-59-107.compute-1.amazon
> aws.com
> 2013-02-27 14:14:58.0615 INFO pool-2-thread-1 
> com.netflix.priam.utils.SystemUtils Calling URL API: 
> http://169.254.169.254/latest/meta-data/public-ipv4 returns: 174.129.59.107
> 2013-02-27 14:14:58.0618 INFO pool-2-thread-1 
> com.netflix.priam.utils.SystemUtils Calling URL API: 
> http://169.254.169.254/latest/meta-data/instance-id returns: i-88b32bfb
> 2013-02-27 14:14:58.0618 INFO pool-2-thread-1 
> com.netflix.priam.utils.SystemUtils Calling URL API: 
> http://169.254.169.25

Re: Cassandra instead of memcached

[Ben Bromhead]

Check out 
http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html

Netflix used Cassandra with SSDs and were able to drop their memcache layer. 
Mind you they were not using it purely as an in memory KV store.

Ben
Instaclustr | www.instaclustr.com | @instaclustr



On 05/03/2013, at 4:33 PM, Drew Kutcharian  wrote:

> Hi Guys,
> 
> I'm thinking about using Cassandra as an in-memory key/value store instead of 
> memcached for a new project (just to get rid of a dependency if possible). I 
> was thinking about setting the replication factor to 1, enabling off-heap 
> row-cache and setting gc_grace_period to zero for the CF that will be used 
> for the key/value store.
> 
> Has anyone tried this? Any comments?
> 
> Thanks,
> 
> Drew
> 
> 

Re: [Question] Automated cluster cleanup

[Ben Bromhead]

Hi Luigi

Under the hood, nodetool is actually just a command line wrapper around
certain JMX calls. If you are looking to automate some more of commonplace
nodetool actions, have a look at the nodetool source and it will show
exactly what JMX calls (and parameters) are being passed.

One thing to keep in mind with JMX, is it does allow a remote user to do
some scary things to Cassandra and it has included remote code execution
vulns. So ensure you lock down JMX thoroughly (password/username auth,
certification auth, fw rules etc).

For the other most common management, repairs, check out Cassandra reaper
https://github.com/thelastpickle/cassandra-reaper.

Ben

On Thu, 27 Apr 2017 at 16:37 Luigi Tagliamonte  wrote:

> Hello Cassandra users,
> my cluster is getting bigger and I was looking into automating some
> tedious operations like the node cleanup after adding a new node to the
> cluster.
>
> I gave a quick search and I didn't find any good available option, so I
> decided to look into the JMX interface (In the storage service, I found the
> method: forceKeyspaceCleanup that seems a good candidate), before going
> hardcore with SSH+nodetool sessions.
>
> I was wondering if somebody here wants to share his experiences about this
> task, and what do you think about JMX approach instead of the SSH one.
>
> Thank you.
>
> --
> Luigi
> ---
> “The only way to get smarter is by playing a smarter opponent.”
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Reg:- Apache Solr with DSE Query

[Ben Bromhead]

Hi Nandan

I would suggest referring your question to Datastax, otherwise you could
look at some open source alternatives such as:

https://github.com/strapdata/elassandra
https://github.com/Stratio/cassandra-lucene-index

Cheers

Ben

On Thu, 11 May 2017 at 03:06 @Nandan@ 
wrote:

> Hi,
>
> Details are as below:-
> 1) Have table:- video_info
> 2) PRIMARY KEY:- video_id UUID
> 3) having records around 5.
> 4) Table is having around 30-35 columns
> 5) Using DSE 4.8
>
> Need clarifications and suggestions:-
> 1) I need to search by few 3-4 columns like Video_title, video_actor etc..
> 2) If I am implementing Solr indexing on this single table, then we can
> able to do a query from other columns and much more.. but is it going to
> effect my READ and WRITE speed.
> 3) is it will be a good idea or not to implement SOLR directly.
>
> Please suggest on above.
> Thanks.
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Cassandra 2.1.13: Using JOIN_RING=False

[Ben Bromhead]

A rough guess on the best way to do this with cql, without looking too
deeply into it:

You would likely have to implement a custom load balancing policy within
the driver that only uses coordinator nodes and/or modify
refreshNodeListAndTokenMap() behaviour in ControlConnection.java (there
should also be an equivalent class in the python driver).

On Thu, 11 May 2017 at 08:19 Dikang Gu  wrote:

> 1. The coordinator still store system data and hints, but they should not
> store any user data since they are not part of ring.
> 2. We are using coordinator for thrift client. For cql based drivers, they
> needs to talk to nodes in the ring, so I think coordinator mode won't work
> for them.
>
> -Dikang
>
> On Tue, May 9, 2017 at 2:01 PM, Anubhav Kale <
> anubhav.k...@microsoft.com.invalid> wrote:
>
>> Hello,
>>
>>
>>
>> With some inspiration from the Cassandra Summit talk from last year, we
>> are trying to setup a cluster with coordinator-only nodes. We setup
>> join_ring=false in env.sh, disabled auth in YAML and the nodes are able to
>> start just fine. However, we’re running into a few problems
>>
>>
>>
>> 1] The nodes marked with join_ring=false continue to store data. Why ?
>>
>> 2] We tried Python driver’s whitelistedpolicy. But we notice message like
>> below, so we are not able to send queries to all nodes marked as
>> coordinators. We also changed the Scala driver to support whitelisting, but
>> see the same thing. What are we missing ?
>>
>> 3] Is there any way to concretely tell that only coordinator nodes are
>> getting requests from clients ? We don’t have OpsCenter.
>>
>>
>>
>> Thanks !
>>
>>
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Removing host not found in peers metadata: 
>>
>> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
>> 10.80.10.128 removed
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
>> 10.80.10.128
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Removing host not found in peers metadata: 
>>
>> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
>> 10.80.10.127 removed
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
>> 10.80.10.127
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Removing host not found in peers metadata: 
>>
>> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
>> 10.80.10.129 removed
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
>> 10.80.10.129
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Finished fetching ring info
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Rebuilding token map due to topology changes
>>
>> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user functions table
>> not found
>>
>> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user aggregates table
>> not found
>>
>> 2017-05-09 20:45:25,098 [DEBUG] cassandra.cluster: Control connection
>> created
>>
>> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection
>> for host 10.80.10.125
>>
>> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection
>> for host 10.80.10.126
>>
>>
>>
>
>
>
> --
> Dikang
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: DSE 5.0 Upgrade

[Ben Bromhead]

Hi Cass Savy

I would suggest directing your question to Datastax, or you could look at
an open source alternative such as Apache Cassandra :)

Ben

On Wed, 10 May 2017 at 22:50 cass savy  wrote:

> Team,
>
> 1. What is the stable version for DSE 5.0 to upgrade from DSE 4.8.x?
>
> 2. Is anybody switched to using DSE Unified auth model which enfores to
> use one auth policy as primary and other secondary?
>
> 3. Do I need to use multi-auth/DSE Unified auth for me upgrade to DSE 5.0
> or higher?
> Our old clusters are using internal authentication and we learnt that
> we have to go with DSE Unifiedauth model as part of upgrade?
>
> 4. Does anybody migrate from Java driver to Datastax enterprise Java
> driver 1.1 or 1.2 recently which will be only driver that will support DSE
> 5.0 and above and new auth policies added by DSE?
>
> Or
>
> Has anybody made Java driver version 3.2 work with DSE 5.0 or 5.1?
>
> 4. For AWS, what is prod recommended AMI with CentOS and for DSE 5.x
> versions?
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Answering the question of Cassandra Summit 2017

[Ben Bromhead]

Thanks for explanation Patrick, I think it helps the community to hear
direct from the horses mouth.

I also wanted to thank you, your team and Datastax for being the organising
force behind the Cassandra Summit for all these years. Politics aside it
has been an awesome place for the community to meet, bond and make some
great (and sometimes hazy) memories!

In terms of the future, I'm pretty positive there will be a place and
format for the Apache Cassandra community to meet in person in the new year
and the fact that Datastax would be an enthusiastic participant is
wonderful.

<3 The Instaclustr team

On Fri, 11 Aug 2017 at 17:27 Patrick McFadin  wrote:

> Hello Cassandra Community,
>
> I know this is a hot topic so let me TL;DR for those of you looking for
> the short answer. Will there be a Cassandra Summit in 2017? DataStax will
> not be holding a Cassandra Summit in 2017, but instead multiple DataStax
> Summits in 2018.
>
> More details. Last year was pretty chaotic for the Cassandra community and
> where DataStax fit with the project. I don’t need to re-cap all the drama.
> You can go look at the dev and user lists around this time last year if you
> want to re-live it. It’s safe to say 2016 is a year I wouldn’t want to do
> again for a lot of reasons. Those of us at the Cassandra Summit 2016 knew
> it was the end of something, now the question is what’s next and what will
> it be?
>
> Having a place for people together and talk about what we as a community
> do is really important. Those of you that know me, know how much I live
> that. When we started talking summit inside DataStax, we realized it would
> be a hot button issue. When I started talking to people in the community,
> it was even more of a hot button issue. Having DataStax run the Cassandra
> Summit was going to cause a lot of heartache and would further divide the
> community with questions and accusations. It’s just much easier to hold a
> DataStax Summit so we are just out there plainly and move forward.
>
> What DataStax will be doing different.
>
> We will be moving back to a more regional format instead of the big bang
> single event in San Jose starting in 2018. Fun fact. Almost 80% of
> attendees of Cassandra Summit were from the Bay Area. That means we have
> developers and operators from a lot of other places being excluded which
> isn’t cool. We will also be inviting talks from the Cassandra Community.
> You don’t have to be a DataStax customer or partner to get on the speaking
> list.
>
> If there is some new group or company that launches a Cassandra Summit,
> DataStax will happily be a sponsor. There are some for-profit, professional
> conference companies like the Linux Foundation out there that just may and
> if so, I’ll see you there. After being involved in making the Cassandra
> Summit happen for years, I can say it’s no small effort.
>
> There it is. Fire away with your questions, comments. All I ask is keep it
> respectful because this is a community of amazing people. You have changed
> the world over these years and I know it won’t stop. You know I got a hug
> for you wherever we just happen to meet.
>
> Patrick
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Full table scan with cassandra

[Ben Bromhead]

Apache Cassandra is not great in terms of performance at the moment for
batch analytics workloads that require a full table scan. I would look at
FiloDB for all the benefits and familiarity of Cassandra with better
streaming and analytics performance: https://github.com/filodb/FiloDB

There are also some outstanding tickets around improving bulk reads in
Cassandra (see https://issues.apache.org/jira/browse/CASSANDRA-9259 for the
full gory details), but it appears to be abandonded by the initial set of
contributors.

On Wed, 16 Aug 2017 at 09:51 Alex Kotelnikov 
wrote:

> Hey,
>
> we are trying Cassandra as an alternative for storage huge stream of data
> coming from our customers.
>
> Storing works quite fine, and I started to validate how retrieval does. We
> have two types of that: fetching specific records and bulk retrieval for
> general analysis.
> Fetching single record works like charm. But it is not so with bulk fetch.
>
> With a moderately small table of ~2 million records, ~10Gb raw data I
> observed very slow operation (using token(partition key) ranges). It takes
> minutes to perform full retrieval. We tried a couple of configurations
> using virtual machines, real hardware and overall looks like it is not
> possible to all table data in a reasonable time (by reasonable I mean that
> since we have 1Gbit network 10Gb can be transferred in a couple of minutes
> from one server to another and when we have 10+ cassandra servers and 10+
> spark executors total time should be even smaller).
>
> I tried datastax spark connector. Also I wrote a simple test case using
> datastax java driver and see how fetch of 10k records takes ~10s so I
> assume that "sequential" scan will take 200x more time, equals ~30 minutes.
>
> May be we are totally wrong trying to use Cassandra this way?
>
> --
>
> Best Regards,
>
>
> *Alexander Kotelnikov*
>
> *Team Lead*
>
> DIGINETICA
> Retail Technology Company
>
> m: +7.921.915.06.28 <+7%20921%20915-06-28>
>
> *www.diginetica.com <http://www.diginetica.com/>*
>
-- 
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: C* 3 node issue -Urgent

[Ben Bromhead]

Just to clarify that behaviour. QUORUM only applies to the default
superuser, subsequent superusers you create later on are still only queried
at LOCAL_ONE. E.g.

protected static ConsistencyLevel consistencyForRole(String role)
{
if (role.equals(DEFAULT_SUPERUSER_NAME))
return ConsistencyLevel.QUORUM;
else
return ConsistencyLevel.LOCAL_ONE;
}


Despite the fact it suggests consistency for a given role... the function
actually gets passed the username not the role (role lookup happens after
authentication iirc).

Best practices would suggest you change the default superuser password to
some long random password and throw it away and use other superuser
accounts. The Cassandra user is only their to bootstrap auth and nothing
else.

If your RF for the system auth table is very high it will not make it
difficult to login, just to change your password :)




On Wed, 6 Sep 2017 at 11:43 Jeff Jirsa  wrote:

> More explicitly - if you have 60 nodes, setting rf=60 will likely make it
> very difficult for you to log in as a superuser.
>
> --
> Jeff Jirsa
>
>
> > On Sep 6, 2017, at 11:40 AM, Jon Haddad 
> wrote:
> >
> > I wouldn’t worry about being meticulous about keeping RF = N as the
> cluster grows.  If you had 60 nodes and your auth data was only on 9 you’d
> be completely fine.
> >
> >> On Sep 6, 2017, at 11:36 AM, Cogumelos Maravilha <
> cogumelosmaravi...@sapo.pt> wrote:
> >>
> >> After insert a new node we should:
> >>
> >> ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' : ...
> >> 'replication_factor' : x };
> >>
> >> x = number of nodes in dc
> >>
> >> The default user and password should work:
> >> -u cassandra -p cassandra
> >>
> >> Cheers.
> >>
> >>> On 23-08-2017 11:14, kurt greaves wrote:
> >>> The cassandra user requires QUORUM consistency to be achieved for
> >>> authentication. Normal users only require ONE. I suspect your
> >>> system_auth keyspace has an RF of 1, and the node that owns the
> >>> cassandra users data is down.
> >>>
> >>> Steps to recover:
> >>> 1. Turn off authentication on all the nodes
> >>> 2. Restart the nodes and make sure they are UN
> >>> 3. Alter system_auth to have a higher RF than 1 (3 is probably
> >>> appropriate)
> >>> 4. Turn auth back on and restart
> >>> 5. Create a new user and use that from now on.
> >>>
> >>> ​
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>
> >
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: user-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Amazon Time Sync Service + ntpd vs chrony

[Ben Bromhead]

I wouldn't 100% rely on your clients to generate timestamps (actually don't
100% rely on timestamps at all!) .

Clients tend to be stateless, scaled up and down,  stopped, started, ntp
takes time to skew a clock and they are more likely to be moved between
hypervisor's in cloud environments etc. All these combine to give you more
scenarios where clients have unreliable clocks that are not roughly in sync
with each other.

By far and large the worst time related bugs I've experienced are due to
Cassandra clients having the wrong timestamp set for it's writetime.

Of course this depends on what you are prioritising... relative accuracy of
any given writetime on one row to any other given row or just respecting
what the client thinks is right.


On Thu, Mar 8, 2018, 21:24 Jeff Jirsa  wrote:

> Clients can race (and go backward), so the more computer answer tends to
> be to use LWT/CAS to guarantee state if you have a data model where it
> matters.
>
>
> --
> Jeff Jirsa
>
>
> On Mar 8, 2018, at 6:18 PM, Dor Laor  wrote:
>
> While NTP on the servers is important, make sure that you use client
> timestamps and
> not server. Since the last write wins, the data generator should be the
> one setting its timestamp.
>
> On Thu, Mar 8, 2018 at 2:12 PM, Ben Slater 
> wrote:
>
>> It is important to make sure you are using the same NTP servers across
>> your cluster - we used to see relatively frequent NTP issues across our
>> fleet using default/public NTP servers until (back in 2015) we implemented
>> our own NTP pool (see
>> https://www.instaclustr.com/apache-cassandra-synchronization/ which
>> references some really good and detailed posts from logentries.com on
>> the potential issues).
>>
>> Cheers
>> Ben
>>
>> On Fri, 9 Mar 2018 at 02:07 Michael Shuler 
>> wrote:
>>
>>> As long as your nodes are syncing time using the same method, that
>>> should be good. Don't mix daemons, however, since they may sync from
>>> different sources. Whether you use ntpd, openntp, ntpsec, chrony isn't
>>> really important, since they are all just background daemons to sync the
>>> system clock. There is nothing Cassandra-specific.
>>>
>>> --
>>> Kind regards,
>>> Michael
>>>
>>> On 03/08/2018 04:15 AM, Kyrylo Lebediev wrote:
>>> > Hi!
>>> >
>>> > Recently Amazon announced launch of Amazon Time Sync Service
>>> > (
>>> https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/
>>> )
>>> > and now it's AWS-recommended way for time sync on EC2 instances
>>> > (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html).
>>> > It's stated there that chrony is faster / more precise than ntpd.
>>> >
>>> > Nothing to say correct time sync configuration is very important for
>>> any
>>> > C* setup.
>>> >
>>> > Does anybody have positive experience using crony, Amazon Time Sync
>>> > Service with Cassandra and/or combination of them?
>>> > Any concerns regarding chrony + Amazon Time Sync Service + Cassandra?
>>> > Are there any chrony best-practices/custom settings for C* setups?
>>> >
>>> > Thanks,
>>> > Kyrill
>>> >
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>> --
>>
>>
>> *Ben Slater*
>>
>> *Chief Product Officer <https://www.instaclustr.com/>*
>>
>> <https://www.facebook.com/instaclustr>
>> <https://twitter.com/instaclustr>
>> <https://www.linkedin.com/company/instaclustr>
>>
>> Read our latest technical blog posts here
>> <https://www.instaclustr.com/blog/>.
>>
>> This email has been sent on behalf of Instaclustr Pty. Limited
>> (Australia) and Instaclustr Inc (USA).
>>
>> This email and any attachments may contain confidential and legally
>> privileged information.  If you are not the intended recipient, do not copy
>> or disclose its content, but please reply to this email immediately and
>> highlight the error to the sender and then immediately delete the message.
>>
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Reliability at Scale
Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer

Re: stalled nodetool repair?

[Ben Bromhead]

https://github.com/mstump/cassandra_range_repair

Also very useful. 

Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359




On 22/08/2014, at 6:12 AM, Robert Coli  wrote:

> On Thu, Aug 21, 2014 at 12:32 PM, Kevin Burton  wrote:
> How do I watch the progress of nodetool repair.
> 
> This is a very longstanding operational problem in Cassandra. Repair barely 
> works and is opaque, yet one is expected to run it once a week in the default 
> configuration.
> 
> An unreasonably-hostile-in-tone-but-otherwise-accurate description of the 
> status quo before the re-write of streaming in 2.0 :
> 
> https://issues.apache.org/jira/browse/CASSANDRA-5396
> 
> A proposal to change the default for gc_grace_seconds to 34 days, so that 
> this fragile and heavyweight operation only has to be done once a month :
> 
> https://issues.apache.org/jira/browse/CASSANDRA-5850
>  
> granted , this is a lot of data, but it would be nice to at least see some 
> progress.
> 
> Here's the rewrite of streaming, where progress indication improves 
> dramatically over the prior status quo :
> 
> https://issues.apache.org/jira/browse/CASSANDRA-5286
> 
> And here's two open tickets on making repair less opaque (thx 
> yukim@#cassandra) :
> 
> https://issues.apache.org/jira/browse/CASSANDRA-5483
> https://issues.apache.org/jira/browse/CASSANDRA-5839
> 
> =Rob
> 
> 

Re: stalled nodetool repair?

[Ben Bromhead]

Ah sorry that is the original repo, see 
https://github.com/BrianGallew/cassandra_range_repair for the updated version 
of the script with vnode support 

Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359

On 22 Aug 2014, at 2:19 pm, DuyHai Doan  wrote:

> Thanks Ben for the link. Still this script does not work with vnodes, which 
> exclude a wide range of C* config
> 
> 
> On Thu, Aug 21, 2014 at 5:51 PM, Ben Bromhead  wrote:
> https://github.com/mstump/cassandra_range_repair
> 
> Also very useful. 
> 
> Ben Bromhead
> Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359
> 
> 
> 
> 
> On 22/08/2014, at 6:12 AM, Robert Coli  wrote:
> 
>> On Thu, Aug 21, 2014 at 12:32 PM, Kevin Burton  wrote:
>> How do I watch the progress of nodetool repair.
>> 
>> This is a very longstanding operational problem in Cassandra. Repair barely 
>> works and is opaque, yet one is expected to run it once a week in the 
>> default configuration.
>> 
>> An unreasonably-hostile-in-tone-but-otherwise-accurate description of the 
>> status quo before the re-write of streaming in 2.0 :
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-5396
>> 
>> A proposal to change the default for gc_grace_seconds to 34 days, so that 
>> this fragile and heavyweight operation only has to be done once a month :
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-5850
>>  
>> granted , this is a lot of data, but it would be nice to at least see some 
>> progress.
>> 
>> Here's the rewrite of streaming, where progress indication improves 
>> dramatically over the prior status quo :
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-5286
>> 
>> And here's two open tickets on making repair less opaque (thx 
>> yukim@#cassandra) :
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-5483
>> https://issues.apache.org/jira/browse/CASSANDRA-5839
>> 
>> =Rob
>> 
>> 
> 
> 

Re: Can't Add AWS Node due to /mnt/cassandra/data directory

[Ben Bromhead]

Make sure you have also setup the ephemeral drives as a raid device (use mdadm) 
and mounted it under /mnt/cassandra otherwise your data dir is the os partition 
which is usually very small.

Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359

On 27 Aug 2014, at 8:21 pm, Stephen Portanova  wrote:

> Worked great! Thanks Mark!
> 
> 
> On Wed, Aug 27, 2014 at 2:00 AM, Mark Reddy  wrote:
> Hi stephen,
> 
> I have never added a node via OpsCenter, so this may be a short coming of 
> that process. However in non OpsCenter installs you would have to create the 
> data directories first:
> 
> sudo mkdir -p /mnt/cassandra/commitlog
> sudo mkdir -p /mnt/cassandra/data
> sudo mkdir -p /mnt/cassandra/saved_caches
> 
> And then give the cassandra user ownership of those directories:
> 
> sudo chown -R "cassandra:cassandra" /mnt/cassandra 
> 
> Once this is done Cassandra will have the correct directories and permission 
> to start up.
> 
> 
> Mark
> 
> 
> On 27 August 2014 09:50, Stephen Portanova  wrote:
> I already have a 3node m3.large DSE cluster, but I can't seem to add another 
> m3.large node. I'm using the ubuntu-trusty-14.04-amd64-server-20140607.1 
> (ami-a7fdfee2) AMI (instance-store backed, PV) on AWS, I install java 7 and 
> the JNA, then I go into opscenter to add a node. Things look good for 3 or 4 
> green circles, until I either get this error: Start Errored: Timed out 
> waiting for Cassandra to start. or this error: Agent Connection Errored: 
> Timed out waiting for agent to connect.
> 
> I check the system.log and output.log, and they both say:
> INFO [main] 2014-08-27 08:17:24,642 CLibrary.java (line 121) JNA mlockall 
> successful
> ERROR [main] 2014-08-27 08:17:24,644 CassandraDaemon.java (line 235) 
> Directory /mnt/cassandra/data doesn't exist
> ERROR [main] 2014-08-27 08:17:24,645 CassandraDaemon.java (line 239) Has no 
> permission to create /mnt/cassandra/data directory
>  INFO [Thread-1] 2014-08-27 08:17:24,646 DseDaemon.java (line 477) DSE 
> shutting down...
> ERROR [Thread-1] 2014-08-27 08:17:24,725 CassandraDaemon.java (line 199) 
> Exception in thread Thread[Thread-1,5,main]
> java.lang.AssertionError
> at 
> org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1263)
> at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:171)
> at com.datastax.bdp.server.DseDaemon.stop(DseDaemon.java:478)
> at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:384)
> 
> My agent.log file says:
> Node is still provisioning, not attempting to determine ip.
> 
>  INFO [Initialization] 2014-08-27 08:40:57,848 Sleeping for 20s before trying 
> to determine IP over JMX again
> 
>  INFO [Initialization] 2014-08-27 08:41:17,849 Node is still provisioning, 
> not attempting to determine ip.
> 
>  INFO [Initialization] 2014-08-27 08:41:17,849 Sleeping for 20s before trying 
> to determine IP over JMX again
> 
>  INFO [Initialization] 2014-08-27 08:41:37,849 Node is still provisioning, 
> not attempting to determine ip.
> 
>  INFO [Initialization] 2014-08-27 08:41:37,850 Sleeping for 20s before trying 
> to determine IP over JMX again
> 
>  INFO [Initialization] 2014-08-27 08:41:57,850 Node is still provisioning, 
> not attempting to determine ip.
> 
> 
> 
> I feel like I'm missing something easy with the mount, so if you could point 
> me in the right direction, I would really appreciate it!
> 
> -- 
> Stephen Portanova
> (480) 495-2634
> 
> 
> 
> 
> -- 
> Stephen Portanova
> (480) 495-2634

Re: Heterogenous cluster and vnodes

[Ben Bromhead]

> Hey,
> 
> I have a few of VM host (bare metal) machines with varying amounts of free 
> hard drive space on them. For simplicity let’s say I have three machine like 
> so:
>  * Machine 1
>   - Harddrive 1: 150 GB available.
>  * Machine 2:
>   - Harddrive 1: 150 GB available.
>   - Harddrive 2: 150 GB available.
>  * Machine 3.
>   - Harddrive 1: 150 GB available.
> 
> I am setting up a Cassandra cluster between them and as I see it I have two 
> options:
> 
> 1. I set up one Cassandra node/VM per bare metal machine. I assign all free 
> hard drive space to each Cassandra node and I balance the cluster using 
> vnodes proportionally to the amount of free hard drive space (CPU/RAM is not 
> going to be a bottle neck here).
> 
> 2. I set up four VMs, each running a Cassandra node with equal amount of hard 
> drive space and equal amount of vnodes. Machine 2 runs two VMs.

This setup will potentially create a situation where if Machine 2 goes down you 
may lose two replicas. As the two VMs on Machine 2 might be replicas for the 
same key.

> 
> General question: Is any of these preferable to the other? I understand 1) 
> yields lower high-availability (since nodes are on the same hardware).

Other way around (2 would be potentially lower availability)… Cassandra thinks 
two of the vm's are separate when they in fact rely on the same underlying 
machine.

> 
> Question about alternative 1: With varying vnodes, can I always be sure that 
> replicas are never put on the same virtual machine?

Yes… mostly https://issues.apache.org/jira/browse/CASSANDRA-4123

> Or is varying vnodes really only useful/recommended when migrating from 
> machines with varying hardware (like mentioned in [1])?

Changing the number of vnodes changes the portion of the ring a node is 
responsible for. You can use it to account for different types of hardware, you 
can also use it for creating awesome situations like hotspots if you aren't 
careful… ymmv.

At the end of the day I would throw out the extra hard drive / not use it / put 
more hard drives in the other machines. Why? Hard drives are cheap and your 
time as an admin for the cluster isn't. If you do add more hard drives you can 
also split out the commit log etc onto different disks.

I would take less problems over trying to draw every last scrap of performance 
out of the available hardware any day of the year. 


Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359

Re: Question about EC2 and SSDs

[Ben Bromhead]

On 5 Sep 2014, at 10:05 am, Steve Robenalt  wrote:

> We are migrating a small cluster on AWS from instances based on spinning 
> disks (using instance store) to SSD-backed instances and we're trying to pick 
> the proper instance type. Some of the recommendations for spinning disks say 
> to use different drives for log vs data partitions to avoid issues with seek 
> delays and contention for the disk heads. Since SSDs don't have the same seek 
> delays, is it still recommended to use 2 SSD drives? Or is one sufficient?

As a side note, splitting the commit log and data dirs into different volumes 
doesn’t do a whole lot of good on AWS irrespective of whether you are on 
spinning disks or SSDs. Simply because the volumes presented to the vm may be 
on the same disk. 

Just raid the available volumes and be done with it.

Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359

Re: Moving Cassandra from EC2 Classic into VPC

[Ben Bromhead]

On 8 Sep 2014, at 12:34 pm, Oleg Dulin  wrote:

> Another idea I had was taking the ec2-snitch configuration and converting it 
> into a Property file snitch. But I still don't understand how to perform this 
> move since I need my newly created VPC instances to have public IPs -- 
> something I would like to avoid.

Off the top of my head something like this might work if you want a no downtime 
approach:

Use the gossiping property file snitch in the VPC data centre. 

Use a public elastic ip for each node.

Have the instances in the VPC join your existing cluster.

Decommission old cluster.

Change the advertised endpoint addresses afterwards to the private addresses 
for nodes in the VPC using the following:
https://engineering.eventbrite.com/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/

Once that is done, remove the elastic IPs from the instances.

Re: Moving Cassandra from EC2 Classic into VPC

[Ben Bromhead]

On 9 Sep 2014, at 7:33 am, Nate McCall  wrote:

> Other thoughts:
> - Go slowly and verify that clients and gossip are talking to the new nodes 
> after each lift and shift
> - Don't forget to change seeds afterwards
> - This is not the time to upgrade/change *anything* else - match the version 
> exactly even if its older

This +1000

take your time, wait for everything to settle and you will save yourself a lot 
of pain

Re: no change observed in read latency after switching from EBS to SSD storage

[Ben Bromhead]

EBS vs local SSD in terms of latency you are using ms as your unit of
measurement.
If your query runs for 10s you will not notice anything. What is a few less
ms for the life of a 10 second query.

To reiterate what Rob said. The query is probably slow because of your use
case / data model, not the underlying disk.



On 17 September 2014 14:21, Tony Anecito  wrote:

> If you cached your tables or the database you may not see any difference
> at all.
>
> Regards,
> -Tony
>
>
>   On Tuesday, September 16, 2014 6:36 PM, Mohammed Guller <
> moham...@glassbeam.com> wrote:
>
>
> Hi -
>
> We are running Cassandra 2.0.5 on AWS on m3.large instances. These
> instances were using EBS for storage (I know it is not recommended). We
> replaced the EBS storage with SSDs. However, we didn't see any change in
> read latency. A query that took 10 seconds when data was stored on EBS
> still takes 10 seconds even after we moved the data directory to SSD. It is
> a large query returning 200,000 CQL rows from a single partition. We are
> reading 3 columns from each row and the combined data in these three
> columns for each row is around 100 bytes. In other words, the raw data
> returned by the query is approximately 20MB.
>
> I was expecting at least 5-10 times reduction in read latency going from
> EBS to SSD, so I am puzzled why we are not seeing any change in performance.
>
> Does anyone have insight as to why we don't see any performance impact on
> the reads going from EBS to SSD?
>
> Thanks,
> Mohammed
>
>
>


-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: Repair taking long time

[Ben Bromhead]

use https://github.com/BrianGallew/cassandra_range_repair



On 30 September 2014 05:24, Ken Hancock  wrote:

>
> On Mon, Sep 29, 2014 at 2:29 PM, Robert Coli  wrote:
>
>>
>> As an aside, you "just lose" with vnodes and clusters of the size. I
>> presume you plan to grow over appx 9 nodes per DC, in which case you
>> probably do want vnodes enabled.
>>
>
> I typically only see discussion on vnodes vs. non-vnodes, but it seems to
> me that might be more important to discuss the number of vnodes per node.
> A small cluster having 256 vnodes/node is unwise given some of the
> sequential operations that are still done.  Even if operations were done in
> parallel, having a 256x increase in parallelization seems an equally bad
> choice.
>
> I've never seen any discussion on how many vnodes per node might be an
> appropriate answer based a planned cluster size -- does such a thing exist?
>
> Ken
>
>
>
>
>


-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: best practice for waiting for schema changes to propagate

[Ben Bromhead]

The system.peers table which is a copy of some gossip info the node has
stored, including the schema version. You should query this and wait until
all schema versions have converged.

http://www.datastax.com/documentation/cql/3.0/cql/cql_using/use_sys_tab_cluster_t.html

http://www.datastax.com/dev/blog/the-data-dictionary-in-cassandra-1-2

As ensuring that the driver keeps talking to the node you made the schema
change on I would ask the drivers specific mailing list / IRC:


   - MAILING LIST:
   https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user
   - IRC: #datastax-drivers on irc.freenode.net <http://freenode.net/>



On 30 September 2014 10:16, Clint Kelly  wrote:

> Hi all,
>
> I often have problems with code that I write that uses the DataStax Java
> driver to create / modify a keyspace or table and then soon after reads the
> metadata for the keyspace to verify that whatever changes I made the
> keyspace or table are complete.
>
> As an example, I may create a table called `myTableName` and then very
> soon after do something like:
>
> assert(session
>   .getCluster()
>   .getMetaData()
>   .getKeyspace(myKeyspaceName)
>   .getTable(myTableName) != null)
>
> I assume this fails sometimes because the default round-robin load
> balancing policy for the Java driver will send my create-table request to
> one node and the metadata read to another, and because it takes some time
> for the table creation to propagate across all of the nodes in my cluster.
>
> What is the best way to deal with this problem?  Is there a standard way
> to wait for schema changes to propagate?
>
> Best regards,
> Clint
>



-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: Experience with multihoming cassandra?

[Ben Bromhead]

I'm guessing your talking about multi-homing because you want to have
multiple tenants (different apps/ teams etc) to make better use of
resources ?

As Jared mentioned running multiple Cassandra processes on the same
hardware that participate in the same cluster doesn't make much sense from
a failure domain point of view (it could mess up how C* replicates with
replicas for a key being potentially on the same physical server).

As for splitting up a server for multi-tenancy purposes this then becomes a
question of virtualisation as while there is some multi-tenant support in
C* (auth, throttling per keyspace), it is fairly limited at best.

There a whole range of options out there ranging from xen, vmware etc
through to lightweight virtualisation like linux namespaces with cgroups,
etc.

I think Spotify run C* in production using namespaces with cgroup iirc and
you could using something like docker to help manage this for you.

Docker will also help with managing network addressing etc (the multi homed
aspect). We've also had a lot of success running C* with docker (and
previously SmartOS and solaris zones). Though you will be treading new /
undocumented ground and thus expect to have to solve a few issues along the
way.



On 26 September 2014 04:32, Jared Biel 
wrote:

> Doing this seems counter-productive to Cassandra's design/use-cases. It's
> best at home running on a large number of smaller servers rather than a
> small number of large servers. Also, as you said, you won't get any of the
> high availability benefits that it offers if you run multiple copies of
> Cassandra on the same box.
>
>
> On 25 September 2014 16:58, Donald Smith  > wrote:
>
>>  We have large boxes with 256G of RAM and SSDs.  From iostat, top,
>> and sar we think the system has excess capacity.  Anyone have
>> recommendations about multihoming
>> <http://en.wikipedia.org/wiki/Multihoming> cassandra on such a node
>> (connecting it to multiple IPs and running multiple cassandras
>> simultaneously)?  I’m skeptical, since Cassandra already has built-in
>> multi-threading and since if the node went down multiple nodes would
>> disappear.  We’re using C* version 2.0.9.
>>
>>
>>
>> A google/bing search for  multihoming cassandra doesn’t turn much up.
>>
>>
>>
>> *Donald A. Smith* | Senior Software Engineer
>> P: 425.201.3900 x 3866
>> C: (206) 819-5965
>> F: (646) 443-2333
>> dona...@audiencescience.com
>>
>>
>> [image: AudienceScience]
>>
>>
>>
>
>


-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: DSE install interfering with apache Cassandra 2.1.0

[Ben Bromhead]

check your cqlshrc file (sometimes in ~/.cassandra) ?

I've been caught out before when playing with a RC of 2.1

On 30 September 2014 01:25, Andrew Cobley  wrote:

>  Without the apache cassandra running I ran jps -l on this machine ,the
> only result was
>
>  338 sun.tool.jps.Jps
>
>  The Mac didn’t like the netstat command so I ran
>
>  netstat -atp tcp |  grep 9160
>
>  no result
>
>  Also  for the native port:
>
>  netstat-atp tcp | grep 9042
>
>  gave no result (command may be wrong)
>
>  So I ran port scan using the network utility (between 0 and 1).
> Results as shown:
>
>
>  Port Scan has started…
>
> Port Scanning host: 127.0.0.1
>
>  Open TCP Port: 631ipp
> Port Scan has completed…
>
>
>  Hope this helps.
>
>  Andy
>
>
>  On 29 Sep 2014, at 15:09, Sumod Pawgi  wrote:
>
>  Please run jps to check which Java services are still running and to
> make sure if c* is running. Then please check if 9160 port is in use.
> netstat -nltp | grep 9160
>
>  This will confirm what is happening in your case.
>
> Sent from my iPhone
>
> On 29-Sep-2014, at 7:15 pm, Andrew Cobley  wrote:
>
>  Hi All,
>
>  Just come across this one, I’m at a bit of a loss on how to fix it.
>
>  A user here did the following steps
>
>  On a MAC
> Install Datastax Enterprise (DSE) using the dmg file
> test he can connect using the DSE cqlsh window
> Unistall DSE (full uninstall which stops the services)
>
>  download apache cassandra 2.1.0
> unzip
> change to the non directory run sudo ./cassandra
>
>  Now when he tries to connect using cqlsh from apache cassandra 2.1.0 bin
>   he gets
>
>  Connection error: ('Unable to connect to any servers', {'127.0.0.1':
> ConnectionShutdown('Connection  127.0.0.1:9160 (closed)> is already closed',)})
>
>  This is probably related to
>
> http://mail-archives.apache.org/mod_mbox/incubator-cassandra-user/201409.mbox/%3CCALHCZd7RGSahJUbK32WoTr9JRoA+4K=mrfocmxuk0nbzoqq...@mail.gmail.com%3E
>
>  but I can’t see why the uninstall of DSE is leaving the apache cassandra
> release cqlsh unable to attach to the apache cassandra runtime.
>
>  Ta
> Andy
>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>



-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: DSE install interfering with apache Cassandra 2.1.0

[Ben Bromhead]

Only recently! Moving off list (c* users bcc'd).

On 30 September 2014 19:20, Andrew Cobley  wrote:

>  HI Ben,
>
>  yeah, that was it, recovered from the Cassandra summit ?
>
>
>  Andy
>
>   On 30 Sep 2014, at 08:19, Ben Bromhead  wrote:
>
>  check your cqlshrc file (sometimes in ~/.cassandra) ?
>
>  I've been caught out before when playing with a RC of 2.1
>
> On 30 September 2014 01:25, Andrew Cobley  wrote:
>
>> Without the apache cassandra running I ran jps -l on this machine ,the
>> only result was
>>
>>  338 sun.tool.jps.Jps
>>
>>  The Mac didn’t like the netstat command so I ran
>>
>>  netstat -atp tcp |  grep 9160
>>
>>  no result
>>
>>  Also  for the native port:
>>
>>  netstat-atp tcp | grep 9042
>>
>>  gave no result (command may be wrong)
>>
>>  So I ran port scan using the network utility (between 0 and 1).
>> Results as shown:
>>
>>
>>  Port Scan has started…
>>
>> Port Scanning host: 127.0.0.1
>>
>>  Open TCP Port: 631ipp
>> Port Scan has completed…
>>
>>
>>  Hope this helps.
>>
>>  Andy
>>
>>
>>  On 29 Sep 2014, at 15:09, Sumod Pawgi  wrote:
>>
>>  Please run jps to check which Java services are still running and to
>> make sure if c* is running. Then please check if 9160 port is in use.
>> netstat -nltp | grep 9160
>>
>>  This will confirm what is happening in your case.
>>
>> Sent from my iPhone
>>
>> On 29-Sep-2014, at 7:15 pm, Andrew Cobley 
>> wrote:
>>
>>  Hi All,
>>
>>  Just come across this one, I’m at a bit of a loss on how to fix it.
>>
>>  A user here did the following steps
>>
>>  On a MAC
>> Install Datastax Enterprise (DSE) using the dmg file
>> test he can connect using the DSE cqlsh window
>> Unistall DSE (full uninstall which stops the services)
>>
>>  download apache cassandra 2.1.0
>> unzip
>> change to the non directory run sudo ./cassandra
>>
>>  Now when he tries to connect using cqlsh from apache cassandra 2.1.0
>> bin   he gets
>>
>>  Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>> ConnectionShutdown('Connection > 127.0.0.1:9160 (closed)> is already closed',)})
>>
>>  This is probably related to
>>
>> http://mail-archives.apache.org/mod_mbox/incubator-cassandra-user/201409.mbox/%3CCALHCZd7RGSahJUbK32WoTr9JRoA+4K=mrfocmxuk0nbzoqq...@mail.gmail.com%3E
>>
>>  but I can’t see why the uninstall of DSE is leaving the apache
>> cassandra release cqlsh unable to attach to the apache cassandra runtime.
>>
>>  Ta
>> Andy
>>
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>
>
>
>
>  --
>
> Ben Bromhead
>
> Instaclustr | www.instaclustr.com | @instaclustr
> <http://twitter.com/instaclustr> | +61 415 936 359
>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>



-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: How to enable client-to-node encrypt communication with Astyanax cassandra client

[Ben Bromhead]

Haven't personally followed this but give it a go:
http://lyubent.github.io/security/planetcassandra/2013/05/31/ssl-for-astyanax.html

On 8 October 2014 20:46, Lu, Boying  wrote:

> Hi, All,
>
>
>
> I’m trying to enable client-to-node encrypt communication in Cassandra
> (2.0.7) with Astyanax client library (version=1.56.48)
>
>
>
> I found the links about how to enable this feature:
>
>
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/security/secureSSLClientToNode_t.html
>
> But this only says how to set up in the server side, but not the client
> side.
>
>
>
> Here is my configuration on the server side (in yaml):
>
> client_encryption_options:
>
> enabled: true
>
> keystore:  full-path-to-keystore-file   *#same file used by Cassandra
> server*
>
> keystore_password: some-password
>
> truststore: fullpath-to-truststore-file  *#same file used by
> Cassandra server*
>
> truststore_password: some-password
>
> # More advanced defaults below:
>
> # protocol: TLS
>
> # algorithm: SunX509
>
> # store_type: JKS
>
> cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA]
>
> require_client_auth: true
>
>
>
>
> http://www.datastax.com/dev/blog/accessing-secure-dse-clusters-with-cql-native-protocol
>
> This link says something about client side, but not how to do it with the
> Astyanax client library.
>
>
>
> Searching the Astyanax source codes, I found the class
> SSLConnectionContext maybe useful
>
> And here is my code snippet:
>
> AstyanaxContext clusterContext = new AstyanaxContext.Builder()
>
> .forCluster(clusterName)
>
> .forKeyspace(keyspaceName)
>
> .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
>
> .setRetryPolicy(new QueryRetryPolicy(10, 1000)))
>
> .withConnectionPoolConfiguration(new
> ConnectionPoolConfigurationImpl(_clusterName)
>
> .setMaxConnsPerHost(1)
>
> .setAuthenticationCredentials(credentials)
>
> *.setSSLConnectionContext(sslContext)*
>
> .setSeeds(String.format("%1$s:%2$d", uri.getHost(),
>
> uri.getPort()))
>
> )
>
> .buildCluster(ThriftFamilyFactory.getInstance());
>
>
>
> But when I tried to connect to the Cassandra server, I got following error:
>
> Caused by: org.apache.thrift.transport.TTransportException:
> javax.net.ssl.SSLHandshakeException: Remote host closed connection during
> handshake
>
> at
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>
> at
> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
>
> at
> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
>
> at
> org.apache.cassandra.thrift.Cassandra$Client.send_login(Cassandra.java:567)
>
> at
> org.apache.cassandra.thrift.Cassandra$Client.login(Cassandra.java:559)
>
> at
> com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.open(ThriftSyncConnectionFactoryImpl.java:203)
>
> ... 6 more
>
>
>
> It looks like that my SSL settings are incorrect.
>
>
>
> Does anyone know how to resolve this issue?
>
>
>
> Thanks
>
>
>
> Boying
>



-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: Error: No module named cql

[Ben Bromhead]

It looks like easy_install is using python2.6 and installing cql in
the 2.6 packages directory: /usr/lib/python2.6/site-packages/

cqlsh is using the python executable for you environment (which looks
like 2.7) and thus is looking for cql in the site packages dir
(amongst others).

To quickly install the cql module explicitly for python 2.7 run:
python2.7 -m easy_install cql

Though you might also want to sort out your easy_install so it matches
the version of python that is used by default.

On 15 October 2014 11:48, Tim Dunphy  wrote:
> Hey all,
>
>  I'm using cassandra 2.1.0 on CentOS 6.5
>
>  And when I try to run cqlsh on the command line I get this error:
>
> root@beta-new:~] #cqlsh
>
> Python CQL driver not installed, or not on PYTHONPATH.
> You might try "easy_install cql".
>
> Python: /usr/local/bin/python
> Module load path: ['/usr/local/apache-cassandra-2.1.0/bin',
> '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7',
> '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk',
> '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload',
> '/root/.local/lib/python2.7/site-packages',
> '/usr/local/lib/python2.7/site-packages']
>
> Error: No module named cql
>
>
> I tried following the advice from the error and ran that command:
>
> [root@beta-new:~] #easy_install cql
> Searching for cql
> Best match: cql 1.4.0
> Processing cql-1.4.0-py2.6.egg
> cql 1.4.0 is already the active version in easy-install.pth
>
> Using /usr/lib/python2.6/site-packages/cql-1.4.0-py2.6.egg
> Processing dependencies for cql
> Finished processing dependencies for cql
>
> And that seems to go ok!
>
> However when I try to run it again:
>
> [root@beta-new:~] #cqlsh
>
> Python CQL driver not installed, or not on PYTHONPATH.
> You might try "easy_install cql".
>
> Python: /usr/local/bin/python
> Module load path: ['/usr/local/apache-cassandra-2.1.0/bin',
> '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7',
> '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk',
> '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload',
> '/root/.local/lib/python2.7/site-packages',
> '/usr/local/lib/python2.7/site-packages']
>
> Error: No module named cql
>
> I get the same exact error. How on earth do I break out of this feeback
> loop?
>
> Thanks!
> Tim
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>



-- 
Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359

Re: cqsl doesn't work cassandra 2.1.0

[Ben Bromhead]

check if you have an existing ~/.cassandra/cqlshrc file

On 21 October 2014 12:48, Michael Shuler  wrote:

> On 10/19/2014 06:23 PM, Tim Dunphy wrote:
>
>> I've upgraded to Cassandra 2.1.0.
>>
>
> Via what method, tar or deb, and from what version to 2.1.0?
>
>  This is what I bet when I run the cqslh command:
>>
>> Can't locate transport factory function
>> cqlshlib.tfactory.regular_transport_factory
>>
>
> cqlshlib.tfactory is thrift. cqlsh in 2.1.0 does not use thrift. I imagine
> your $PATH is finding an old version of cqlsh or the old cql jar is in your
> classpath or something along those lines.
>
> --
> Kind regards,
> Michael
>



-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: Dependency Hell: STORM 0.9.2 and Cassandra 2.0

[Ben Bromhead]

I haven't had to deal with this problem specifically and don't know if
there is a storm specific solution, but the general Java way of dealing
with projects who have conflicting dependencies would be to either exclude
one of the conflicting dependencies using maven and see if it works,
otherwise rename the conflicting dependency using
http://maven.apache.org/plugins/maven-dependency-plugin/usage.html so both
projects can use there own versions of guava without the package names
conflicting (and the jvm will load the correct classes for each dep).

On 25 October 2014 06:13, Gary Zhao  wrote:

> Hello
>
> Anyone encountered the following issue and any workaround? Our Storm
> topology was written in Clojure.
>
> 
> Our team is upgrading one of our storm topology from using cassandra 1.2
> to cassandra 2.0, and we have found one problem that is difficult to
> tackle. Cassandra 2.0Java driver requires google guava 1.6. Unfortuanately,
> storm 0.9.2 provides a lower version. Because of that, a topology will not
> be able to contact Cassandra databases.
>
> Thanks
> Gary
>



-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: Repair/Compaction Completion Confirmation

[Ben Bromhead]

https://github.com/BrianGallew/cassandra_range_repair

This breaks down the repair operation into very small portions of the ring
as a way to try and work around the current fragile nature of repair.

Leveraging range repair should go some way towards automating repair (this
is how the automatic repair service in DataStax opscenter works, this is
how we perform repairs).

We have had a lot of success running repairs in a similar manner against
vnode enabled clusters. Not 100% bullet proof, but way better than nodetool
repair



On 28 October 2014 08:32, Tim Heckman  wrote:

> On Mon, Oct 27, 2014 at 1:44 PM, Robert Coli  wrote:
>
>> On Mon, Oct 27, 2014 at 1:33 PM, Tim Heckman  wrote:
>>
>>> I know that when issuing some operations via nodetool, the command
>>> blocks until the operation is finished. However, is there a way to reliably
>>> determine whether or not the operation has finished without monitoring that
>>> invocation of nodetool?
>>>
>>> In other words, when I run 'nodetool repair' what is the best way to
>>> reliably determine that the repair is finished without running something
>>> equivalent to a 'pgrep' against the command I invoked? I am curious about
>>> trying to do the same for major compactions too.
>>>
>>
>> This is beyond a FAQ at this point, unfortunately; non-incremental repair
>> is awkward to deal with and probably impossible to automate.
>>
>> In The Future [1] the correct solution will be to use incremental repair,
>> which mitigates but does not solve this challenge entirely.
>>
>> As brief meta commentary, it would have been nice if the project had
>> spent more time optimizing the operability of the critically important
>> thing you must do once a week [2].
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-5483
>>
>> =Rob
>> [1] http://www.datastax.com/dev/blog/anticompaction-in-cassandra-2-1
>> [2] Or, more sensibly, once a month with gc_grace_seconds set to 34 days.
>>
>
> Thank you for getting back to me so quickly. Not the answer that I was
> secretly hoping for, but it is nice to have confirmation. :)
>
> Cheers!
> -Tim
>



-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: bootstrapping manually when auto_bootstrap=false ?

[Ben Bromhead]

   - In Cassandra yaml set auto_bootstrap = false
   - Boot node
   - nodetool rebuild

Very similar to
http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html

On 18 December 2014 at 14:04, Kevin Burton  wrote:
>
> I’m trying to figure out the best way to bootstrap our nodes.
>
> I *think* I want our nodes to be manually bootstrapped.  This way an admin
> has to explicitly bring up the node in the cluster and I don’t have to
> worry about a script accidentally provisioning new nodes.
>
> The problem is HOW do you do it?
>
> I couldn’t find any reference anywhere in the documentation.
>
> I *think* I run nodetool repair? but it’s unclear..
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>

-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: simple data movement ?

[Ben Bromhead]

Just copy the data directory from each prod node to your test node (and
relevant configuration files etc).

If your IP addresses are different between test and prod, follow
https://engineering.eventbrite.com/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/


On 18 December 2014 at 09:10, Langston, Jim 
wrote:
>
>  Hi all,
>
>  I have set up a test environment with C* 2.1.2, wanting to test our
> applications against it. I currently have C* 1.2.9 in production and want
> to use that data for testing. What would be a good approach for simply
> taking a copy of the production data and moving it into the test env and
> having the test env C* use that data ?
>
>  The test env. is identical is size, with the difference being the
> versions
> of C*.
>
>  Thanks,
>
>  Jim
>  The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it
>


-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | +61 415 936 359

Re: run cassandra on a small instance

[Ben Bromhead]

t;  'native_transport_min_threads' => "2",
>>>  'native_transport_max_threads' => "4",
>>>  'notify_restart' => true,
>>>  'reporter' => {
>>>        'riemann' => {
>>>  'enable' => true,
>>>  'host' => '192.168.33.51'
>>>},
>>>'graphite' => {
>>>  'enable' => true,
>>>  'host' => '192.168.33.51'
>>>}
>>>  }
>>>},
>>>
>>> The most important things to note:
>>> - don't include JNA (it needs to lock pages larger than what will be
>>> available)
>>> - turn down threadpools for transports
>>> - turn compaction throughput way down
>>> - make concurrent reads and writes very small
>>>
>>> I have used the above run a healthy 5 node clusters locally in it's own
>>> private network with a 6th monitoring server for light to moderate local
>>> testing in 16g of laptop ram. YMMV but it is possible.
>>>
>>
>>
>>
>> --
>> GPG me!!
>>
>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>
>>
>
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>
>


-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | (650) 284 9692

Re: Unexplained query slowness

[Ben Bromhead]

Have a look at your column family histograms (nodetool cfhistograms iirc),
if you notice things like a very long tail, a double hump or outliers it
would indicate something wrong with your data model or you have a hot
partition key/s.

Also looking at your 99 and 95 percentile latencies will just hide these
occasional high latency reads as they fall outside these percentiles.

If you are running a stock config, first rule out that its not your data
model, then investigate things like disk latency, noisy neighbours (if you
are on vms/ in the cloud).

On 26 February 2015 at 03:01, Marcelo Valle (BLOOMBERG/ LONDON) <
mvallemil...@bloomberg.net> wrote:

> I am sorry if it's too basic and you already looked at that, but the first
> thing I would ask would be the data model.
>
> What data model are you using (how is your data partitioned)? What queries
> are you running? If you are using ALLOW FILTERING, for instance, it will be
> very easy to say why it's slow.
>
> Most times people get slow queries in Cassandra they are using the wrong
> data model.
>
> []s
>
> From: user@cassandra.apache.org
> Subject: Re:Unexplained query slowness
>
> Our Cassandra database just rolled to live last night. I’m looking at our
> query performance, and overall it is very good, but perhaps 1 in 10,000
> queries takes several hundred milliseconds (up to a full second). I’ve
> grepped for GC in the system.log on all nodes, and there aren’t any recent
> GC events. I’m executing ~500 queries per second, which produces negligible
> load and CPU utilization. I have very minimal writes (one every few
> minutes). The slow queries are across the board. There isn’t one particular
> query that is slow.
>
> I’m running 2.0.12 with SSD’s. I’ve got a 10 node cluster with RF=3.
>
> I have no idea where to even begin to look. Any thoughts on where to start
> would be greatly appreciated.
>
> Robert
>
>
>


-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | (650) 284 9692

Re: how to clear data from disk

[Ben Bromhead]

To clarify on why this behaviour occurs, by default Cassandra will snapshot
a table when you perform any destructive action (TRUNCATE, DROP etc)

see
http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/truncate_r.html

To free disk space after such an operation you will always need to clear
the snapshots (using either of above suggested methods). Unfortunately this
can be a bit painful if you are rotating your tables, say by month, and
want to remove the oldest one from disk as your client will need to speak
JMX as well.

You can disable this behaviour through the use of auto_snapshot in
cassandra.yaml. Though I would strongly recommend leaving this feature
enabled in any sane production environment and cleaning up snapshots as an
independent task!!

On 10 March 2015 at 20:43, Patrick McFadin  wrote:

> Or just manually delete the files. The directories are broken down by
> keyspace and table.
>
> Patrick
>
> On Mon, Mar 9, 2015 at 7:50 PM, 曹志富  wrote:
>
>> nodetool clearsnapshot
>>
>> --
>> Ranger Tsao
>>
>> 2015-03-10 10:47 GMT+08:00 鄢来琼 :
>>
>>>  Hi ALL,
>>>
>>>
>>>
>>> After drop table, I found the data is not removed from disk, I should
>>> reduce the gc_grace_seconds before the drop operation.
>>>
>>> I have to wait for 10 days, but there is not enough disk.
>>>
>>> Could you tell me there is method to clear the data from disk quickly?
>>>
>>> Thank you very much!
>>>
>>>
>>>
>>> Peter
>>>
>>
>>
>


-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | (650) 284 9692

Re: Deleted snapshot files filling up /var/lib/cassandra

[Ben Bromhead]

Cassandra will by default snapshot your data directory on the following
events:

   - TRUNCATE and DROP schema events
   - when you run nodetool repair
   - when you run nodetool snapshot

Snapshots are just hardlinks to existing SSTables so the only disk space
they take up is for files that have since been compacted away. Disk space
for snapshots will be freed when the last link to the files are removed.
You can remove all snapshots in a cluster using nodetool clearsnapshot

Snapshots will fail if you are out of disk space (this is counterintuitive
to the above, but it is true), if you have not increased the number of
available file descriptors or if there are permissions issues.

Out of curiosity, how often are you running repair?

On 16 March 2015 at 15:52, David Wahler  wrote:

> On Mon, Mar 16, 2015 at 5:28 PM, Jan  wrote:
> > David;
> >
> > all the packaged installations use the /var/lib/cassandra directory.
> > Could you check your yaml config files and see if you are using this
> default
> > directory  for backups
> >
> > May want to change it to a location with more disk space.
>
> We're using the default /var/lib/cassandra as our data directory,
> mounted as its own LVM volume. I don't see anything in cassandra.yaml
> about a backup directory. There is an "incremental_backups" option
> which is set to false.
>
> Increasing the available disk space doesn't really seem like a
> solution. We have only about 450MB of live data on the most
> heavily-loaded server, and the space taken up by these deleted files
> is growing by several GB per day. For now we can work around the
> problem by periodically restarting servers to close the file handles,
> but that hurts our availability and seems like a hack.
>



-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | (650) 284 9692

Re: Deleted snapshot files filling up /var/lib/cassandra

[Ben Bromhead]

If you are running a sequential repair (or have previously run a sequential
repair that is still running) Cassandra will still have the file
descriptors open for files in the snapshot it is using for the repair
operation.

>From the http://www.datastax.com/dev/blog/repair-in-cassandra

*Cassandra 1.2 introduced a new option to repair to help manage the
problems caused by the nodes all repairing with each other at the same
time, it is call a snapshot repair, or sequential repair. As of Cassandra
2.1, sequential repair is the default, and the old parallel repair an
option. Sequential repair has all of the nodes involved take a snapshot,
the snapshot lives until the repair finishes, and then is removed. By
taking a snapshot, repair can procede in a serial fashion, such that only
two nodes are ever comparing with each other at a time. This makes the
overall repair process slower, but decreases the burden placed on the
nodes, and means you have less impact on reads/writes to the system.*

On 16 March 2015 at 16:33, David Wahler  wrote:

> On Mon, Mar 16, 2015 at 6:12 PM, Ben Bromhead  wrote:
> > Cassandra will by default snapshot your data directory on the following
> > events:
> >
> > TRUNCATE and DROP schema events
> > when you run nodetool repair
> > when you run nodetool snapshot
> >
> > Snapshots are just hardlinks to existing SSTables so the only disk space
> > they take up is for files that have since been compacted away. Disk space
> > for snapshots will be freed when the last link to the files are removed.
> You
> > can remove all snapshots in a cluster using nodetool clearsnapshot
> >
> > Snapshots will fail if you are out of disk space (this is
> counterintuitive
> > to the above, but it is true), if you have not increased the number of
> > available file descriptors or if there are permissions issues.
> >
> > Out of curiosity, how often are you running repair?
>
> Thanks for the information. We're running repair once per week, as
> recommended by the Datastax documentation. The repair is staggered to
> run on one machine at a time with the --partitioner-range option in
> order to spread out the load.
>
> Running "nodetool clearsnapshot" doesn't free up any space. I'm
> guessing that because the snapshot files have been deleted from the
> filesystem, Cassandra thinks the snapshots are already gone. But
> because it still has the file descriptors open, the disk space hasn't
> actually been reclaimed.
>



-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | (650) 284 9692

Re: Deleted snapshot files filling up /var/lib/cassandra

[Ben Bromhead]

Sorry for the late reply.

To immediately solve the problem you can restart Cassandra and all the open
file descriptors to the deleted snapshots should disappear.

As for why it happened I would first address the disk space issue and see
if the snapshot errors + open file descriptors issue still occurs (I am
unclear as to whether you got the snapshot exception after the disk filled
up or before), if you still have issues with repair not letting go of
snapshotted files even with free disk space I would look to raise a ticket
in Jira.

On 17 March 2015 at 12:46, David Wahler  wrote:

> On Mon, Mar 16, 2015 at 6:51 PM, Ben Bromhead  wrote:
> > If you are running a sequential repair (or have previously run a
> sequential
> > repair that is still running) Cassandra will still have the file
> descriptors
> > open for files in the snapshot it is using for the repair operation.
>
> Yeah, that aligns with my understanding of how the repair process
> works. But the cluster has no repair sessions active (I think; when I
> run "nodetool tpstats", the AntiEntropyStage and AntiEntropySessions
> values are zero on all nodes) and the space still hasn't been freed.
>



-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | (650) 284 9692

Re: Really high read latency

[Ben Bromhead]

whether you're hitting a lot of tombstones (are you
>>> deleting lots of values in the same partition over time?).
>>>
>>> On Mon, Mar 23, 2015 at 4:01 AM, Dave Galbraith <
>>> david92galbra...@gmail.com> wrote:
>>>
>>>> Duncan: I'm thinking it might be something like that. I'm also seeing
>>>> just a ton of garbage collection on the box, could it be pulling rows for
>>>> all 100k attrs for a given row_time into memory since only row_time is the
>>>> partition key?
>>>>
>>>> Jens: I'm not using EBS (although I used to until I read up on how
>>>> useless it is). I'm not sure what constitutes proper paging but my client
>>>> has a pretty small amount of available memory so I'm doing pages of size 5k
>>>> using the C++ Datastax driver.
>>>>
>>>> Thanks for the replies!
>>>>
>>>> -Dave
>>>>
>>>> On Mon, Mar 23, 2015 at 2:00 AM, Jens Rantil 
>>>> wrote:
>>>>
>>>>> Also, two control questions:
>>>>>
>>>>>- Are you using EBS for data storage? It might introduce
>>>>>additional latencies.
>>>>>- Are you doing proper paging when querying the keyspace?
>>>>>
>>>>> Cheers,
>>>>> Jens
>>>>>
>>>>> On Mon, Mar 23, 2015 at 5:56 AM, Dave Galbraith <
>>>>> david92galbra...@gmail.com> wrote:
>>>>>
>>>>>> Hi! So I've got a table like this:
>>>>>>
>>>>>> CREATE TABLE "default".metrics (row_time int,attrs varchar,offset
>>>>>> int,value double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT
>>>>>> STORAGE AND bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND
>>>>>> comment='' AND dclocal_read_repair_chance=0 AND gc_grace_seconds=864000 
>>>>>> AND
>>>>>> index_interval=128 AND read_repair_chance=1 AND replicate_on_write='true'
>>>>>> AND populate_io_cache_on_flush='false' AND default_time_to_live=0 AND
>>>>>> speculative_retry='NONE' AND memtable_flush_period_in_ms=0 AND
>>>>>> compaction={'class':'DateTieredCompactionStrategy','timestamp_resolution':'MILLISECONDS'}
>>>>>> AND compression={'sstable_compression':'LZ4Compressor'};
>>>>>>
>>>>>> and I'm running Cassandra on an EC2 m3.2xlarge out in the cloud, with
>>>>>> 4 GB of heap space. So it's timeseries data that I'm doing so I increment
>>>>>> "row_time" each day, "attrs" is additional identifying information about
>>>>>> each series, and "offset" is the number of milliseconds into the day for
>>>>>> each data point. So for the past 5 days, I've been inserting 3k
>>>>>> points/second distributed across 100k distinct "attrs"es. And now when I
>>>>>> try to run queries on this data that look like
>>>>>>
>>>>>> "SELECT * FROM "default".metrics WHERE row_time = 5 AND attrs =
>>>>>> 'potatoes_and_jam'"
>>>>>>
>>>>>> it takes an absurdly long time and sometimes just times out. I did
>>>>>> "nodetool cftsats default" and here's what I get:
>>>>>>
>>>>>> Keyspace: default
>>>>>> Read Count: 59
>>>>>> Read Latency: 397.12523728813557 ms.
>>>>>> Write Count: 155128
>>>>>> Write Latency: 0.3675690719921613 ms.
>>>>>> Pending Flushes: 0
>>>>>> Table: metrics
>>>>>> SSTable count: 26
>>>>>> Space used (live): 35146349027
>>>>>> Space used (total): 35146349027
>>>>>> Space used by snapshots (total): 0
>>>>>> SSTable Compression Ratio: 0.10386468749216264
>>>>>> Memtable cell count: 141800
>>>>>> Memtable data size: 31071290
>>>>>> Memtable switch count: 41
>>>>>> Local read count: 59
>>>>>> Local read latency: 397.126 ms
>>>>>> Local write count: 155128
>>>>>> Local write latency: 0.368 ms
>>>>>> Pending flushes: 0
>>>>>> Bloom filter false positives: 0
>>>>>> Bloom filter false ratio: 0.0
>>>>>> Bloom filter space used: 2856
>>>>>> Compacted partition minimum bytes: 104
>>>>>> Compacted partition maximum bytes: 36904729268
>>>>>> Compacted partition mean bytes: 986530969
>>>>>> Average live cells per slice (last five minutes):
>>>>>> 501.66101694915255
>>>>>> Maximum live cells per slice (last five minutes): 502.0
>>>>>> Average tombstones per slice (last five minutes): 0.0
>>>>>> Maximum tombstones per slice (last five minutes): 0.0
>>>>>>
>>>>>> Ouch! 400ms of read latency, orders of magnitude higher than it has
>>>>>> any right to be. How could this have happened? Is there something
>>>>>> fundamentally broken about my data model? Thanks!
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jens Rantil
>>>>> Backend engineer
>>>>> Tink AB
>>>>>
>>>>> Email: jens.ran...@tink.se
>>>>> Phone: +46 708 84 18 32
>>>>> Web: www.tink.se
>>>>>
>>>>> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
>>>>> <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
>>>>>  Twitter <https://twitter.com/tink>
>>>>>
>>>>
>>>>
>>>
>>
>


-- 

Ben Bromhead

Instaclustr | www.instaclustr.com | @instaclustr
<http://twitter.com/instaclustr> | (650) 284 9692