Re: upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

[Oleksandr Shulgin]

On Wed, Mar 27, 2019 at 6:36 PM Carl Mueller
 wrote:

>
> EIPs per the aws experts cost money,
>

>From what I know they only cost you when you're not using them.  This page
shows that you are also charged if you remap them too often (more then 100
times a month), this I didn't realize:
https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Addresses

are limited in resources (we have a lot of VMs) and cause a lot of
> headaches in our autoscaling / infrastructure as code systems.
>

But you are not trying to autoscale Cassandra, do you?  Cloud Formation has
a decent support of EIPs, e.g. you can allocate them by declaring in
resources and then resolve the address to inject configuration parameters
for the application if needed.

> We are probably going to just have a VM startup script for now that
automatically updates the yaml on instance restart. It seems to be the
least-sucky approach at this point.

This is what we do for our docker-based setup.  I think we were just
following the documentation, though you're looking at the code proves that
this shouldn't be required..

Regards,
--
Alex

Five Questions for Cassandra Users

[Kenneth Brotman]

I'm looking to get a better feel for how people use Cassandra in practice.
I thought others would benefit as well so may I ask you the following five
questions:

 

1.   Do the same people where you work operate the cluster and write the
code to develop the application?

 

2.   Do you have a metrics stack that allows you to see graphs of
various metrics with all the nodes displayed together?

 

3.   Do you have a log stack that allows you to see the logs for all the
nodes together?

 

4.   Do you regularly repair your clusters - such as by using Reaper?

 

5.   Do you use artificial intelligence to help manage your clusters?

 

 

Thank you for taking your time to share this information!

 

Kenneth Brotman

Re: Five Questions for Cassandra Users

[Abhishek Singh]

1.   Do the same people where you work operate the cluster and write
the code to develop the application?

   Different teams. Infra separate , Dev separate.

2.   Do you have a metrics stack that allows you to see graphs of
various metrics with all the nodes displayed together?

   Use third party APM tool to monitor cluster

3.   Do you have a log stack that allows you to see the logs for all
the nodes together?

   No.Would like to.

4.   Do you regularly repair your clusters - such as by using Reaper?

Yes

5.   Do you use artificial intelligence to help manage your clusters?

   No

On Thu, 28 Mar, 2019, 2:33 PM Kenneth Brotman, 
wrote:

> I’m looking to get a better feel for how people use Cassandra in
> practice.  I thought others would benefit as well so may I ask you the
> following five questions:
>
>
>
> 1.   Do the same people where you work operate the cluster and write
> the code to develop the application?
>
>
>
> 2.   Do you have a metrics stack that allows you to see graphs of
> various metrics with all the nodes displayed together?
>
>
>
> 3.   Do you have a log stack that allows you to see the logs for all
> the nodes together?
>
>
>
> 4.   Do you regularly repair your clusters - such as by using Reaper?
>
>
>
> 5.   Do you use artificial intelligence to help manage your clusters?
>
>
>
>
>
> Thank you for taking your time to share this information!
>
>
>
> Kenneth Brotman
>

Re: Five Questions for Cassandra Users

[Jürgen Albersdorfer]

1.   Do the same people where you work operate the cluster and write
the code to develop the application?

yes

2.   Do you have a metrics stack that allows you to see graphs of
various metrics with all the nodes displayed together?

 no

3.   Do you have a log stack that allows you to see the logs for all
the nodes together?

 no

4.   Do you regularly repair your clusters - such as by using Reaper?

 no, never did - try to stay away from ever doing.

5.   Do you use artificial intelligence to help manage your clusters?

no

Am Do., 28. März 2019 um 10:03 Uhr schrieb Kenneth Brotman
:

> I’m looking to get a better feel for how people use Cassandra in
> practice.  I thought others would benefit as well so may I ask you the
> following five questions:
>
>
>
> 1.   Do the same people where you work operate the cluster and write
> the code to develop the application?
>
>
>
> 2.   Do you have a metrics stack that allows you to see graphs of
> various metrics with all the nodes displayed together?
>
>
>
> 3.   Do you have a log stack that allows you to see the logs for all
> the nodes together?
>
>
>
> 4.   Do you regularly repair your clusters - such as by using Reaper?
>
>
>
> 5.   Do you use artificial intelligence to help manage your clusters?
>
>
>
>
>
> Thank you for taking your time to share this information!
>
>
>
> Kenneth Brotman
>

Best practices while designing backup storage system for big Cassandra cluster

[manish khandelwal]

Hi



I would like to know is there any guideline for selecting storage device
(disk type) for Cassandra backups.



As per my current observation, NearLine (NL) disk on SAN  slows down
significantly while copying backup files (taking full backup) from all node
simultaneously. Will using SSD disk on SAN help us in this regard?

Apart from using SSD disk, what are the alternative approach to make my
backup process fast?

What are the best practices while designing backup storage system for big
Cassandra cluster?


Regards

Manish

Re: Five Questions for Cassandra Users

[Tom van der Woerdt]

1.   Do the same people where you work operate the cluster and write
the code to develop the application?

No, we have a small infrastructure team, and many people developing
applications using Cassandra

2.   Do you have a metrics stack that allows you to see graphs of
various metrics with all the nodes displayed together?

Yes, we use a re-implementation of Graphite, which we open-sourced and now
lives at https://github.com/go-graphite

3.   Do you have a log stack that allows you to see the logs for all
the nodes together?

Yes, although in practice we don't use it much for Cassandra

4.   Do you regularly repair your clusters - such as by using Reaper?

Yes, we have built our own tools for this

5.   Do you use artificial intelligence to help manage your clusters?

It's not "artificial intelligence" the way most people would describe it,
but we certainly don't run our clusters manually



Tom van der Woerdt
Site Reliability Engineer

Booking.com B.V.
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
[image: Booking.com] 
Empowering people to experience the world since 1996
43 languages, 214+ offices worldwide, 141,000+ global destinations, 29
million reported listings
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)


On Thu, Mar 28, 2019 at 10:03 AM Kenneth Brotman
 wrote:

> I’m looking to get a better feel for how people use Cassandra in
> practice.  I thought others would benefit as well so may I ask you the
> following five questions:
>
>
>
> 1.   Do the same people where you work operate the cluster and write
> the code to develop the application?
>
>
>
> 2.   Do you have a metrics stack that allows you to see graphs of
> various metrics with all the nodes displayed together?
>
>
>
> 3.   Do you have a log stack that allows you to see the logs for all
> the nodes together?
>
>
>
> 4.   Do you regularly repair your clusters - such as by using Reaper?
>
>
>
> 5.   Do you use artificial intelligence to help manage your clusters?
>
>
>
>
>
> Thank you for taking your time to share this information!
>
>
>
> Kenneth Brotman
>

Re: Five Questions for Cassandra Users

[Elliott Sims]

1.   Do the same people where you work operate the cluster and write
the code to develop the application?

Mostly.  Ops vs dev, although there's some overlap

2.   Do you have a metrics stack that allows you to see graphs of
various metrics with all the nodes displayed together?

 Yes, Prometheus+Grafana (currently custom script reporting to Prometheus,
but that needs revisiting)

3.   Do you have a log stack that allows you to see the logs for all
the nodes together?

 Yep, graylog.

4.   Do you regularly repair your clusters - such as by using Reaper?

 Yes, with reaper.  Every day or two, more or less.  It would be
almost-constant if Reaper could work off queues with blacklisted time
windows instead of a schedule

5.   Do you use artificial intelligence to help manage your clusters?

No.

On Thu, Mar 28, 2019 at 8:46 AM Tom van der Woerdt
 wrote:

> 1.   Do the same people where you work operate the cluster and write
> the code to develop the application?
>
> No, we have a small infrastructure team, and many people developing
> applications using Cassandra
>
> 2.   Do you have a metrics stack that allows you to see graphs of
> various metrics with all the nodes displayed together?
>
> Yes, we use a re-implementation of Graphite, which we open-sourced and now
> lives at https://github.com/go-graphite
>
> 3.   Do you have a log stack that allows you to see the logs for all
> the nodes together?
>
> Yes, although in practice we don't use it much for Cassandra
>
> 4.   Do you regularly repair your clusters - such as by using Reaper?
>
> Yes, we have built our own tools for this
>
> 5.   Do you use artificial intelligence to help manage your clusters?
>
> It's not "artificial intelligence" the way most people would describe it,
> but we certainly don't run our clusters manually
>
>
>
> Tom van der Woerdt
> Site Reliability Engineer
>
> Booking.com B.V.
> Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
> [image: Booking.com] 
> Empowering people to experience the world since 1996
> 43 languages, 214+ offices worldwide, 141,000+ global destinations, 29
> million reported listings
> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
>
>
> On Thu, Mar 28, 2019 at 10:03 AM Kenneth Brotman
>  wrote:
>
>> I’m looking to get a better feel for how people use Cassandra in
>> practice.  I thought others would benefit as well so may I ask you the
>> following five questions:
>>
>>
>>
>> 1.   Do the same people where you work operate the cluster and write
>> the code to develop the application?
>>
>>
>>
>> 2.   Do you have a metrics stack that allows you to see graphs of
>> various metrics with all the nodes displayed together?
>>
>>
>>
>> 3.   Do you have a log stack that allows you to see the logs for all
>> the nodes together?
>>
>>
>>
>> 4.   Do you regularly repair your clusters - such as by using Reaper?
>>
>>
>>
>> 5.   Do you use artificial intelligence to help manage your clusters?
>>
>>
>>
>>
>>
>> Thank you for taking your time to share this information!
>>
>>
>>
>> Kenneth Brotman
>>
>

Re: Five Questions for Cassandra Users

[Jonathan Koppenhofer]

1. PaaS model. We have a team responsible for the deployment and
self-service tooling, as well as SME for both development and Cassandra
operations. End users consume the service, and are responsible for app
development and operations. Larger apps have separate teams for this,
smaller apps have a single text for both

2. Homegrown with custom agent piping stats to a Cassandra cluster. Grafana
with custom http reader to read metrics from homegrown API. If it would
have existed when we first did this, we probably would have worked in
Prometheus.

3. Yes. ELK and/or Splunk

4. Used homegrown repair mechanism before 2.2. Now use reaper. PaaS
consumers responsible for configuring repairs.

5. No. Need to get better here, but "real" AI seems to be a.bew trend we
have seen talked about on this list.


On Thu, Mar 28, 2019, 5:03 AM Kenneth Brotman 
wrote:

> I’m looking to get a better feel for how people use Cassandra in
> practice.  I thought others would benefit as well so may I ask you the
> following five questions:
>
>
>
> 1.   Do the same people where you work operate the cluster and write
> the code to develop the application?
>
>
>
> 2.   Do you have a metrics stack that allows you to see graphs of
> various metrics with all the nodes displayed together?
>
>
>
> 3.   Do you have a log stack that allows you to see the logs for all
> the nodes together?
>
>
>
> 4.   Do you regularly repair your clusters - such as by using Reaper?
>
>
>
> 5.   Do you use artificial intelligence to help manage your clusters?
>
>
>
>
>
> Thank you for taking your time to share this information!
>
>
>
> Kenneth Brotman
>

Re: Five Questions for Cassandra Users

[Jonathan Koppenhofer]

I think it would also be interesting to hear how people are handling
automation (which to me is different than AI) and config management.

For us it is a combo of custom Java workflows and Saltstack.

On Thu, Mar 28, 2019, 5:03 AM Kenneth Brotman 
wrote:

> I’m looking to get a better feel for how people use Cassandra in
> practice.  I thought others would benefit as well so may I ask you the
> following five questions:
>
>
>
> 1.   Do the same people where you work operate the cluster and write
> the code to develop the application?
>
>
>
> 2.   Do you have a metrics stack that allows you to see graphs of
> various metrics with all the nodes displayed together?
>
>
>
> 3.   Do you have a log stack that allows you to see the logs for all
> the nodes together?
>
>
>
> 4.   Do you regularly repair your clusters - such as by using Reaper?
>
>
>
> 5.   Do you use artificial intelligence to help manage your clusters?
>
>
>
>
>
> Thank you for taking your time to share this information!
>
>
>
> Kenneth Brotman
>

RE: Five Questions for Cassandra Users

[Kenneth Brotman]

Yes, absolutely!

 

From: Jonathan Koppenhofer [mailto:j...@koppedomain.com] 
Sent: Thursday, March 28, 2019 1:18 PM
To: user@cassandra.apache.org
Subject: Re: Five Questions for Cassandra Users

 

I think it would also be interesting to hear how people are handling automation 
(which to me is different than AI) and config management.

 

For us it is a combo of custom Java workflows and Saltstack.

 

On Thu, Mar 28, 2019, 5:03 AM Kenneth Brotman  
wrote:

I’m looking to get a better feel for how people use Cassandra in practice.  I 
thought others would benefit as well so may I ask you the following five 
questions:

 

1.   Do the same people where you work operate the cluster and write the 
code to develop the application?

 

2.   Do you have a metrics stack that allows you to see graphs of various 
metrics with all the nodes displayed together?

 

3.   Do you have a log stack that allows you to see the logs for all the 
nodes together?

 

4.   Do you regularly repair your clusters - such as by using Reaper?

 

5.   Do you use artificial intelligence to help manage your clusters?

 

 

Thank you for taking your time to share this information!

 

Kenneth Brotman

Cassandra Possible read/write race condition in LOCAL_ONE?

[Richard Xin]

Hi, Our Cassandra Consistency level is currently set to LOCAL_ONE, we have 
script doing followings
1) insert one record into table_A 
2) select last_inserted_record from table_A and do something ...

step #1 & 2 are running sequentially without pause,  and I assume 1 & 2 suppose 
to run in same DC
we are facing sporadic issues that step #2 didnt get inserted data by #1.is it 
possible to have a race condition when LOCAL_ONE that #2 might not get inserted 
data on step #1?
Thanks in advance!Richard

Re: Cassandra Possible read/write race condition in LOCAL_ONE?

[Jeff Jirsa]

Yes it can race; if you don't want to race, you'd want to use SERIAL or
LOCAL_SERIAL.

On Thu, Mar 28, 2019 at 3:04 PM Richard Xin 
wrote:

> Hi,
> Our Cassandra Consistency level is currently set to LOCAL_ONE, we have
> script doing followings
> 1) insert one record into table_A
> 2) select last_inserted_record from table_A and do something ...
>
> step #1 & 2 are running sequentially without pause,  and I assume 1 & 2
> suppose to run in same DC
>
> we are facing sporadic issues that step #2 didnt get inserted data by #1.
> is it possible to have a race condition when LOCAL_ONE that #2 might not
> get inserted data on step #1?
>
> Thanks in advance!
> Richard
>

Re: Cassandra Possible read/write race condition in LOCAL_ONE?

[Jon Haddad]

I'm reading the OP as doing this from a single server, if that's the
case QUORUM / LOCAL_QUORUM will work.

On Thu, Mar 28, 2019 at 3:29 PM Jeff Jirsa  wrote:
>
> Yes it can race; if you don't want to race, you'd want to use SERIAL or 
> LOCAL_SERIAL.
>
> On Thu, Mar 28, 2019 at 3:04 PM Richard Xin  
> wrote:
>>
>> Hi,
>> Our Cassandra Consistency level is currently set to LOCAL_ONE, we have 
>> script doing followings
>> 1) insert one record into table_A
>> 2) select last_inserted_record from table_A and do something ...
>>
>> step #1 & 2 are running sequentially without pause,  and I assume 1 & 2 
>> suppose to run in same DC
>>
>> we are facing sporadic issues that step #2 didnt get inserted data by #1.
>> is it possible to have a race condition when LOCAL_ONE that #2 might not get 
>> inserted data on step #1?
>>
>> Thanks in advance!
>> Richard

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org