Re: A cluster (RF=3) not recovering after two nodes are stopped

[Hiroyuki Yamada]

Hello,

Sorry again.
We found yet another weird thing in this.
If we stop nodes with systemctl or just kill (TERM), it causes the problem,
but if we kill -9, it doesn't cause the problem.

Thanks,
Hiro

On Wed, Apr 24, 2019 at 11:31 PM Hiroyuki Yamada  wrote:

> Sorry, I didn't write the version and the configurations.
> I've tested with C* 3.11.4, and
> the configurations are mostly set to default except for the replication
> factor and listen_address for proper networking.
>
> Thanks,
> Hiro
>
> On Wed, Apr 24, 2019 at 5:12 PM Hiroyuki Yamada 
> wrote:
>
>> Hello Ben,
>>
>> Thank you for the quick reply.
>> I haven't tried that case, but it does't recover even if I stopped the
>> stress.
>>
>> Thanks,
>> Hiro
>>
>> On Wed, Apr 24, 2019 at 3:36 PM Ben Slater 
>> wrote:
>>
>>> Is it possible that stress is overloading node 1 so it’s not recovering
>>> state properly when node 2 comes up? Have you tried running with a lower
>>> load (say 2 or 3 threads)?
>>>
>>> Cheers
>>> Ben
>>>
>>> ---
>>>
>>>
>>> *Ben Slater*
>>> *Chief Product Officer*
>>>
>>>
>>> 
>>> 
>>> 
>>>
>>> Read our latest technical blog posts here
>>> .
>>>
>>> This email has been sent on behalf of Instaclustr Pty. Limited
>>> (Australia) and Instaclustr Inc (USA).
>>>
>>> This email and any attachments may contain confidential and legally
>>> privileged information.  If you are not the intended recipient, do not copy
>>> or disclose its content, but please reply to this email immediately and
>>> highlight the error to the sender and then immediately delete the message.
>>>
>>>
>>> On Wed, 24 Apr 2019 at 16:28, Hiroyuki Yamada 
>>> wrote:
>>>
 Hello,

 I faced a weird issue when recovering a cluster after two nodes are
 stopped.
 It is easily reproduce-able and looks like a bug or an issue to fix,
 so let me write down the steps to reproduce.

 === STEPS TO REPRODUCE ===
 * Create a 3-node cluster with RF=3
- node1(seed), node2, node3
 * Start requests to the cluster with cassandra-stress (it continues
 until the end)
- what we did: cassandra-stress mixed cl=QUORUM duration=10m
 -errors ignore -node node1,node2,node3 -rate threads\>=16
 threads\<=256
 * Stop node3 normally (with systemctl stop)
- the system is still available because the quorum of nodes is
 still available
 * Stop node2 normally (with systemctl stop)
- the system is NOT available after it's stopped.
- the client gets `UnavailableException: Not enough replicas
 available for query at consistency QUORUM`
- the client gets errors right away (so few ms)
- so far it's all expected
 * Wait for 1 mins
 * Bring up node2
- The issue happens here.
- the client gets ReadTimeoutException` or WriteTimeoutException
 depending on if the request is read or write even after the node2 is
 up
- the client gets errors after about 5000ms or 2000ms, which are
 request timeout for write and read request
- what node1 reports with `nodetool status` and what node2 reports
 are not consistent. (node2 thinks node1 is down)
- It takes very long time to recover from its state
 === STEPS TO REPRODUCE ===

 Is it supposed to happen ?
 If we don't start cassandra-stress, it's all fine.

 Some workarounds we found to recover the state are the followings:
 * Restarting node1 and it recovers its state right after it's restarted
 * Setting lower value in dynamic_snitch_reset_interval_in_ms (to 6
 or something)

 I don't think either of them is a really good solution.
 Can anyone explain what is going on and what is the best way to make
 it not happen or recover ?

 Thanks,
 Hiro

 -
 To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Can we set the precision of decimal column

[Alain RODRIGUEZ]

Hello,



My 2 cents? Do not use floats for money, less for billing. It has never
been good for any database due to the representation of that number for the
computer. It's still the case with Cassandra, and it even seems that
internally we have a debate about how we are doing things there:
https://lists.apache.org/thread.html/5eb0ae8391c1e03bf1926cda11c62a9cc076774bfff73c3014d0d3d4@%3Cdev.cassandra.apache.org%3E

I would use an integer type and create an interface that will multiply the
number by 10 000 where storing it and divide it by the same number when
reading it. This way, no imprecise representation of your number for the
machine anymore. I said 1 arbitrarily because I've never seen a system
that would need more, but it can be 100 or 10 depending on how big the
number you work with are and the precision you want to keep.

C*heers,
---
Alain Rodriguez - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com


Le mar. 23 avr. 2019 à 06:04, xiaobo  a écrit :

> Hi,
>
> We usually use decaimal data type for money, can we set decimal columns
> with a specific precision?
>
> Thanks.
>

Backup Restore

[Ivan Junckes Filho]

Hi guys,

I am trying do a bakup and restore script in a simple way. Is there a way I
can do one command to backup and one to restore a backup?

Or the only way is to create snapshots of all the tables and then restore
one by one?

Re: A cluster (RF=3) not recovering after two nodes are stopped

[Ben Slater]

In the absence of anyone else having any bright ideas - it still sounds to
me like the kind of scenario that can occur in a heavily overloaded
cluster. I would try again with a lower load.

What size machines are you using for stress client and the nodes? Are they
all on separate machines?

Cheers
Ben

---


*Ben Slater**Chief Product Officer*



   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.


On Thu, 25 Apr 2019 at 17:26, Hiroyuki Yamada  wrote:

> Hello,
>
> Sorry again.
> We found yet another weird thing in this.
> If we stop nodes with systemctl or just kill (TERM), it causes the problem,
> but if we kill -9, it doesn't cause the problem.
>
> Thanks,
> Hiro
>
> On Wed, Apr 24, 2019 at 11:31 PM Hiroyuki Yamada 
> wrote:
>
>> Sorry, I didn't write the version and the configurations.
>> I've tested with C* 3.11.4, and
>> the configurations are mostly set to default except for the replication
>> factor and listen_address for proper networking.
>>
>> Thanks,
>> Hiro
>>
>> On Wed, Apr 24, 2019 at 5:12 PM Hiroyuki Yamada 
>> wrote:
>>
>>> Hello Ben,
>>>
>>> Thank you for the quick reply.
>>> I haven't tried that case, but it does't recover even if I stopped the
>>> stress.
>>>
>>> Thanks,
>>> Hiro
>>>
>>> On Wed, Apr 24, 2019 at 3:36 PM Ben Slater 
>>> wrote:
>>>
 Is it possible that stress is overloading node 1 so it’s not recovering
 state properly when node 2 comes up? Have you tried running with a lower
 load (say 2 or 3 threads)?

 Cheers
 Ben

 ---


 *Ben Slater*
 *Chief Product Officer*


 
 
 

 Read our latest technical blog posts here
 .

 This email has been sent on behalf of Instaclustr Pty. Limited
 (Australia) and Instaclustr Inc (USA).

 This email and any attachments may contain confidential and legally
 privileged information.  If you are not the intended recipient, do not copy
 or disclose its content, but please reply to this email immediately and
 highlight the error to the sender and then immediately delete the message.


 On Wed, 24 Apr 2019 at 16:28, Hiroyuki Yamada 
 wrote:

> Hello,
>
> I faced a weird issue when recovering a cluster after two nodes are
> stopped.
> It is easily reproduce-able and looks like a bug or an issue to fix,
> so let me write down the steps to reproduce.
>
> === STEPS TO REPRODUCE ===
> * Create a 3-node cluster with RF=3
>- node1(seed), node2, node3
> * Start requests to the cluster with cassandra-stress (it continues
> until the end)
>- what we did: cassandra-stress mixed cl=QUORUM duration=10m
> -errors ignore -node node1,node2,node3 -rate threads\>=16
> threads\<=256
> * Stop node3 normally (with systemctl stop)
>- the system is still available because the quorum of nodes is
> still available
> * Stop node2 normally (with systemctl stop)
>- the system is NOT available after it's stopped.
>- the client gets `UnavailableException: Not enough replicas
> available for query at consistency QUORUM`
>- the client gets errors right away (so few ms)
>- so far it's all expected
> * Wait for 1 mins
> * Bring up node2
>- The issue happens here.
>- the client gets ReadTimeoutException` or WriteTimeoutException
> depending on if the request is read or write even after the node2 is
> up
>- the client gets errors after about 5000ms or 2000ms, which are
> request timeout for write and read request
>- what node1 reports with `nodetool status` and what node2 reports
> are not consistent. (node2 thinks node1 is down)
>- It takes very long time to recover from its state
> === STEPS TO REPRODUCE ===
>
> Is it supposed to happen ?
> If we don't start cassandra-stress, it's all fine.
>
> Some workarounds we found to recover the state are the followings:
> * Restarting node1 and it recovers its state right after it's restarted
> * Setting lower value in dynamic_snitch_reset_interval_in_ms (to 6
> or something)
>
> I don't think either of them is a really good solution.
> Can anyone expl

Re: A cluster (RF=3) not recovering after two nodes are stopped

[Alok Dwivedi]

Could it be related to hinted hand offs being stored in Node1 and then 
attempted to be replayed in Node2 when it comes back causing more load as new 
mutations are also being applied from cassandra-stress at same time?

Alok Dwivedi
Senior Consultant 
https://www.instaclustr.com/




> On 26 Apr 2019, at 09:04, Ben Slater  wrote:
> 
> In the absence of anyone else having any bright ideas - it still sounds to me 
> like the kind of scenario that can occur in a heavily overloaded cluster. I 
> would try again with a lower load. 
> 
> What size machines are you using for stress client and the nodes? Are they 
> all on separate machines?
> 
> Cheers
> Ben
> --- 
> Ben Slater
> Chief Product Officer
> 
>  
>        
> 
> Read our latest technical blog posts here .
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia) 
> and Instaclustr Inc (USA).
> This email and any attachments may contain confidential and legally 
> privileged information.  If you are not the intended recipient, do not copy 
> or disclose its content, but please reply to this email immediately and 
> highlight the error to the sender and then immediately delete the message.
> 
> 
> On Thu, 25 Apr 2019 at 17:26, Hiroyuki Yamada  > wrote:
> Hello,
> 
> Sorry again.
> We found yet another weird thing in this.
> If we stop nodes with systemctl or just kill (TERM), it causes the problem,
> but if we kill -9, it doesn't cause the problem.
> 
> Thanks,
> Hiro
> 
> On Wed, Apr 24, 2019 at 11:31 PM Hiroyuki Yamada  > wrote:
> Sorry, I didn't write the version and the configurations.
> I've tested with C* 3.11.4, and 
> the configurations are mostly set to default except for the replication 
> factor and listen_address for proper networking.
> 
> Thanks,
> Hiro
> 
> On Wed, Apr 24, 2019 at 5:12 PM Hiroyuki Yamada  > wrote:
> Hello Ben,
> 
> Thank you for the quick reply.
> I haven't tried that case, but it does't recover even if I stopped the stress.
> 
> Thanks,
> Hiro
> 
> On Wed, Apr 24, 2019 at 3:36 PM Ben Slater  > wrote:
> Is it possible that stress is overloading node 1 so it’s not recovering state 
> properly when node 2 comes up? Have you tried running with a lower load (say 
> 2 or 3 threads)?
> 
> Cheers
> Ben
> --- 
> Ben Slater
> Chief Product Officer
> 
>        
> 
> Read our latest technical blog posts here .
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia) 
> and Instaclustr Inc (USA).
> This email and any attachments may contain confidential and legally 
> privileged information.  If you are not the intended recipient, do not copy 
> or disclose its content, but please reply to this email immediately and 
> highlight the error to the sender and then immediately delete the message.
> 
> 
> On Wed, 24 Apr 2019 at 16:28, Hiroyuki Yamada  > wrote:
> Hello,
> 
> I faced a weird issue when recovering a cluster after two nodes are stopped.
> It is easily reproduce-able and looks like a bug or an issue to fix,
> so let me write down the steps to reproduce.
> 
> === STEPS TO REPRODUCE ===
> * Create a 3-node cluster with RF=3
>- node1(seed), node2, node3
> * Start requests to the cluster with cassandra-stress (it continues
> until the end)
>- what we did: cassandra-stress mixed cl=QUORUM duration=10m
> -errors ignore -node node1,node2,node3 -rate threads\>=16
> threads\<=256
> * Stop node3 normally (with systemctl stop)
>- the system is still available because the quorum of nodes is
> still available
> * Stop node2 normally (with systemctl stop)
>- the system is NOT available after it's stopped.
>- the client gets `UnavailableException: Not enough replicas
> available for query at consistency QUORUM`
>- the client gets errors right away (so few ms)
>- so far it's all expected
> * Wait for 1 mins
> * Bring up node2
>- The issue happens here.
>- the client gets ReadTimeoutException` or WriteTimeoutException
> depending on if the request is read or write even after the node2 is
> up
>- the client gets errors after about 5000ms or 2000ms, which are
> request timeout for write and read request
>- what node1 reports with `nodetool status` and what node2 reports
> are not consistent. (node2 thinks node1 is down)
>- It takes very long time to recover from its state
> === STEPS TO REPRODUCE ===
> 
> Is it supposed to happen ?
> If we don't start cassandra-stress, it's all fine.
> 
> Some workarounds we found to recover the state are the followings:
> * Restarting node1 and it r

Re: when the "delete statement" would be deleted?

[Alok Dwivedi]

Hi
Just to add to that, this is the way C* handles deletes. Cassandra creates 
delete markers called tombstones on delete requests. They are retained (even 
after compaction) for a period of time configured using gc_grace_seconds 
(default 10 days) to ensure that if a node  was down when delete was issued, 
using this delete marker i.e. tombstone that node node can later be sychronised 
with delete information else the subsequent reads may lead to reading the 
deleted data i.e. ghost records getting reappeared. Here is a good link to 
explain more
https://www.instaclustr.com/support/documentation/cluster-management/managing-tombstones/
 


Alok Dwivedi
Senior Consultant 
https://www.instaclustr.com/




> On 24 Apr 2019, at 20:20, onmstester onmstester  
> wrote:
> 
> Found the answer: it would be deleted after gc_grace
> Just decreased the gc_grace, run compact, and the "marked_deleted" partitions 
> purged from sstable
> 
> Sent using Zoho Mail 
> 
> 
>  On Wed, 24 Apr 2019 14:15:33 +0430 onmstester onmstester 
>  wrote 
> 
> Just deleted multiple partitions from one of my tables, dumping sstables 
> shows that the data successfully deleted, but the 'marked_deleted' rows for 
> each of partitions still exists on sstable and allocates storage. 
> Is there any way to get rid of these delete statements storage overhead 
> (everything be deleted after final compactions, even the delete statements)?
> Sent using Zoho Mail 
> 
> 
> 

Re: Backup Restore

[Naman Gupta]

You should take a look into ssTableLoader cassandra utility,
https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsBulkloader.html

On Fri, Apr 26, 2019 at 1:33 AM Ivan Junckes Filho 
wrote:

> Hi guys,
>
> I am trying do a bakup and restore script in a simple way. Is there a way
> I can do one command to backup and one to restore a backup?
>
> Or the only way is to create snapshots of all the tables and then restore
> one by one?
>

Re: A cluster (RF=3) not recovering after two nodes are stopped

[Hiroyuki Yamada]

Hello,

Thank you for some feedbacks.

>Ben
Thank you.
I've tested with lower concurrency in my side, the issue still occurs.
We are using 3 x T3.xlarge instances for C* and small and separate instance
for the client program.
But if we tried with 1 host with 3 C* nodes, the issue didn't occur.

> Alok
We also thought so and tested with hints disabled, but it doesn't make any
difference. (the issue still occurs)

Thanks,
Hiro




On Fri, Apr 26, 2019 at 8:19 AM Alok Dwivedi 
wrote:

> Could it be related to hinted hand offs being stored in Node1 and then
> attempted to be replayed in Node2 when it comes back causing more load as
> new mutations are also being applied from cassandra-stress at same time?
>
> Alok Dwivedi
> Senior Consultant
> https://www.instaclustr.com/
>
>
>
>
> On 26 Apr 2019, at 09:04, Ben Slater  wrote:
>
> In the absence of anyone else having any bright ideas - it still sounds to
> me like the kind of scenario that can occur in a heavily overloaded
> cluster. I would try again with a lower load.
>
> What size machines are you using for stress client and the nodes? Are they
> all on separate machines?
>
> Cheers
> Ben
>
> ---
>
>
> *Ben Slater**Chief Product Officer*
>
> 
>
>    
>
>
> Read our latest technical blog posts here
> .
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>
> On Thu, 25 Apr 2019 at 17:26, Hiroyuki Yamada  wrote:
>
>> Hello,
>>
>> Sorry again.
>> We found yet another weird thing in this.
>> If we stop nodes with systemctl or just kill (TERM), it causes the
>> problem,
>> but if we kill -9, it doesn't cause the problem.
>>
>> Thanks,
>> Hiro
>>
>> On Wed, Apr 24, 2019 at 11:31 PM Hiroyuki Yamada 
>> wrote:
>>
>>> Sorry, I didn't write the version and the configurations.
>>> I've tested with C* 3.11.4, and
>>> the configurations are mostly set to default except for the replication
>>> factor and listen_address for proper networking.
>>>
>>> Thanks,
>>> Hiro
>>>
>>> On Wed, Apr 24, 2019 at 5:12 PM Hiroyuki Yamada 
>>> wrote:
>>>
 Hello Ben,

 Thank you for the quick reply.
 I haven't tried that case, but it does't recover even if I stopped the
 stress.

 Thanks,
 Hiro

 On Wed, Apr 24, 2019 at 3:36 PM Ben Slater 
 wrote:

> Is it possible that stress is overloading node 1 so it’s not
> recovering state properly when node 2 comes up? Have you tried running 
> with
> a lower load (say 2 or 3 threads)?
>
> Cheers
> Ben
>
> ---
>
>
> *Ben Slater*
> *Chief Product Officer*
>
>
> 
> 
> 
>
> Read our latest technical blog posts here
> .
>
> This email has been sent on behalf of Instaclustr Pty. Limited
> (Australia) and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not 
> copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>
> On Wed, 24 Apr 2019 at 16:28, Hiroyuki Yamada 
> wrote:
>
>> Hello,
>>
>> I faced a weird issue when recovering a cluster after two nodes are
>> stopped.
>> It is easily reproduce-able and looks like a bug or an issue to fix,
>> so let me write down the steps to reproduce.
>>
>> === STEPS TO REPRODUCE ===
>> * Create a 3-node cluster with RF=3
>>- node1(seed), node2, node3
>> * Start requests to the cluster with cassandra-stress (it continues
>> until the end)
>>- what we did: cassandra-stress mixed cl=QUORUM duration=10m
>> -errors ignore -node node1,node2,node3 -rate threads\>=16
>> threads\<=256
>> * Stop node3 normally (with systemctl stop)
>>- the system is still available because the quorum of nodes is
>> still available
>> * Stop node2 normally (with systemctl stop)
>>- the system is NOT available after it's stopped.
>>- the client gets `UnavailableException: Not enough replicas
>> available for query at consistency QUORUM`
>>- the client gets errors right away (so few ms)
>>- so far it's all expected
>> * Wait for 1 min