Re: Hints folder missing in Cassandra

[Kyrylo Lebediev]

Hi,

Not sure what could be the reason, but the issue is obvious: directory 
/var/lib/cassandra/hints is missing or cassandra user doesn't have enough 
permissions on it.


Regards,

Kyrill


From: test user 
Sent: Wednesday, February 7, 2018 8:14:59 AM
To: user@cassandra.apache.org
Subject: Fwd: Hints folder missing in Cassandra


Has anyone run into the issue mentioned below?


-- Forwarded message --
From: test user mailto:casst...@gmail.com>>
Date: Tue, Feb 6, 2018 at 3:28 PM
Subject: Hints folder missing in Cassandra
To: 
user-subscr...@cassandra.apache.org



Hello All,



I am using Cassandra 3.10. I have 3-node cluster (say node0, node1, node2).

I have been using nodetool utility to monitor the cluster.

nodetool status – indicates all nodes are up and in normal state

nodetool ring – indicates all nodes are up



Running into following issue:

WARN  [HintsWriteExecutor:1] 2018-02-05 06:57:09,607 CLibrary.java:295 - 
open(/var/lib/cassandra/hints, O_RDONLY) failed, errno (2).

For some reason, the hints folder on node1 is missing. Not sure how it got 
deleted. One node in the cluster always seems to run into this issue.

The other two nodes node0 and node2 do not have this issue, and the hints 
folder is present.



The debug logs do not seem to indicate the root cause.



Please let me know how to debug this further to understand the root cause.



Regards,

Cassandra User

SSH remote access,permissions issue

[milenko markovic]

My friend has installed 4 nodes at his server in Norway.I wanted to add new 
nodes to cluster.I have logged in with ssh and password.ssh pi3@84.208.89.002 
-p 22022

Last login: Wed Feb  7 06:59:05 2018 from 94.189.152.722

Later sudo su,and have repeated 100% exactly the same operations as he did.When 
I go for
 nodetool statususage: nodetool [(-h  | --host )] [(-p  | 
--port )]        [(-u  | --username )]        [(-pw 
 | --password )]        [(-pwf  | 
--password-file )]         []
The most commonly used nodetool commands are:    assassinate                  
Forcefully remove a dead node without re-replicating any data.  Use as a last 
resort if you cannot removenode    bootstrap                    Monitor/manage 
node's bootstrap process

It happens the same with any other nodetool command.less /etc/passwd
qb3:x:1000:1000:qb3,,,:/home/qb3:/bin/bashsshd:x:121:65534::/var/run/sshd:/usr/sbin/nologinmilenko:x:1001:1001:Milenko,,,:/home/milenko:/bin/bashntp:x:122:129::/home/ntp:/bin/falsecassandra:x:123:130:Cassandra
 database,,,:/var/lib/cassandra:/bin/false
Why? User cassandra shell is not enabled?How to change Cassandra permissions?Is 
this related to remote accesss?He can add nodes without any problems.

Re: SSH remote access,permissions issue

[Nicolas Guyomar]

Hi Milenko,

Can you check the JMX configuration in jvm.options file and make sure you
can login without user/pwd ?
Also, the node might be listening to a specific network interface, can you
output 'netstat -an | grep 7199' for us ?  (assuming your JMX port config
is 7199)


On 7 February 2018 at 11:01, milenko markovic <
milenko...@yahoo.co.uk.invalid> wrote:

> My friend has installed 4 nodes at his server in Norway.I wanted to add
> new nodes to cluster.I have logged in with ssh and password.
> ssh pi3@84.208.89.002 -p 22022
>
> Last login: Wed Feb  7 06:59:05 2018 from 94.189.152.722
>
>
> Later sudo su,and have repeated 100% exactly the same operations as he
> did.When I go for
>  nodetool status
> usage: nodetool [(-h  | --host )] [(-p  | --port )]
> [(-u  | --username )]
> [(-pw  | --password )]
> [(-pwf  | --password-file )]
> 
> []
>
> The most commonly used nodetool commands are:
> assassinate  Forcefully remove a dead node without
> re-replicating any data.  Use as a last resort if you cannot removenode
> bootstrapMonitor/manage node's bootstrap process
>
>
> It happens the same with any other nodetool command.
> less /etc/passwd
>
> qb3:x:1000:1000:qb3,,,:/home/qb3:/bin/bash
> sshd:x:121:65534::/var/run/sshd:/usr/sbin/nologin
> milenko:x:1001:1001:Milenko,,,:/home/milenko:/bin/bash
> ntp:x:122:129::/home/ntp:/bin/false
> cassandra:x:123:130:Cassandra database,,,:/var/lib/cassandra:/bin/false
>
> Why?
> User cassandra shell is not enabled?
> How to change Cassandra permissions?Is this related to remote accesss?He
> can add nodes without any problems.
>
>

Re: SSH remote access,permissions issue

[Matija Gobec]

Hi Milenko,

There is a high chance that JMX is listening on the instance IP instead of
locahost which is default when you run nodetool command.
Try executing nodetool with -h nodeip or check netstat as Nicolas mentioned.
If you run nodetool status on one of the existing nodes can you see that
new node joined in?
What does the log say?

Side note:
I hope thats not a public IP of that instance

Best,
Matija

On Wed, Feb 7, 2018 at 11:18 AM, Nicolas Guyomar 
wrote:

> Hi Milenko,
>
> Can you check the JMX configuration in jvm.options file and make sure you
> can login without user/pwd ?
> Also, the node might be listening to a specific network interface, can you
> output 'netstat -an | grep 7199' for us ?  (assuming your JMX port config
> is 7199)
>
>
> On 7 February 2018 at 11:01, milenko markovic  invalid> wrote:
>
>> My friend has installed 4 nodes at his server in Norway.I wanted to add
>> new nodes to cluster.I have logged in with ssh and password.
>> ssh pi3@84.208.89.002 -p 22022
>>
>> Last login: Wed Feb  7 06:59:05 2018 from 94.189.152.722
>>
>>
>> Later sudo su,and have repeated 100% exactly the same operations as he
>> did.When I go for
>>  nodetool status
>> usage: nodetool [(-h  | --host )] [(-p  | --port
>> )]
>> [(-u  | --username )]
>> [(-pw  | --password )]
>> [(-pwf  | --password-file )]
>> 
>> []
>>
>> The most commonly used nodetool commands are:
>> assassinate  Forcefully remove a dead node without
>> re-replicating any data.  Use as a last resort if you cannot removenode
>> bootstrapMonitor/manage node's bootstrap process
>>
>>
>> It happens the same with any other nodetool command.
>> less /etc/passwd
>>
>> qb3:x:1000:1000:qb3,,,:/home/qb3:/bin/bash
>> sshd:x:121:65534::/var/run/sshd:/usr/sbin/nologin
>> milenko:x:1001:1001:Milenko,,,:/home/milenko:/bin/bash
>> ntp:x:122:129::/home/ntp:/bin/false
>> cassandra:x:123:130:Cassandra database,,,:/var/lib/cassandra:/bin/false
>>
>> Why?
>> User cassandra shell is not enabled?
>> How to change Cassandra permissions?Is this related to remote accesss?He
>> can add nodes without any problems.
>>
>>
>

Re: SSH remote access,permissions issue

[Dimitry Lvovsky]

Milenko,
Check your friends env variables, perhaps he has something set (port, etc)
that you don't.

Cheers.

On Wed, Feb 7, 2018 at 11:29 AM, Matija Gobec  wrote:

> Hi Milenko,
>
> There is a high chance that JMX is listening on the instance IP instead of
> locahost which is default when you run nodetool command.
> Try executing nodetool with -h nodeip or check netstat as Nicolas
> mentioned.
> If you run nodetool status on one of the existing nodes can you see that
> new node joined in?
> What does the log say?
>
> Side note:
> I hope thats not a public IP of that instance
>
> Best,
> Matija
>
> On Wed, Feb 7, 2018 at 11:18 AM, Nicolas Guyomar <
> nicolas.guyo...@gmail.com> wrote:
>
>> Hi Milenko,
>>
>> Can you check the JMX configuration in jvm.options file and make sure you
>> can login without user/pwd ?
>> Also, the node might be listening to a specific network interface, can
>> you output 'netstat -an | grep 7199' for us ?  (assuming your JMX port
>> config is 7199)
>>
>>
>> On 7 February 2018 at 11:01, milenko markovic <
>> milenko...@yahoo.co.uk.invalid> wrote:
>>
>>> My friend has installed 4 nodes at his server in Norway.I wanted to add
>>> new nodes to cluster.I have logged in with ssh and password.
>>> ssh pi3@84.208.89.002 -p 22022
>>>
>>> Last login: Wed Feb  7 06:59:05 2018 from 94.189.152.722
>>>
>>>
>>> Later sudo su,and have repeated 100% exactly the same operations as he
>>> did.When I go for
>>>  nodetool status
>>> usage: nodetool [(-h  | --host )] [(-p  | --port
>>> )]
>>> [(-u  | --username )]
>>> [(-pw  | --password )]
>>> [(-pwf  | --password-file )]
>>> 
>>> []
>>>
>>> The most commonly used nodetool commands are:
>>> assassinate  Forcefully remove a dead node without
>>> re-replicating any data.  Use as a last resort if you cannot removenode
>>> bootstrapMonitor/manage node's bootstrap process
>>>
>>>
>>> It happens the same with any other nodetool command.
>>> less /etc/passwd
>>>
>>> qb3:x:1000:1000:qb3,,,:/home/qb3:/bin/bash
>>> sshd:x:121:65534::/var/run/sshd:/usr/sbin/nologin
>>> milenko:x:1001:1001:Milenko,,,:/home/milenko:/bin/bash
>>> ntp:x:122:129::/home/ntp:/bin/false
>>> cassandra:x:123:130:Cassandra database,,,:/var/lib/cassandra:/bin/false
>>>
>>> Why?
>>> User cassandra shell is not enabled?
>>> How to change Cassandra permissions?Is this related to remote accesss?He
>>> can add nodes without any problems.
>>>
>>>
>>
>


-- 


*VP of Data and Architecture
dimi...@reviewpro.com
*Tel: *+34 66 134 05 26

*ReviewPro* | Guest Intelligence | *www.reviewpro.com
* | *reviewpro.com/blog
*
Follow us on Twitter: *@reviewpro.com *

Re: SSH remote access,permissions issue

[Nicolas Guyomar]

Milenko, please reply to user@cassandra.apache.org not directly to me  ;)


You do not seem to have a cassandra process up and running, can you check
with your friend what he did please ? Maybe something is going on on this
cluster, do not try to start the node he might be working on it :)



On 7 February 2018 at 11:40, milenko markovic 
wrote:

> qb3@qb3-Latitude-E6430-3:~$ cassandra -v
> 3.11.1
> qb3@qb3-Latitude-E6430-3:~$ ps -ef | grep cassandra
> qb3   7017  6859  0 11:40 pts/200:00:00 grep --color=auto cassandra
>
>
>
> On Wednesday, 7 February 2018, 11:34:28 CET, Nicolas Guyomar <
> nicolas.guyo...@gmail.com> wrote:
>
>
> I'm sorry Milenko but I do not understand what you are trying to do.
>
> Are you connected with ssh on a Cassandra node ?
> If so can you check that the process is running ?  (ps -ef | grep
> cassandra)  ?
>
> On 7 February 2018 at 11:31, milenko markovic 
> wrote:
>
> Hi Nicolas
>
> Where is jvm.options file?I have only(after rm -rf
> /var/lib/cassandra/data/ system/*)
>
> /etc/cassandra$ ls
> cassandra.yaml  triggers
>
> netstat -an | grep 7199
>
> does not print anything
>
> Regards
> On Wednesday, 7 February 2018, 11:19:07 CET, Nicolas Guyomar <
> nicolas.guyo...@gmail.com> wrote:
>
>
> Hi Milenko,
>
> Can you check the JMX configuration in jvm.options file and make sure you
> can login without user/pwd ?
> Also, the node might be listening to a specific network interface, can you
> output 'netstat -an | grep 7199' for us ?  (assuming your JMX port config
> is 7199)
>
>
> On 7 February 2018 at 11:01, milenko markovic  invalid > wrote:
>
> My friend has installed 4 nodes at his server in Norway.I wanted to add
> new nodes to cluster.I have logged in with ssh and password.
> ssh pi3@84.208.89.002 -p 22022
>
> Last login: Wed Feb  7 06:59:05 2018 from 94.189.152.722
>
>
> Later sudo su,and have repeated 100% exactly the same operations as he
> did.When I go for
>  nodetool status
> usage: nodetool [(-h  | --host )] [(-p  | --port )]
> [(-u  | --username )]
> [(-pw  | --password )]
> [(-pwf  | --password-file )]
> 
> []
>
> The most commonly used nodetool commands are:
> assassinate  Forcefully remove a dead node without
> re-replicating any data.  Use as a last resort if you cannot removenode
> bootstrapMonitor/manage node's bootstrap process
>
>
> It happens the same with any other nodetool command.
> less /etc/passwd
>
> qb3:x:1000:1000:qb3,,,:/home/ qb3:/bin/bash
> sshd:x:121:65534::/var/run/ sshd:/usr/sbin/nologin
> milenko:x:1001:1001:Milenko,,, :/home/milenko:/bin/bash
> ntp:x:122:129::/home/ntp:/bin/ false
> cassandra:x:123:130:Cassandra database,,,:/var/lib/ cassandra:/bin/false
>
> Why?
> User cassandra shell is not enabled?
> How to change Cassandra permissions?Is this related to remote accesss?He
> can add nodes without any problems.
>
>
>
>

Bootstrapping fails with < 128GB RAM ...

[Jürgen Albersdorfer]

Hi, I always face an issue when bootstrapping a Node having less than 184GB
RAM (156GB JVM HEAP) on our 10 Node C* 3.11.1 Cluster.
During bootstrap, when I watch the cassandra.log I observe a growth in JVM
Heap Old Gen which gets not significantly freed any more.
I know that JVM collects on Old Gen only when really needed. I can see
collections, but there is always a remainder which
seems to grow forever without ever getting freed.
After the Node successfully Joined the Cluster, I can remove the extra
128GB of RAM I have given it for bootstrapping without any further effect.

It feels like Cassandra will not forget about every single byte streamed
over the Network over time during bootstrapping, - which would be a memory
leak and a major problem, too.

Or is there something I'm doing wrong? - Any Ideas?

Here my observations on a failing Bootstrap - The following Node has 72GB
RAM installed, 64GB of it are configured for JVM Heap Space.

cassandra.log (truncated):
INFO  [Service Thread] 2018-02-07 11:12:49,604 GCInspector.java:284 - G1
Young Generation GC in 984ms.  G1 Eden Space: 14763950080 -> 0; G1 Old Gen:
36960206856 -> 39661338640; G1 Survivor Space: 2785017856 -> 1476395008;
INFO  [Service Thread] 2018-02-07 11:13:00,108 GCInspector.java:284 - G1
Young Generation GC in 784ms.  G1 Eden Space: 18387828736 -> 0; G1 Old Gen:
39661338640 -> 41053847560; G1 Survivor Space: 1476395008 -> 1845493760;
INFO  [Service Thread] 2018-02-07 11:13:08,639 GCInspector.java:284 - G1
Young Generation GC in 718ms.  G1 Eden Space: 16743661568 -> 0; G1 Old Gen:
41053847560 -> 42832232472; G1 Survivor Space: 1845493760 -> 1375731712;
INFO  [Service Thread] 2018-02-07 11:13:18,271 GCInspector.java:284 - G1
Young Generation GC in 546ms.  G1 Eden Space: 15535702016 -> 0; G1 Old Gen:
42831004832 -> 44206736544; G1 Survivor Space: 1375731712 -> 1006632960;
INFO  [Service Thread] 2018-02-07 11:13:35,364 GCInspector.java:284 - G1
Young Generation GC in 638ms.  G1 Eden Space: 14025752576 -> 0; G1 Old Gen:
44206737048 -> 45213369488; G1 Survivor Space: 1778384896 -> 1610612736;
INFO  [Service Thread] 2018-02-07 11:13:42,898 GCInspector.java:284 - G1
Young Generation GC in 614ms.  G1 Eden Space: 13388218368 -> 0; G1 Old Gen:
45213369488 -> 46152893584; G1 Survivor Space: 1610612736 -> 1006632960;
INFO  [Service Thread] 2018-02-07 11:13:58,291 GCInspector.java:284 - G1
Young Generation GC in 400ms.  G1 Eden Space: 13119782912 -> 0; G1 Old Gen:
46136116376 -> 47171400848; G1 Survivor Space: 1275068416 -> 771751936;
INFO  [Service Thread] 2018-02-07 11:14:23,071 GCInspector.java:284 - G1
Young Generation GC in 303ms.  G1 Eden Space: 11676942336 -> 0; G1 Old Gen:
47710958232 -> 48239699096; G1 Survivor Space: 1207959552 -> 973078528;
INFO  [Service Thread] 2018-02-07 11:14:46,157 GCInspector.java:284 - G1
Young Generation GC in 305ms.  G1 Eden Space: 11005853696 -> 0; G1 Old Gen:
48903342232 -> 49289001104; G1 Survivor Space: 939524096 -> 973078528;
INFO  [Service Thread] 2018-02-07 11:14:53,045 GCInspector.java:284 - G1
Young Generation GC in 380ms.  G1 Eden Space: 10569646080 -> 0; G1 Old Gen:
49289001104 -> 49586732696; G1 Survivor Space: 973078528 -> 1308622848;
INFO  [Service Thread] 2018-02-07 11:15:04,692 GCInspector.java:284 - G1
Young Generation GC in 360ms.  G1 Eden Space: 9294577664 -> 0; G1 Old Gen:
50671712912 -> 51269944472; G1 Survivor Space: 905969664 -> 805306368;
WARN  [Service Thread] 2018-02-07 11:15:07,317 GCInspector.java:282 - G1
Young Generation GC in 1102ms.  G1 Eden Space: 2617245696 -> 0; G1 Old Gen:
51269944472 -> 47310521496; G1 Survivor Space: 805306368 -> 301989888;

INFO  [Service Thread] 2018-02-07 11:16:36,535 GCInspector.java:284 - G1
Young Generation GC in 377ms.  G1 Eden Space: 7683964928 -> 0; G1 Old Gen:
51958433432 -> 52658554008; G1 Survivor Space: 1073741824 -> 1040187392;
INFO  [Service Thread] 2018-02-07 11:16:41,756 GCInspector.java:284 - G1
Young Generation GC in 340ms.  G1 Eden Space: 7046430720 -> 0; G1 Old Gen:
52624999576 -> 53299987616; G1 Survivor Space: 1040187392 -> 805306368;
WARN  [Service Thread] 2018-02-07 11:16:44,087 GCInspector.java:282 - G1
Young Generation GC in 1005ms.  G1 Eden Space: 2617245696 -> 0; G1 Old Gen:
53299987616 -> 49659331752; G1 Survivor Space: 805306368 -> 436207616;
...
INFO  [Service Thread] 2018-02-07 11:25:40,902 GCInspector.java:284 - G1
Young Generation GC in 254ms.  G1 Eden Space: 11475615744 -> 0; G1 Old Gen:
48904357040 -> 48904357544; G1 Survivor Space: 704643072 -> 805306368;
INFO  [Service Thread] 2018-02-07 11:26:11,424 GCInspector.java:284 - G1
Young Generation GC in 202ms.  G1 Eden Space: 11005853696 -> 0; G1 Old Gen:
48904357544 -> 49321014960; G1 Survivor Space: 939524096 -> 536870912;
WARN  [Service Thread] 2018-02-07 11:26:44,484 GCInspector.java:282 - G1
Young Generation GC in 1295ms.  G1 Eden Space: 2617245696 -> 0; G1 Old Gen:
49321014960 -> 46255753384; G1 Survivor Space: 805306368 -> 402653184;
...
INFO  [Service Thread] 2018-02-0

Re: Bootstrapping fails with < 128GB RAM ...

[Nicolas Guyomar]

Hi Jurgen,

It does feel like some OOM during boostrap from previous C* v2, but that
sould be fixed in your version.

Do you know how many sstables is this new node suppose to receive ?

Juste a wild guess : it may have something to do with compaction not
keeping up because every other nodes are streaming data to this new one
(resulting in long lived object in the heap), does disabling
compaction_throughput_mb_per_sec or increasing concurrent_compactors has
any effect ?

Which memtable_allocation_type are you using ?


On 7 February 2018 at 12:38, Jürgen Albersdorfer 
wrote:

> Hi, I always face an issue when bootstrapping a Node having less than
> 184GB RAM (156GB JVM HEAP) on our 10 Node C* 3.11.1 Cluster.
> During bootstrap, when I watch the cassandra.log I observe a growth in JVM
> Heap Old Gen which gets not significantly freed any more.
> I know that JVM collects on Old Gen only when really needed. I can see
> collections, but there is always a remainder which
> seems to grow forever without ever getting freed.
> After the Node successfully Joined the Cluster, I can remove the extra
> 128GB of RAM I have given it for bootstrapping without any further effect.
>
> It feels like Cassandra will not forget about every single byte streamed
> over the Network over time during bootstrapping, - which would be a memory
> leak and a major problem, too.
>
> Or is there something I'm doing wrong? - Any Ideas?
>
> Here my observations on a failing Bootstrap - The following Node has 72GB
> RAM installed, 64GB of it are configured for JVM Heap Space.
>
> cassandra.log (truncated):
> INFO  [Service Thread] 2018-02-07 11:12:49,604 GCInspector.java:284 - G1
> Young Generation GC in 984ms.  G1 Eden Space: 14763950080 -> 0; G1 Old Gen:
> 36960206856 -> 39661338640; G1 Survivor Space: 2785017856 -> 1476395008;
> INFO  [Service Thread] 2018-02-07 11:13:00,108 GCInspector.java:284 - G1
> Young Generation GC in 784ms.  G1 Eden Space: 18387828736 -> 0; G1 Old Gen:
> 39661338640 -> 41053847560; G1 Survivor Space: 1476395008 -> 1845493760;
> INFO  [Service Thread] 2018-02-07 11:13:08,639 GCInspector.java:284 - G1
> Young Generation GC in 718ms.  G1 Eden Space: 16743661568 -> 0; G1 Old Gen:
> 41053847560 -> 42832232472; G1 Survivor Space: 1845493760 -> 1375731712;
> INFO  [Service Thread] 2018-02-07 11:13:18,271 GCInspector.java:284 - G1
> Young Generation GC in 546ms.  G1 Eden Space: 15535702016 -> 0; G1 Old Gen:
> 42831004832 -> 44206736544; G1 Survivor Space: 1375731712 -> 1006632960;
> INFO  [Service Thread] 2018-02-07 11:13:35,364 GCInspector.java:284 - G1
> Young Generation GC in 638ms.  G1 Eden Space: 14025752576 -> 0; G1 Old Gen:
> 44206737048 -> 45213369488; G1 Survivor Space: 1778384896 -> 1610612736;
> INFO  [Service Thread] 2018-02-07 11:13:42,898 GCInspector.java:284 - G1
> Young Generation GC in 614ms.  G1 Eden Space: 13388218368 -> 0; G1 Old Gen:
> 45213369488 -> 46152893584; G1 Survivor Space: 1610612736 -> 1006632960;
> INFO  [Service Thread] 2018-02-07 11:13:58,291 GCInspector.java:284 - G1
> Young Generation GC in 400ms.  G1 Eden Space: 13119782912 -> 0; G1 Old Gen:
> 46136116376 -> 47171400848; G1 Survivor Space: 1275068416 -> 771751936;
> INFO  [Service Thread] 2018-02-07 11:14:23,071 GCInspector.java:284 - G1
> Young Generation GC in 303ms.  G1 Eden Space: 11676942336 -> 0; G1 Old Gen:
> 47710958232 -> 48239699096; G1 Survivor Space: 1207959552 -> 973078528;
> INFO  [Service Thread] 2018-02-07 11:14:46,157 GCInspector.java:284 - G1
> Young Generation GC in 305ms.  G1 Eden Space: 11005853696 -> 0; G1 Old Gen:
> 48903342232 -> 49289001104; G1 Survivor Space: 939524096 -> 973078528;
> INFO  [Service Thread] 2018-02-07 11:14:53,045 GCInspector.java:284 - G1
> Young Generation GC in 380ms.  G1 Eden Space: 10569646080 -> 0; G1 Old Gen:
> 49289001104 -> 49586732696; G1 Survivor Space: 973078528 -> 1308622848;
> INFO  [Service Thread] 2018-02-07 11:15:04,692 GCInspector.java:284 - G1
> Young Generation GC in 360ms.  G1 Eden Space: 9294577664 -> 0; G1 Old Gen:
> 50671712912 -> 51269944472; G1 Survivor Space: 905969664 -> 805306368;
> WARN  [Service Thread] 2018-02-07 11:15:07,317 GCInspector.java:282 - G1
> Young Generation GC in 1102ms.  G1 Eden Space: 2617245696 -> 0; G1 Old
> Gen: 51269944472 -> 47310521496; G1 Survivor Space: 805306368 ->
> 301989888;
> 
> INFO  [Service Thread] 2018-02-07 11:16:36,535 GCInspector.java:284 - G1
> Young Generation GC in 377ms.  G1 Eden Space: 7683964928 -> 0; G1 Old Gen:
> 51958433432 -> 52658554008; G1 Survivor Space: 1073741824 -> 1040187392;
> INFO  [Service Thread] 2018-02-07 11:16:41,756 GCInspector.java:284 - G1
> Young Generation GC in 340ms.  G1 Eden Space: 7046430720 -> 0; G1 Old Gen:
> 52624999576 -> 53299987616; G1 Survivor Space: 1040187392 -> 805306368;
> WARN  [Service Thread] 2018-02-07 11:16:44,087 GCInspector.java:282 - G1
> Young Generation GC in 1005ms.  G1 Eden Space: 2617245696 -> 0; G1 Old
> Gen: 53299987616 -> 49659331752; G1 Survivor Space: 805306368 -

Re: Bootstrapping fails with < 128GB RAM ...

[Jürgen Albersdorfer]

Hi Nicolas,

Do you know how many sstables is this new node suppose to receive ?


If I can find out this via nodetool netstats, then this would be 619 as
following:

# nodetool netstats
Bootstrap b95371e0-0c0a-11e8-932b-f775227bf21c
/192.168.1.215 - Receiving 71 files, 7744612158 bytes total. Already
received 0 files, 893897583 bytes total
/192.168.1.214 - Receiving 58 files, 5693392001 bytes total. Already
received 0 files, 1078372756 bytes total
/192.168.1.206 - Receiving 52 files, 3389096409 bytes total. Already
received 3 files, 508592758 bytes total
/192.168.1.213 - Receiving 59 files, 6041633329 bytes total. Already
received 0 files, 1038760653 bytes total
/192.168.1.231 - Receiving 79 files, 7579181689 bytes total. Already
received 4 files, 38387859 bytes total
/192.168.1.208 - Receiving 51 files, 3272885123 bytes total. Already
received 3 files, 362450903 bytes total
/192.168.1.207 - Receiving 56 files, 3028344200 bytes total. Already
received 3 files, 57790197 bytes total
/192.168.1.232 - Receiving 79 files, 7268716317 bytes total. Already
received 1 files, 1127174421 bytes total
/192.168.1.209 - Receiving 114 files, 21381846105 bytes total. Already
received 1 files, 961497222 bytes total


does disabling compaction_throughput_mb_per_sec or increasing
concurrent_compactors
> has any effect ?


I will give it a try:

# nodetool getcompactionthroughput
Current compaction throughput: 128 MB/s

# nodetool setcompactionthroughput 0

# nodetool getcompactionthroughput
Current compaction throughput: 0 MB/s

# nodetool getconcurrentcompactors
Current concurrent compactors in the system is:
16


Which memtable_allocation_type are you using ?


# grep memtable_allocation_type /etc/cassandra/conf/cassandra.yaml
memtable_allocation_type: heap_buffers


thanks so far, regards
Juergen

2018-02-07 14:29 GMT+01:00 Nicolas Guyomar :

> Hi Jurgen,
>
> It does feel like some OOM during boostrap from previous C* v2, but that
> sould be fixed in your version.
>
> Do you know how many sstables is this new node suppose to receive ?
>
> Juste a wild guess : it may have something to do with compaction not
> keeping up because every other nodes are streaming data to this new one
> (resulting in long lived object in the heap), does disabling
> compaction_throughput_mb_per_sec or increasing concurrent_compactors has
> any effect ?
>
> Which memtable_allocation_type are you using ?
>
>
> On 7 February 2018 at 12:38, Jürgen Albersdorfer 
> wrote:
>
>> Hi, I always face an issue when bootstrapping a Node having less than
>> 184GB RAM (156GB JVM HEAP) on our 10 Node C* 3.11.1 Cluster.
>> During bootstrap, when I watch the cassandra.log I observe a growth in
>> JVM Heap Old Gen which gets not significantly freed any more.
>> I know that JVM collects on Old Gen only when really needed. I can see
>> collections, but there is always a remainder which
>> seems to grow forever without ever getting freed.
>> After the Node successfully Joined the Cluster, I can remove the extra
>> 128GB of RAM I have given it for bootstrapping without any further effect.
>>
>> It feels like Cassandra will not forget about every single byte streamed
>> over the Network over time during bootstrapping, - which would be a memory
>> leak and a major problem, too.
>>
>> Or is there something I'm doing wrong? - Any Ideas?
>>
>> Here my observations on a failing Bootstrap - The following Node has 72GB
>> RAM installed, 64GB of it are configured for JVM Heap Space.
>>
>> cassandra.log (truncated):
>> INFO  [Service Thread] 2018-02-07 11:12:49,604 GCInspector.java:284 - G1
>> Young Generation GC in 984ms.  G1 Eden Space: 14763950080 -> 0; G1 Old Gen:
>> 36960206856 -> 39661338640; G1 Survivor Space: 2785017856 -> 1476395008;
>> INFO  [Service Thread] 2018-02-07 11:13:00,108 GCInspector.java:284 - G1
>> Young Generation GC in 784ms.  G1 Eden Space: 18387828736
>> <(838)%20782-8736> -> 0; G1 Old Gen: 39661338640 -> 41053847560; G1
>> Survivor Space: 1476395008 -> 1845493760;
>> INFO  [Service Thread] 2018-02-07 11:13:08,639 GCInspector.java:284 - G1
>> Young Generation GC in 718ms.  G1 Eden Space: 16743661568 -> 0; G1 Old Gen:
>> 41053847560 -> 42832232472; G1 Survivor Space: 1845493760 -> 1375731712;
>> INFO  [Service Thread] 2018-02-07 11:13:18,271 GCInspector.java:284 - G1
>> Young Generation GC in 546ms.  G1 Eden Space: 15535702016 -> 0; G1 Old Gen:
>> 42831004832 -> 44206736544; G1 Survivor Space: 1375731712 -> 1006632960;
>> INFO  [Service Thread] 2018-02-07 11:13:35,364 GCInspector.java:284 - G1
>> Young Generation GC in 638ms.  G1 Eden Space: 14025752576
>> <(402)%20575-2576> -> 0; G1 Old Gen: 44206737048 -> 45213369488; G1
>> Survivor Space: 1778384896 -> 1610612736;
>> INFO  [Service Thread] 2018-02-07 11:13:42,898 GCInspector.java:284 - G1
>> Young Generation GC in 614ms.  G1 Eden Space: 13388218368 -> 0; G1 Old Gen:
>> 45213369488 -> 46152893584; G1 Survivor Space: 1610612736 -> 1006632960;
>> INFO  [Service Thread]

Re: Bootstrapping fails with < 128GB RAM ...

[Nicolas Guyomar]

Ok then, following up on the wild guess : because you have quite a lot of
concurrent compactors, maybe it is too much concurrent compactions for the
jvm to deal with (taking into account that your load average of 106 seems
really high IMHO)

55Gb of data is not that much, you can try to reduce those concurrent
compactor to make sure your box is not under too much stress (how many
compaction do you have in parallel during boostrap ? )

In the end, it does seem that you're gonna have to share some heap dump for
further investigation (sorry I'm not gonna help lot on this matter)

On 7 February 2018 at 14:43, Jürgen Albersdorfer 
wrote:

> Hi Nicolas,
>
> Do you know how many sstables is this new node suppose to receive ?
>
>
> If I can find out this via nodetool netstats, then this would be 619 as
> following:
>
> # nodetool netstats
> Bootstrap b95371e0-0c0a-11e8-932b-f775227bf21c
> /192.168.1.215 - Receiving 71 files, 7744612158 bytes total. Already
> received 0 files, 893897583 bytes total
> /192.168.1.214 - Receiving 58 files, 5693392001 bytes total. Already
> received 0 files, 1078372756 bytes total
> /192.168.1.206 - Receiving 52 files, 3389096409 bytes total. Already
> received 3 files, 508592758 bytes total
> /192.168.1.213 - Receiving 59 files, 6041633329 bytes total. Already
> received 0 files, 1038760653 bytes total
> /192.168.1.231 - Receiving 79 files, 7579181689 bytes total. Already
> received 4 files, 38387859 bytes total
> /192.168.1.208 - Receiving 51 files, 3272885123 bytes total. Already
> received 3 files, 362450903 bytes total
> /192.168.1.207 - Receiving 56 files, 3028344200 bytes total. Already
> received 3 files, 57790197 bytes total
> /192.168.1.232 - Receiving 79 files, 7268716317 bytes total. Already
> received 1 files, 1127174421 bytes total
> /192.168.1.209 - Receiving 114 files, 21381846105 bytes total.
> Already received 1 files, 961497222 bytes total
>
>
> does disabling compaction_throughput_mb_per_sec or increasing concurrent_
>> compactors has any effect ?
>
>
> I will give it a try:
>
> # nodetool getcompactionthroughput
> Current compaction throughput: 128 MB/s
>
> # nodetool setcompactionthroughput 0
>
> # nodetool getcompactionthroughput
> Current compaction throughput: 0 MB/s
>
> # nodetool getconcurrentcompactors
> Current concurrent compactors in the system is:
> 16
>
>
> Which memtable_allocation_type are you using ?
>
>
> # grep memtable_allocation_type /etc/cassandra/conf/cassandra.yaml
> memtable_allocation_type: heap_buffers
>
>
> thanks so far, regards
> Juergen
>
> 2018-02-07 14:29 GMT+01:00 Nicolas Guyomar :
>
>> Hi Jurgen,
>>
>> It does feel like some OOM during boostrap from previous C* v2, but that
>> sould be fixed in your version.
>>
>> Do you know how many sstables is this new node suppose to receive ?
>>
>> Juste a wild guess : it may have something to do with compaction not
>> keeping up because every other nodes are streaming data to this new one
>> (resulting in long lived object in the heap), does disabling
>> compaction_throughput_mb_per_sec or increasing concurrent_compactors has
>> any effect ?
>>
>> Which memtable_allocation_type are you using ?
>>
>>
>> On 7 February 2018 at 12:38, Jürgen Albersdorfer > > wrote:
>>
>>> Hi, I always face an issue when bootstrapping a Node having less than
>>> 184GB RAM (156GB JVM HEAP) on our 10 Node C* 3.11.1 Cluster.
>>> During bootstrap, when I watch the cassandra.log I observe a growth in
>>> JVM Heap Old Gen which gets not significantly freed any more.
>>> I know that JVM collects on Old Gen only when really needed. I can see
>>> collections, but there is always a remainder which
>>> seems to grow forever without ever getting freed.
>>> After the Node successfully Joined the Cluster, I can remove the extra
>>> 128GB of RAM I have given it for bootstrapping without any further effect.
>>>
>>> It feels like Cassandra will not forget about every single byte streamed
>>> over the Network over time during bootstrapping, - which would be a memory
>>> leak and a major problem, too.
>>>
>>> Or is there something I'm doing wrong? - Any Ideas?
>>>
>>> Here my observations on a failing Bootstrap - The following Node has
>>> 72GB RAM installed, 64GB of it are configured for JVM Heap Space.
>>>
>>> cassandra.log (truncated):
>>> INFO  [Service Thread] 2018-02-07 11:12:49,604 GCInspector.java:284 - G1
>>> Young Generation GC in 984ms.  G1 Eden Space: 14763950080 -> 0; G1 Old Gen:
>>> 36960206856 -> 39661338640; G1 Survivor Space: 2785017856 -> 1476395008;
>>> INFO  [Service Thread] 2018-02-07 11:13:00,108 GCInspector.java:284 - G1
>>> Young Generation GC in 784ms.  G1 Eden Space: 18387828736
>>> <(838)%20782-8736> -> 0; G1 Old Gen: 39661338640 -> 41053847560; G1
>>> Survivor Space: 1476395008 -> 1845493760;
>>> INFO  [Service Thread] 2018-02-07 11:13:08,639 GCInspector.java:284 - G1
>>> Young Generation GC in 718ms.  G1 Eden Space: 16743661568 -> 0; G1 Old Gen:
>>> 41053847560 -> 42832

Re: node restart causes application latency

[Mike Torra]

Thanks for the feedback guys. That example data model was indeed
abbreviated - the real queries have the partition key in them. I am using
RF 3 on the keyspace, so I don't think a node being down would mean the key
I'm looking for would be unavailable. The load balancing policy of the
driver seems correct (
https://docs.datastax.com/en/developer/nodejs-driver/3.4/features/tuning-policies/#load-balancing-policy,
and I am using the default `TokenAware` policy with
`DCAwareRoundRobinPolicy` as a child), but I will look more closely at the
implementation.

It was an oversight of mine to not include `nodetool disablebinary`, but I
still experience the same issue with that.

One other thing I've noticed is that after restarting a node and seeing
application latency, I also see that the node I just restarted sees many
other nodes in the same DC as being down (ie status 'DN'). However,
checking `nodetool status` on those other nodes shows all nodes as
up/normal. To me this could kind of explain the problem - node comes back
online, thinks it is healthy but many others are not, so it gets traffic
from the client application. But then it gets requests for ranges that
belong to a node it thinks is down, so it responds with an error. The
latency issue seems to start roughly when the node goes down, but persists
long (ie 15-20 mins) after it is back online and accepting connections. It
seems to go away once the bounced node shows the other nodes in the same DC
as up again.

As for speculative retry, my CF is using the default of '99th percentile'.
I could try something different there, but nodes being seen as down seems
like an issue.

On Tue, Feb 6, 2018 at 6:26 PM, Jeff Jirsa  wrote:

> Unless you abbreviated, your data model is questionable (SELECT without
> any equality in the WHERE clause on the partition key will always cause a
> range scan, which is super inefficient). Since you're doing LOCAL_ONE and a
> range scan, timeouts sorta make sense - the owner of at least one range
> would be down for a bit.
>
> If you actually have a partition key in your where clause, then the next
> most likely guess is your clients aren't smart enough to route around the
> node as it restarts, or your key cache is getting cold during the bounce.
> Double check your driver's load balancing policy.
>
> It's also likely the case that speculative retry may help other nodes
> route around the bouncing instance better - if you're not using it, you
> probably should be (though with CL: LOCAL_ONE, it seems like it'd be less
> of an issue).
>
> We need to make bouncing nodes easier (or rather, we need to make drain do
> the right thing), but in this case, your data model looks like the biggest
> culprit (unless it's an incomplete recreation).
>
> - Jeff
>
>
> On Tue, Feb 6, 2018 at 10:58 AM, Mike Torra  wrote:
>
>> Hi -
>>
>> I am running a 29 node cluster spread over 4 DC's in EC2, using C* 3.11.1
>> on Ubuntu. Occasionally I have the need to restart nodes in the cluster,
>> but every time I do, I see errors and application (nodejs) timeouts.
>>
>> I restart a node like this:
>>
>> nodetool disablethrift && nodetool disablegossip && nodetool drain
>> sudo service cassandra restart
>>
>> When I do that, I very often get timeouts and errors like this in my
>> nodejs app:
>>
>> Error: Cannot achieve consistency level LOCAL_ONE
>>
>> My queries are all pretty much the same, things like: "select * from
>> history where ts > {current_time}"
>>
>> The errors and timeouts seem to go away on their own after a while, but
>> it is frustrating because I can't track down what I am doing wrong!
>>
>> I've tried waiting between steps of shutting down cassandra, and I've
>> tried stopping, waiting, then starting the node. One thing I've noticed is
>> that even after `nodetool drain`ing the node, there are open connections to
>> other nodes in the cluster (ie looking at the output of netstat) until I
>> stop cassandra. I don't see any errors or warnings in the logs.
>>
>> What can I do to prevent this? Is there something else I should be doing
>> to gracefully restart the cluster? It could be something to do with the
>> nodejs driver, but I can't find anything there to try.
>>
>> I appreciate any suggestions or advice.
>>
>> - Mike
>>
>
>

Re: node restart causes application latency

[Jeff Jirsa]

Are you using internode ssl?


-- 
Jeff Jirsa


> On Feb 7, 2018, at 8:24 AM, Mike Torra  wrote:
> 
> Thanks for the feedback guys. That example data model was indeed abbreviated 
> - the real queries have the partition key in them. I am using RF 3 on the 
> keyspace, so I don't think a node being down would mean the key I'm looking 
> for would be unavailable. The load balancing policy of the driver seems 
> correct 
> (https://docs.datastax.com/en/developer/nodejs-driver/3.4/features/tuning-policies/#load-balancing-policy,
>  and I am using the default `TokenAware` policy with 
> `DCAwareRoundRobinPolicy` as a child), but I will look more closely at the 
> implementation.
> 
> It was an oversight of mine to not include `nodetool disablebinary`, but I 
> still experience the same issue with that.
> 
> One other thing I've noticed is that after restarting a node and seeing 
> application latency, I also see that the node I just restarted sees many 
> other nodes in the same DC as being down (ie status 'DN'). However, checking 
> `nodetool status` on those other nodes shows all nodes as up/normal. To me 
> this could kind of explain the problem - node comes back online, thinks it is 
> healthy but many others are not, so it gets traffic from the client 
> application. But then it gets requests for ranges that belong to a node it 
> thinks is down, so it responds with an error. The latency issue seems to 
> start roughly when the node goes down, but persists long (ie 15-20 mins) 
> after it is back online and accepting connections. It seems to go away once 
> the bounced node shows the other nodes in the same DC as up again.
> 
> As for speculative retry, my CF is using the default of '99th percentile'. I 
> could try something different there, but nodes being seen as down seems like 
> an issue.
> 
>> On Tue, Feb 6, 2018 at 6:26 PM, Jeff Jirsa  wrote:
>> Unless you abbreviated, your data model is questionable (SELECT without any 
>> equality in the WHERE clause on the partition key will always cause a range 
>> scan, which is super inefficient). Since you're doing LOCAL_ONE and a range 
>> scan, timeouts sorta make sense - the owner of at least one range would be 
>> down for a bit. 
>> 
>> If you actually have a partition key in your where clause, then the next 
>> most likely guess is your clients aren't smart enough to route around the 
>> node as it restarts, or your key cache is getting cold during the bounce. 
>> Double check your driver's load balancing policy. 
>> 
>> It's also likely the case that speculative retry may help other nodes route 
>> around the bouncing instance better - if you're not using it, you probably 
>> should be (though with CL: LOCAL_ONE, it seems like it'd be less of an 
>> issue).
>> 
>> We need to make bouncing nodes easier (or rather, we need to make drain do 
>> the right thing), but in this case, your data model looks like the biggest 
>> culprit (unless it's an incomplete recreation).
>> 
>> - Jeff
>> 
>> 
>>> On Tue, Feb 6, 2018 at 10:58 AM, Mike Torra  wrote:
>>> Hi -
>>> 
>>> I am running a 29 node cluster spread over 4 DC's in EC2, using C* 3.11.1 
>>> on Ubuntu. Occasionally I have the need to restart nodes in the cluster, 
>>> but every time I do, I see errors and application (nodejs) timeouts.
>>> 
>>> I restart a node like this:
>>> 
>>> nodetool disablethrift && nodetool disablegossip && nodetool drain
>>> sudo service cassandra restart
>>> 
>>> When I do that, I very often get timeouts and errors like this in my nodejs 
>>> app:
>>> 
>>> Error: Cannot achieve consistency level LOCAL_ONE
>>> 
>>> My queries are all pretty much the same, things like: "select * from 
>>> history where ts > {current_time}"
>>> 
>>> The errors and timeouts seem to go away on their own after a while, but it 
>>> is frustrating because I can't track down what I am doing wrong!
>>> 
>>> I've tried waiting between steps of shutting down cassandra, and I've tried 
>>> stopping, waiting, then starting the node. One thing I've noticed is that 
>>> even after `nodetool drain`ing the node, there are open connections to 
>>> other nodes in the cluster (ie looking at the output of netstat) until I 
>>> stop cassandra. I don't see any errors or warnings in the logs.
>>> 
>>> What can I do to prevent this? Is there something else I should be doing to 
>>> gracefully restart the cluster? It could be something to do with the nodejs 
>>> driver, but I can't find anything there to try.
>>> 
>>> I appreciate any suggestions or advice.
>>> 
>>> - Mike
>> 
> 

Re: node restart causes application latency

[Mike Torra]

No, I am not

On Wed, Feb 7, 2018 at 11:35 AM, Jeff Jirsa  wrote:

> Are you using internode ssl?
>
>
> --
> Jeff Jirsa
>
>
> On Feb 7, 2018, at 8:24 AM, Mike Torra  wrote:
>
> Thanks for the feedback guys. That example data model was indeed
> abbreviated - the real queries have the partition key in them. I am using
> RF 3 on the keyspace, so I don't think a node being down would mean the key
> I'm looking for would be unavailable. The load balancing policy of the
> driver seems correct (https://docs.datastax.com/en/
> developer/nodejs-driver/3.4/features/tuning-policies/#
> load-balancing-policy, and I am using the default `TokenAware` policy
> with `DCAwareRoundRobinPolicy` as a child), but I will look more closely at
> the implementation.
>
> It was an oversight of mine to not include `nodetool disablebinary`, but I
> still experience the same issue with that.
>
> One other thing I've noticed is that after restarting a node and seeing
> application latency, I also see that the node I just restarted sees many
> other nodes in the same DC as being down (ie status 'DN'). However,
> checking `nodetool status` on those other nodes shows all nodes as
> up/normal. To me this could kind of explain the problem - node comes back
> online, thinks it is healthy but many others are not, so it gets traffic
> from the client application. But then it gets requests for ranges that
> belong to a node it thinks is down, so it responds with an error. The
> latency issue seems to start roughly when the node goes down, but persists
> long (ie 15-20 mins) after it is back online and accepting connections. It
> seems to go away once the bounced node shows the other nodes in the same DC
> as up again.
>
> As for speculative retry, my CF is using the default of '99th percentile'.
> I could try something different there, but nodes being seen as down seems
> like an issue.
>
> On Tue, Feb 6, 2018 at 6:26 PM, Jeff Jirsa  wrote:
>
>> Unless you abbreviated, your data model is questionable (SELECT without
>> any equality in the WHERE clause on the partition key will always cause a
>> range scan, which is super inefficient). Since you're doing LOCAL_ONE and a
>> range scan, timeouts sorta make sense - the owner of at least one range
>> would be down for a bit.
>>
>> If you actually have a partition key in your where clause, then the next
>> most likely guess is your clients aren't smart enough to route around the
>> node as it restarts, or your key cache is getting cold during the bounce.
>> Double check your driver's load balancing policy.
>>
>> It's also likely the case that speculative retry may help other nodes
>> route around the bouncing instance better - if you're not using it, you
>> probably should be (though with CL: LOCAL_ONE, it seems like it'd be less
>> of an issue).
>>
>> We need to make bouncing nodes easier (or rather, we need to make drain
>> do the right thing), but in this case, your data model looks like the
>> biggest culprit (unless it's an incomplete recreation).
>>
>> - Jeff
>>
>>
>> On Tue, Feb 6, 2018 at 10:58 AM, Mike Torra 
>> wrote:
>>
>>> Hi -
>>>
>>> I am running a 29 node cluster spread over 4 DC's in EC2, using C*
>>> 3.11.1 on Ubuntu. Occasionally I have the need to restart nodes in the
>>> cluster, but every time I do, I see errors and application (nodejs)
>>> timeouts.
>>>
>>> I restart a node like this:
>>>
>>> nodetool disablethrift && nodetool disablegossip && nodetool drain
>>> sudo service cassandra restart
>>>
>>> When I do that, I very often get timeouts and errors like this in my
>>> nodejs app:
>>>
>>> Error: Cannot achieve consistency level LOCAL_ONE
>>>
>>> My queries are all pretty much the same, things like: "select * from
>>> history where ts > {current_time}"
>>>
>>> The errors and timeouts seem to go away on their own after a while, but
>>> it is frustrating because I can't track down what I am doing wrong!
>>>
>>> I've tried waiting between steps of shutting down cassandra, and I've
>>> tried stopping, waiting, then starting the node. One thing I've noticed is
>>> that even after `nodetool drain`ing the node, there are open connections to
>>> other nodes in the cluster (ie looking at the output of netstat) until I
>>> stop cassandra. I don't see any errors or warnings in the logs.
>>>
>>> What can I do to prevent this? Is there something else I should be doing
>>> to gracefully restart the cluster? It could be something to do with the
>>> nodejs driver, but I can't find anything there to try.
>>>
>>> I appreciate any suggestions or advice.
>>>
>>> - Mike
>>>
>>
>>
>

Re: Hints folder missing in Cassandra

[test user]

The directory */var/lib/cassandra/hints *was created, when I brought up the
cluster. I am running into this issue until after the cluster is up and
running for about 2 weeks.

Regards,
Cassandra User

On Wed, Feb 7, 2018 at 1:28 AM, Kyrylo Lebediev 
wrote:

> Hi,
>
> Not sure what could be the reason, but the issue is obvious: directory
> */var/lib/cassandra/hints* is missing or cassandra user doesn't have
> enough permissions on it.
>
>
> Regards,
>
> Kyrill
> --
> *From:* test user 
> *Sent:* Wednesday, February 7, 2018 8:14:59 AM
> *To:* user@cassandra.apache.org
> *Subject:* Fwd: Hints folder missing in Cassandra
>
>
> Has anyone run into the issue mentioned below?
>
>
> -- Forwarded message --
> From: *test user* 
> Date: Tue, Feb 6, 2018 at 3:28 PM
> Subject: Hints folder missing in Cassandra
> To: user-subscr...@cassandra.apache.org
>
>
> Hello All,
>
>
>
> I am using Cassandra 3.10. I have 3-node cluster (say node0, node1,
> node2).
>
> I have been using nodetool utility to monitor the cluster.
>
> nodetool status – indicates all nodes are up and in normal state
>
> nodetool ring – indicates all nodes are up
>
>
>
> Running into following issue:
>
> *WARN  [HintsWriteExecutor:1] 2018-02-05 06:57:09,607 CLibrary.java:295 -
> open(/var/lib/cassandra/hints, O_RDONLY) failed, errno (2).*
>
> For some reason, the hints folder on node1 is missing. Not sure how it got
> deleted. One node in the cluster always seems to run into this issue.
>
> The other two nodes node0 and node2 do not have this issue, and the hints
> folder is present.
>
>
>
> The debug logs do not seem to indicate the root cause.
>
>
>
> Please let me know how to debug this further to understand the root cause.
>
>
>
> Regards,
>
> Cassandra User
>
>
>
>

Re: Add column if it does not exist?

[Jon Haddad]

All of the drivers also have keyspace / table metadata. For instance: 
https://datastax.github.io/python-driver/api/cassandra/metadata.html 


I’d be *really* careful how you use this.  A lot of teams want to just deploy 
their code to a couple hundred servers and let them race to apply the ALTER.  
That will not be a fun time.  I advise against using LWT to manage this as 
well. If you’re looking to apply schema changes like this, I’d let a single app 
server manage it and avoid the headache of concurrency.

Jon

> On Feb 6, 2018, at 8:13 PM, Irtiza Ali  wrote:
> 
> Hello,
> 
> this link might also be helpful to you for querying table schema.
> 
> Link: 
> https://docs.datastax.com/en/cql/3.3/cql/cql_using/useQuerySystemTable.html 
> 
> 
> Best, Iriiza
> 
> 
> 
> On Tue, Feb 6, 2018 at 9:55 PM, Oliver Ruebenacker  > wrote:
> 
>  Hello,
> 
>   Is there a describe query in CQL? I don't see one on 
> http://cassandra.apache.org/doc/latest/cql/index.html 
> .
> 
>   I also can't find such a query in the DataStax Java driver API.
> 
>   Thanks!
> 
>  Best, Oliver
> 
> On Tue, Feb 6, 2018 at 11:48 AM, Irtiza Ali  > wrote:
> Hello.
> 
> Another thing that you can try is the use the describe table query to get the 
> table schema and parse it. Once done you can check whether column exists or 
> not.
> 
> 
> With Regards 
> Irtiza Ali
> 
> On 6 Feb 2018 21:35, "Oliver Ruebenacker"  > wrote:
>   Thanks for the response!
> 
>   So, the best solution I can come up with is catching the 
> InvalidQueryException and check whether its message contains the phrase 
> "conflicts with an existing column". Seems to work, but super-ugly.
> 
>   I do assume that in general, if a request fails, it does not permanently 
> change the data in Cassandra, right?
> 
>   It would be great if alter-add could have an if-not-exists clause. Would 
> that be hard to implement?
> 
>   I could not find a standard CQL way of asking what columns exist. Did I 
> miss it? Would it be hard to implement?
> 
>   I get that we're only eventually consistent anyway. 
> 
>   Thanks!
> 
>  Best, Oliver
> 
> On Mon, Feb 5, 2018 at 5:12 PM, Rahul Singh  > wrote:
> Yeah, you can handle the exception — what i meant that it wouldnt cause harm 
> to the DB
> 
> --
> Rahul Singh
> rahul.si...@anant.us 
> 
> Anant Corporation
> 
> On Feb 5, 2018, 5:07 PM -0500, Oliver Ruebenacker  >, wrote:
>> Well, it does throw an InvalidQueryException if the column already exists.
>> 
>> On Mon, Feb 5, 2018 at 4:44 PM, Rahul Singh > > wrote:
>> Since CQL != SQL, there’s isnt a syntatical way. Just run the alter table 
>> command and it shouldn't be an issue if its there. 
>> 
>> --
>> Rahul Singh
>> rahul.si...@anant.us 
>> 
>> Anant Corporation
>> 
>> On Feb 5, 2018, 4:15 PM -0500, Oliver Ruebenacker > >, wrote:
>>> 
>>>  Hello,
>>> 
>>>   What's the easiest way to add a column to a table but only if it does not 
>>> exist? Thanks!
>>> 
>>>  Best, Oliver
>>> 
>>> --
>>> Oliver Ruebenacker
>>> Senior Software Engineer, Diabetes Portal 
>>> , Broad Institute 
>>> 
>>> 
>> 
>> 
>> 
>> --
>> Oliver Ruebenacker
>> Senior Software Engineer, Diabetes Portal 
>> , Broad Institute 
>> 
>> 
> 
> 
> 
> -- 
> Oliver Ruebenacker
> Senior Software Engineer, Diabetes Portal 
> , Broad Institute 
> 
> 
> 
> 
> 
> -- 
> Oliver Ruebenacker
> Senior Software Engineer, Diabetes Portal 
> , Broad Institute 
> 
> 
> 

Re: Bootstrapping fails with < 128GB RAM ...

[Jon Haddad]

It would be extremely helpful to get some info about your heap.  At a bare 
minimum, a histogram of the heap dump would be useful, but ideally a full heap 
dump would be best.

jmap  -dump:live,format=b,file=heap.bin PID

Taking a look at that in YourKit should give some pretty quick insight into 
what kinds of objects are allocated then we can get to the bottom of the issue. 
 This should be moved to a JIRA 
(https://issues.apache.org/jira/secure/Dashboard.jspa 
) in order to track and 
fix it, if you could attach that heap dump it would be very helpful.

Jon

> On Feb 7, 2018, at 6:11 AM, Nicolas Guyomar  wrote:
> 
> Ok then, following up on the wild guess : because you have quite a lot of 
> concurrent compactors, maybe it is too much concurrent compactions for the 
> jvm to deal with (taking into account that your load average of 106 seems 
> really high IMHO)
> 
> 55Gb of data is not that much, you can try to reduce those concurrent 
> compactor to make sure your box is not under too much stress (how many 
> compaction do you have in parallel during boostrap ? )
> 
> In the end, it does seem that you're gonna have to share some heap dump for 
> further investigation (sorry I'm not gonna help lot on this matter)
> 
> On 7 February 2018 at 14:43, Jürgen Albersdorfer  > wrote:
> Hi Nicolas,
> 
> Do you know how many sstables is this new node suppose to receive ?
> 
> If I can find out this via nodetool netstats, then this would be 619 as 
> following:
> 
> # nodetool netstats
> Bootstrap b95371e0-0c0a-11e8-932b-f775227bf21c
> /192.168.1.215  - Receiving 71 files, 7744612158 
> bytes total. Already received 0 files, 893897583 bytes total
> /192.168.1.214  - Receiving 58 files, 5693392001 
> bytes total. Already received 0 files, 1078372756 bytes total
> /192.168.1.206  - Receiving 52 files, 3389096409 
> bytes total. Already received 3 files, 508592758 bytes total
> /192.168.1.213  - Receiving 59 files, 6041633329 
> bytes total. Already received 0 files, 1038760653 bytes total
> /192.168.1.231  - Receiving 79 files, 7579181689 
> bytes total. Already received 4 files, 38387859 bytes total
> /192.168.1.208  - Receiving 51 files, 3272885123 
> bytes total. Already received 3 files, 362450903 bytes total
> /192.168.1.207  - Receiving 56 files, 3028344200 
> bytes total. Already received 3 files, 57790197 bytes total
> /192.168.1.232  - Receiving 79 files, 7268716317 
> bytes total. Already received 1 files, 1127174421 bytes total
> /192.168.1.209  - Receiving 114 files, 21381846105 
> bytes total. Already received 1 files, 961497222 bytes total
> 
> 
> does disabling compaction_throughput_mb_per_sec or increasing 
> concurrent_compactors has any effect ?
> 
> I will give it a try:
> 
> # nodetool getcompactionthroughput
> Current compaction throughput: 128 MB/s
> 
> # nodetool setcompactionthroughput 0
> 
> # nodetool getcompactionthroughput
> Current compaction throughput: 0 MB/s
> 
> # nodetool getconcurrentcompactors
> Current concurrent compactors in the system is:
> 16
> 
> 
> Which memtable_allocation_type are you using ?
>  
> # grep memtable_allocation_type /etc/cassandra/conf/cassandra.yaml
> memtable_allocation_type: heap_buffers
> 
> 
> thanks so far, regards
> Juergen
> 
> 2018-02-07 14:29 GMT+01:00 Nicolas Guyomar  >:
> Hi Jurgen,
> 
> It does feel like some OOM during boostrap from previous C* v2, but that 
> sould be fixed in your version.
> 
> Do you know how many sstables is this new node suppose to receive ? 
> 
> Juste a wild guess : it may have something to do with compaction not keeping 
> up because every other nodes are streaming data to this new one (resulting in 
> long lived object in the heap), does disabling 
> compaction_throughput_mb_per_sec or increasing concurrent_compactors has any 
> effect ? 
> 
> Which memtable_allocation_type are you using ? 
> 
> 
> On 7 February 2018 at 12:38, Jürgen Albersdorfer  > wrote:
> Hi, I always face an issue when bootstrapping a Node having less than 184GB 
> RAM (156GB JVM HEAP) on our 10 Node C* 3.11.1 Cluster.
> During bootstrap, when I watch the cassandra.log I observe a growth in JVM 
> Heap Old Gen which gets not significantly freed any more.
> I know that JVM collects on Old Gen only when really needed. I can see 
> collections, but there is always a remainder which
> seems to grow forever without ever getting freed.
> After the Node successfully Joined the Cluster, I can remove the extra 128GB 
> of RAM I have given it for bootstrapping without any further effect.
> 
> It feels like Cassandra will not forget about every single byte streamed over 

Re: Add column if it does not exist?

[Rahul Singh]

Yah. I saw one such migration via Spark Job running concurrently and created 4 
Cfids and migrated data. It was a nightmare to cleanup the duplicated sstables.

Alter schema and migrate should always be different applications separate from 
the actual system.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Feb 7, 2018, 12:39 PM -0600, Jon Haddad , wrote:
> All of the drivers also have keyspace / table metadata. For instance: 
> https://datastax.github.io/python-driver/api/cassandra/metadata.html
>
> I’d be *really* careful how you use this.  A lot of teams want to just deploy 
> their code to a couple hundred servers and let them race to apply the ALTER.  
> That will not be a fun time.  I advise against using LWT to manage this as 
> well. If you’re looking to apply schema changes like this, I’d let a single 
> app server manage it and avoid the headache of concurrency.
>
> Jon
>
> > On Feb 6, 2018, at 8:13 PM, Irtiza Ali  wrote:
> >
> > Hello,
> >
> > this link might also be helpful to you for querying table schema.
> >
> > Link: 
> > https://docs.datastax.com/en/cql/3.3/cql/cql_using/useQuerySystemTable.html
> >
> > Best, Iriiza
> >
> >
> >
> > > On Tue, Feb 6, 2018 at 9:55 PM, Oliver Ruebenacker  
> > > wrote:
> > > >
> > > >  Hello,
> > > >
> > > >   Is there a describe query in CQL? I don't see one on 
> > > > http://cassandra.apache.org/doc/latest/cql/index.html.
> > > >
> > > >   I also can't find such a query in the DataStax Java driver API.
> > > >
> > > >   Thanks!
> > > >
> > > >  Best, Oliver
> > > >
> > > > > On Tue, Feb 6, 2018 at 11:48 AM, Irtiza Ali  wrote:
> > > > > > Hello.
> > > > > >
> > > > > > Another thing that you can try is the use the describe table query 
> > > > > > to get the table schema and parse it. Once done you can check 
> > > > > > whether column exists or not.
> > > > > >
> > > > > >
> > > > > > With Regards
> > > > > > Irtiza Ali
> > > > > >
> > > > > > > On 6 Feb 2018 21:35, "Oliver Ruebenacker"  
> > > > > > > wrote:
> > > > > > > >   Thanks for the response!
> > > > > > > >
> > > > > > > >   So, the best solution I can come up with is catching the 
> > > > > > > > InvalidQueryException and check whether its message contains 
> > > > > > > > the phrase "conflicts with an existing column". Seems to work, 
> > > > > > > > but super-ugly.
> > > > > > > >
> > > > > > > >   I do assume that in general, if a request fails, it does not 
> > > > > > > > permanently change the data in Cassandra, right?
> > > > > > > >
> > > > > > > >   It would be great if alter-add could have an if-not-exists 
> > > > > > > > clause. Would that be hard to implement?
> > > > > > > >
> > > > > > > >   I could not find a standard CQL way of asking what columns 
> > > > > > > > exist. Did I miss it? Would it be hard to implement?
> > > > > > > >
> > > > > > > >   I get that we're only eventually consistent anyway.
> > > > > > > >
> > > > > > > >   Thanks!
> > > > > > > >
> > > > > > > >  Best, Oliver
> > > > > > > >
> > > > > > > > > On Mon, Feb 5, 2018 at 5:12 PM, Rahul Singh 
> > > > > > > > >  wrote:
> > > > > > > > > > Yeah, you can handle the exception — what i meant that it 
> > > > > > > > > > wouldnt cause harm to the DB
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Rahul Singh
> > > > > > > > > > rahul.si...@anant.us
> > > > > > > > > >
> > > > > > > > > > Anant Corporation
> > > > > > > > > >
> > > > > > > > > > On Feb 5, 2018, 5:07 PM -0500, Oliver Ruebenacker 
> > > > > > > > > > , wrote:
> > > > > > > > > > > Well, it does throw an InvalidQueryException if the 
> > > > > > > > > > > column already exists.
> > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 5, 2018 at 4:44 PM, Rahul Singh 
> > > > > > > > > > > >  wrote:
> > > > > > > > > > > > > Since CQL != SQL, there’s isnt a syntatical way. Just 
> > > > > > > > > > > > > run the alter table command and it shouldn't be an 
> > > > > > > > > > > > > issue if its there.
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Rahul Singh
> > > > > > > > > > > > > rahul.si...@anant.us
> > > > > > > > > > > > >
> > > > > > > > > > > > > Anant Corporation
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Feb 5, 2018, 4:15 PM -0500, Oliver Ruebenacker 
> > > > > > > > > > > > > , wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  Hello,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   What's the easiest way to add a column to a table 
> > > > > > > > > > > > > > but only if it does not exist? Thanks!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  Best, Oliver
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Oliver Ruebenacker
> > > > > > > > > > > > > > Senior Software Engineer, Diabetes Portal, Broad 
> > > > > > > > > > > > > > Institute
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Olive

Re: Hints folder missing in Cassandra

[Rahul Singh]

Do you have any automation being applied via chef or likewise? Was the 
environment built using automation or using established images ?

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Feb 7, 2018, 10:37 AM -0600, test user , wrote:
> The directory /var/lib/cassandra/hints was created, when I brought up the 
> cluster. I am running into this issue until after the cluster is up and 
> running for about 2 weeks.
>
> Regards,
> Cassandra User
>
> > On Wed, Feb 7, 2018 at 1:28 AM, Kyrylo Lebediev  
> > wrote:
> > > Hi,
> > > Not sure what could be the reason, but the issue is obvious: directory 
> > > /var/lib/cassandra/hints is missing or cassandra user doesn't have enough 
> > > permissions on it.
> > >
> > > Regards,
> > > Kyrill
> > > From: test user 
> > > Sent: Wednesday, February 7, 2018 8:14:59 AM
> > > To: user@cassandra.apache.org
> > > Subject: Fwd: Hints folder missing in Cassandra
> > >
> > >
> > > Has anyone run into the issue mentioned below?
> > >
> > >
> > > -- Forwarded message --
> > > From: test user 
> > > Date: Tue, Feb 6, 2018 at 3:28 PM
> > > Subject: Hints folder missing in Cassandra
> > > To: user-subscr...@cassandra.apache.org
> > >
> > >
> > > Hello All,
> > >
> > > I am using Cassandra 3.10. I have 3-node cluster (say node0, node1, 
> > > node2).
> > > I have been using nodetool utility to monitor the cluster.
> > > nodetool status – indicates all nodes are up and in normal state
> > > nodetool ring – indicates all nodes are up
> > >
> > > Running into following issue:
> > > WARN  [HintsWriteExecutor:1] 2018-02-05 06:57:09,607 CLibrary.java:295 - 
> > > open(/var/lib/cassandra/hints, O_RDONLY) failed, errno (2).
> > > For some reason, the hints folder on node1 is missing. Not sure how it 
> > > got deleted. One node in the cluster always seems to run into this issue.
> > > The other two nodes node0 and node2 do not have this issue, and the hints 
> > > folder is present.
> > >
> > > The debug logs do not seem to indicate the root cause.
> > >
> > > Please let me know how to debug this further to understand the root cause.
> > >
> > > Regards,
> > > Cassandra User
> > >
> > >
> > >
>

Re: Hints folder missing in Cassandra

[test user]

The environment is built using established images for Cassandra 3.10.
Unfortunately the debug log does not indicate any errors before I start
seeing the WARN for missing hints folder. I understand that hints file will
be deleted after replay is complete, but not sure of the root cause of why
the hints folder is getting deleted.
When I look at the nodetool status or nodetool ring - it indicates that all
nodes are up and running in normal state, no node went down. Also, I do not
see anything the debug logs indicating that a node went down. In such a
scenario, I am not sure why would HintsWriterExecutor would get triggered.

Regards,
Cassandra User

On Wed, Feb 7, 2018 at 11:25 AM, Rahul Singh 
wrote:

> Do you have any automation being applied via chef or likewise? Was the
> environment built using automation or using established images ?
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
>
> On Feb 7, 2018, 10:37 AM -0600, test user , wrote:
>
> The directory */var/lib/cassandra/hints* was created, when I brought up
> the cluster. I am running into this issue until after the cluster is up and
> running for about 2 weeks.
>
> Regards,
> Cassandra User
>
> On Wed, Feb 7, 2018 at 1:28 AM, Kyrylo Lebediev 
> wrote:
>
>> Hi,
>>
>> Not sure what could be the reason, but the issue is obvious: directory
>> */var/lib/cassandra/hints* is missing or cassandra user doesn't have
>> enough permissions on it.
>>
>>
>> Regards,
>>
>> Kyrill
>> --
>> *From:* test user 
>> *Sent:* Wednesday, February 7, 2018 8:14:59 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Fwd: Hints folder missing in Cassandra
>>
>>
>> Has anyone run into the issue mentioned below?
>>
>>
>> -- Forwarded message --
>> From: *test user* 
>> Date: Tue, Feb 6, 2018 at 3:28 PM
>> Subject: Hints folder missing in Cassandra
>> To: user-subscr...@cassandra.apache.org
>>
>>
>> Hello All,
>>
>>
>>
>> I am using Cassandra 3.10. I have 3-node cluster (say node0, node1,
>> node2).
>>
>> I have been using nodetool utility to monitor the cluster.
>>
>> nodetool status – indicates all nodes are up and in normal state
>>
>> nodetool ring – indicates all nodes are up
>>
>>
>>
>> Running into following issue:
>>
>> *WARN  [HintsWriteExecutor:1] 2018-02-05 06:57:09,607 CLibrary.java:295 -
>> open(/var/lib/cassandra/hints, O_RDONLY) failed, errno (2).*
>>
>> For some reason, the hints folder on node1 is missing. Not sure how it
>> got deleted. One node in the cluster always seems to run into this issue.
>>
>> The other two nodes node0 and node2 do not have this issue, and the hints
>> folder is present.
>>
>>
>>
>> The debug logs do not seem to indicate the root cause.
>>
>>
>>
>> Please let me know how to debug this further to understand the root cause.
>>
>>
>>
>> Regards,
>>
>> Cassandra User
>>
>>
>>
>>
>

Re: Hints folder missing in Cassandra

[Nate McCall]

> The environment is built using established images for Cassandra 3.10.
> Unfortunately the debug log does not indicate any errors before I start
> seeing the WARN for missing hints folder. I understand that hints file will
> be deleted after replay is complete, but not sure of the root cause of why
> the hints folder is getting deleted.
> When I look at the nodetool status or nodetool ring - it indicates that
> all nodes are up and running in normal state, no node went down. Also, I do
> not see anything the debug logs indicating that a node went down. In such a
> scenario, I am not sure why would HintsWriterExecutor would get triggered.
>
>
That error code (O_RDONLY) in the log message indicates that the hints
folder has had its permission bits set to read only.

We've had several issues with some of the tools doing this type of thing
when they are run as the root user. Is this specific node one on which you
use any of the tools like sstableloader or similar? If so, are you running
them as root?

Another thought - if it is on a different partition than the data
directory, is there free space left on the underlying device holding:
/var/lib/cassandra/hints?


-- 
-
Nate McCall
Wellington, NZ
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Hints folder missing in Cassandra

[test user]

The problem is even though, I get that O_RDONLY WARN message, if I try to
navigate to the path where the hints folder is stored, the folder is not
present.
I cannot check the permissions on that folder, its already missing, got
deleted somehow.

I believe everything runs as the root user.

I do see a lot of sstable activity performed by MemTableFlushWriter
(ColumnFamilyStore), CompactionExecutor, PerDiskFlushWriter (MemTable)
before and after this WARN message.

It is not a space issue, I checked that already.



On Wed, Feb 7, 2018 at 3:49 PM, Nate McCall  wrote:

>
> The environment is built using established images for Cassandra 3.10.
>> Unfortunately the debug log does not indicate any errors before I start
>> seeing the WARN for missing hints folder. I understand that hints file will
>> be deleted after replay is complete, but not sure of the root cause of why
>> the hints folder is getting deleted.
>> When I look at the nodetool status or nodetool ring - it indicates that
>> all nodes are up and running in normal state, no node went down. Also, I do
>> not see anything the debug logs indicating that a node went down. In such a
>> scenario, I am not sure why would HintsWriterExecutor would get triggered.
>>
>>
> That error code (O_RDONLY) in the log message indicates that the hints
> folder has had its permission bits set to read only.
>
> We've had several issues with some of the tools doing this type of thing
> when they are run as the root user. Is this specific node one on which you
> use any of the tools like sstableloader or similar? If so, are you running
> them as root?
>
> Another thought - if it is on a different partition than the data
> directory, is there free space left on the underlying device holding:
> /var/lib/cassandra/hints?
>
>
> --
> -
> Nate McCall
> Wellington, NZ
> @zznate
>
> CTO
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>