Cassandra 2.2.8 NoSuchFileException

[Jacob Willoughby]

I got cassandra-reaper up and running the other day on our 5 node 2.2.8 
cluster.  Now every once in awhile a node will get the below exception and the 
node shuts down its cql port.  Is this a bug? something I am doing wrong?  Are 
there any workarounds?


-Jacob Willoughby



ERROR [AntiEntropyStage:1] 2017-01-25 12:55:25,609 CassandraDaemon.java:195 - 
Exception in thread Thread[AntiEntropyStage:1,5,main]
org.apache.cassandra.io.FSWriteError: java.nio.file.NoSuchFileException: 
/data/cassandra/data/Objects/Objects-11baea008ccf11e6be00d321fe82b612/snapshots/8a2da7f0-e2fd-11e6-a9da-e96a8b206d4c/lb-127076-big-CompressionInfo.db
 -> /data/cassandra/data/O
bjects/Objects-11baea008ccf11e6be00d321fe82b612/lb-127076-big-CompressionInfo.db
at 
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:92) 
~[apache-cassandra-2.2.8.jar:2.2.8]
at 
org.apache.cassandra.io.sstable.format.SSTableReader.createLinks(SSTableReader.java:1777)
 ~[apache-cassandra-2.2.8.jar:2.2.8]
at 
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:2432)
 ~[apache-cassandra-2.2.8.jar:2.2.8]
at 
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2557) 
~[apache-cassandra-2.2.8.jar:2.2.8]
at 
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:110)
 ~[apache-cassandra-2.2.8.jar:2.2.8]
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) 
~[apache-cassandra-2.2.8.jar:2.2.8]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: java.nio.file.NoSuchFileException: 
/data/cassandra/data/Objects/Objects-11baea008ccf11e6be00d321fe82b612/snapshots/8a2da7f0-e2fd-11e6-a9da-e96a8b206d4c/lb-127076-big-CompressionInfo.db
 -> /data/cassandra/data/Objects/Objects-11baea008ccf
11e6be00d321fe82b612/lb-127076-big-CompressionInfo.db
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) 
~[na:1.8.0_121]
at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) 
~[na:1.8.0_121]
at 
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476) 
~[na:1.8.0_121]
at java.nio.file.Files.createLink(Files.java:1086) ~[na:1.8.0_121]
at 
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:88) 
~[apache-cassandra-2.2.8.jar:2.2.8]
... 10 common frames omitted
ERROR [AntiEntropyStage:1] 2017-01-25 12:55:25,611 StorageService.java:388 - 
Stopping gossiper
WARN  [AntiEntropyStage:1] 2017-01-25 12:55:25,611 StorageService.java:284 - 
Stopping gossip by operator request
INFO  [AntiEntropyStage:1] 2017-01-25 12:55:25,611 Gossiper.java:1451 - 
Announcing shutdown
INFO  [AntiEntropyStage:1] 2017-01-25 12:55:25,612 StorageService.java:1918 - 
Node /10.0.3.229 state jump to shutdown

Expensive to run nodetool status often?

[Xiaolei Li]

I'm planning to run "nodetool status -r" on every node every minute,
storing the output in a file, and aggregating it somewhere else for
monitoring.

Is that a good idea?  How expensive is it to be running status every minute.

Best,
x.

Re: Expensive to run nodetool status often?

[Jonathan Haddad]

You're about to walk down an unfortunate path.  I strongly recommend
getting the information you need for monitoring using JMX.  That's actually
how nodetool gets all it's information.  Instead of parsing output, if you
 use JMX, you'll have access to a *ton* of useful (and some not so useful)
information.

If you aren't familiar, run Cassandra locally and type "jconsole" in your
terminal.  That'll bring up a decent GUI that you can use to browse all the
available metrics.

You can export metrics a bunch of ways.  Jolokia, mx4j, jmx_exporter (for
prometheus), and I know there's a collectd plugin but I haven't used it,
might be worth checking out or maybe someone else can weigh in.

Jon

On Wed, Jan 25, 2017 at 7:48 AM Xiaolei Li  wrote:

> I'm planning to run "nodetool status -r" on every node every minute,
> storing the output in a file, and aggregating it somewhere else for
> monitoring.
>
> Is that a good idea?  How expensive is it to be running status every
> minute.
>
> Best,
> x.
>

Re: Expensive to run nodetool status often?

[Xiaolei Li]

Thanks for the advice!

I do export a lot via JMX already. But I couldn't find the equivalent of
the Status column (Up/Down + Normal/Leaving/Joining/Moving) from the status
output. Does anyone know if those are available via JMX?

Thank you.

Best,
x.

On Wed, Jan 25, 2017 at 8:15 AM, Jonathan Haddad  wrote:

> You're about to walk down an unfortunate path.  I strongly recommend
> getting the information you need for monitoring using JMX.  That's actually
> how nodetool gets all it's information.  Instead of parsing output, if you
>  use JMX, you'll have access to a *ton* of useful (and some not so useful)
> information.
>
> If you aren't familiar, run Cassandra locally and type "jconsole" in your
> terminal.  That'll bring up a decent GUI that you can use to browse all the
> available metrics.
>
> You can export metrics a bunch of ways.  Jolokia, mx4j, jmx_exporter (for
> prometheus), and I know there's a collectd plugin but I haven't used it,
> might be worth checking out or maybe someone else can weigh in.
>
> Jon
>
> On Wed, Jan 25, 2017 at 7:48 AM Xiaolei Li  wrote:
>
>> I'm planning to run "nodetool status -r" on every node every minute,
>> storing the output in a file, and aggregating it somewhere else for
>> monitoring.
>>
>> Is that a good idea?  How expensive is it to be running status every
>> minute.
>>
>> Best,
>> x.
>>
>

Re: Expensive to run nodetool status often?

[Brooke Jensen]

Have a look in org.apache.cassandra.net:type=FailureDetector


*Brooke Jensen*
VP Technical Operations & Customer Services
www.instaclustr.com | support.instaclustr.com


This email has been sent on behalf of Instaclustr Limited (Australia) and
Instaclustr Inc (USA). This email and any attachments may contain
confidential and legally privileged information.  If you are not the
intended recipient, do not copy or disclose its content, but please reply
to this email immediately and highlight the error to the sender and then
immediately delete the message.

On 26 January 2017 at 04:20, Xiaolei Li  wrote:

> Thanks for the advice!
>
> I do export a lot via JMX already. But I couldn't find the equivalent of
> the Status column (Up/Down + Normal/Leaving/Joining/Moving) from the
> status output. Does anyone know if those are available via JMX?
>
> Thank you.
>
> Best,
> x.
>
> On Wed, Jan 25, 2017 at 8:15 AM, Jonathan Haddad 
> wrote:
>
>> You're about to walk down an unfortunate path.  I strongly recommend
>> getting the information you need for monitoring using JMX.  That's actually
>> how nodetool gets all it's information.  Instead of parsing output, if you
>>  use JMX, you'll have access to a *ton* of useful (and some not so useful)
>> information.
>>
>> If you aren't familiar, run Cassandra locally and type "jconsole" in your
>> terminal.  That'll bring up a decent GUI that you can use to browse all the
>> available metrics.
>>
>> You can export metrics a bunch of ways.  Jolokia, mx4j, jmx_exporter (for
>> prometheus), and I know there's a collectd plugin but I haven't used it,
>> might be worth checking out or maybe someone else can weigh in.
>>
>> Jon
>>
>> On Wed, Jan 25, 2017 at 7:48 AM Xiaolei Li  wrote:
>>
>>> I'm planning to run "nodetool status -r" on every node every minute,
>>> storing the output in a file, and aggregating it somewhere else for
>>> monitoring.
>>>
>>> Is that a good idea?  How expensive is it to be running status every
>>> minute.
>>>
>>> Best,
>>> x.
>>>
>>
>

Using Hostname in Seed_Provider

[Nethi, Manoj]

Hi,

According to the documentation
seed_provider 

The addresses of hosts designated as contact points in the cluster. A joining 
node contacts one of the nodes in the -seeds list to learn the topology of the 
ring.
* class_name (Default:org.apache.cassandra.locator.SimpleSeedProvider)

The class within Cassandra that handles the seed logic. It can be customized, 
but this is typically not required.
* - seeds (Default: 127.0.0.1)
A comma-delimited list of IP addresses used by 
gossip
 for bootstrapping new nodes joining a cluster. If your cluster includes 
multiple nodes, you must change the list from the default value to the IP 
address of one of the nodes.

1.Does this work if the seeds value is a list of hostname instead of IP 
address ? consider that we make the listen and broadcast address of the seeds 
as hostname instead of IP address ?

2.If we use hostname and ever the IP address of that server changes will it 
still work the same and all the system tables like peers etc gets updated with 
the new IP address ? Or do we need to restart the node with the new IP address 
etc?

Thanks
Manoj

CS process killed by kernel OOM

[Benjamin Roth]

Hi there,

We installed 2 new nodes these days. They run on ubuntu (Ubuntu 16.04.1
LTS) with kernel 4.4.0-59-generic. On these nodes (and only on these) CS
gets killed by the kernel due to OOM. It seems very strange to me because,
CS only takes roughly 20GB (out of 128GB), most of RAM is allocated to page
cache.

Top looks typically like this:
KiB Mem : 13191691+total,  1974964 free, 20278184 used, 10966376+buff/cache
KiB Swap:0 total,0 free,0 used. 11051503+avail Mem

This is what kern.log says:
https://gist.github.com/brstgt/0f1aa6afb558a56d1cadce958db46cf9

Has anyone encountered sth like this before?

-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer