Too many open files

[Andreou, Arys (Nokia - GR/Athens)]

Hi,

I keep getting a "Last error: Too many open files" followed by a list of node 
IPs.
The output of "lsof -n|grep java|wc -l" is about 674970 on each node.

What is a normal number of open files?

Thank you.

Re: Too many open files

[Dor Laor]

It's a high number, your compaction may run behind and thus
many small sstables exist. However, you're also taking the
number of network connection in the calculation (everything
in *nix is a file). If it makes you feel better my laptop
has 40k open files for Chrome..

On Sun, Jan 21, 2018 at 11:59 PM, Andreou, Arys (Nokia - GR/Athens) <
arys.andr...@nokia.com> wrote:

> Hi,
>
>
>
> I keep getting a “Last error: Too many open files” followed by a list of
> node IPs.
>
> The output of “lsof -n|grep java|wc -l” is about 674970 on each node.
>
>
>
> What is a normal number of open files?
>
>
>
> Thank you.
>
>
>

Re: Too many open files

[Nikolay Mihaylov]

You can increase system open files,
also if you compact, open files will go down.

On Mon, Jan 22, 2018 at 10:19 AM, Dor Laor  wrote:

> It's a high number, your compaction may run behind and thus
> many small sstables exist. However, you're also taking the
> number of network connection in the calculation (everything
> in *nix is a file). If it makes you feel better my laptop
> has 40k open files for Chrome..
>
> On Sun, Jan 21, 2018 at 11:59 PM, Andreou, Arys (Nokia - GR/Athens) <
> arys.andr...@nokia.com> wrote:
>
>> Hi,
>>
>>
>>
>> I keep getting a “Last error: Too many open files” followed by a list of
>> node IPs.
>>
>> The output of “lsof -n|grep java|wc -l” is about 674970 on each node.
>>
>>
>>
>> What is a normal number of open files?
>>
>>
>>
>> Thank you.
>>
>>
>>
>
>

RE: Too many open files

[Andreou, Arys (Nokia - GR/Athens)]

It turns out it was a mistake in the client’s implementation.
The session was created for each request but it was shut down, so all the 
connections were left open.
I only needed to execute a cluste.shutdown() once the request was over.

I do have a follow up question though.
Is it better to have a global session object or to create it and shut it down 
for every request?


From: n...@photonhost.com [mailto:n...@photonhost.com] On Behalf Of Nikolay 
Mihaylov
Sent: Monday, January 22, 2018 11:47 AM
To: user@cassandra.apache.org
Subject: Re: Too many open files

You can increase system open files,
also if you compact, open files will go down.

On Mon, Jan 22, 2018 at 10:19 AM, Dor Laor 
mailto:d...@scylladb.com>> wrote:
It's a high number, your compaction may run behind and thus
many small sstables exist. However, you're also taking the
number of network connection in the calculation (everything
in *nix is a file). If it makes you feel better my laptop
has 40k open files for Chrome..

On Sun, Jan 21, 2018 at 11:59 PM, Andreou, Arys (Nokia - GR/Athens) 
mailto:arys.andr...@nokia.com>> wrote:
Hi,

I keep getting a “Last error: Too many open files” followed by a list of node 
IPs.
The output of “lsof -n|grep java|wc -l” is about 674970 on each node.

What is a normal number of open files?

Thank you.

Re: Repair giving error

[Alain RODRIGUEZ]

Hello,

Some other thoughts:

- Are you using internode secured communications (and then use the port
7001 instead) ?
- A rolling restart might help, have you tried restarting a few / all the
nodes?

This issue is very weird and I am only making poor guesses here. This is
not an issue I have seen in the past, thus It might help to see the raw
outputs (nodetool status , keyspace replication strategy, WARN or
ERR logs,...) and also to have the command you are running.
Also, if there have been operations ran on this cluster recently that might
have trigger this (RF change, Snitch change, new DC, ... or any other major
change). it's good we know about history to have a feel of what the cluster
state can be currently.
Did this same command use to run and now fails or are repairs it something
you are trying to add and that never worked so far?

Some context might help us to help you :-),

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-01-18 19:00 GMT+00:00 Akshit Jain :

> Hi alain
> Thanks for the response.
> I'm using cassandra 3.10
> nodetool status  shows all the nodes up
> No schema disaggrement
> port 7000 is open
>
> Regards
> Akshit Jain
> 9891724697
>
> On Thu, Jan 18, 2018 at 4:53 PM, Alain RODRIGUEZ 
> wrote:
>
>> Hello,
>>
>> I looks like a communication issue.
>>
>> What Cassandra version are you using?
>> What's the result of 'nodetool status '?
>> Any schema disagreement 'nodetool describecluster'?
>> Is the port 7000 opened and the nodes communicating with each other?(Ping
>> is not proving connection is up, even though it is good to know the machine
>> is there and up :)).
>> Any other errors you could see in the logs?
>>
>> You might want to consider this an open source project my coworkers have
>> been working on (and are maintaining) called reaper that aims at making
>> repairs more efficient and easy to manage as repair is one of the most
>> tricky operation to handle for a Cassandra operator: http://cassandra-rea
>> per.io/. I did not work on this project directly but we have good
>> feedbacks and like this tool ourselves.
>>
>> C*heers,
>> ---
>> Alain Rodriguez - @arodream - al...@thelastpickle.com
>> France / Spain
>>
>> The Last Pickle - Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>>
>>
>>
>> 2018-01-14 7:47 GMT+00:00 Akshit Jain :
>>
>>> ​I have a 10 node C* cluster with 4-5 keyspaces​.
>>> I tried to perform nodetool repair one by one for each keyspace.
>>> For some keyspaces the repair passed but for some it gave this error:
>>> ​
>>> I am not able to figure out what is causing this issue.The replica nodes
>>> are up and I am able to ping them from this node.​
>>> ​Any suggestions?​
>>>
>>> *Error I am getting on incremental repair:*
>>>
>>> *[2018-01-10 12:50:14,047] Did not get positive replies from all
>>> endpoints. List of failed endpoint(s): [​a.b.c.d, ​e.f.g.h]*
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *-- StackTrace --java.lang.RuntimeException: Repair job has failed with
>>> the error message: [2018-01-10 12:50:14,047] Did not get positive replies
>>> from all endpoints. List of failed endpoint(s): [​a.b.c.d, ​e.f.g.h]at
>>> org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)at
>>> org.apache.cassandra.utils.pro
>>> gress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)at
>>> com.sun.jmx.remote.internal.Cl
>>> ientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)at
>>> com.sun.jmx.remote.internal.Cl
>>> ientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)at
>>> com.sun.jmx.remote.internal.Cl
>>> ientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)at
>>> com.sun.jmx.remote.internal.Cl
>>> ientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)*
>>>
>>
>>
>

Cassandra 3.11 - nodetool cleanup - Compaction interrupted ...

[Steinmaurer, Thomas]

Hello,

when triggering a "nodetool cleanup" with Cassandra 3.11, the nodetool call 
almost returns instantly and I see the following INFO log.

INFO  [CompactionExecutor:54] 2018-01-22 12:59:53,903 
CompactionManager.java:1777 - Compaction interrupted: 
Compaction@fc9b0073-1008-3a07-aeb9-baf6f3cd0b1c(ruxitdb, Ts2Final05Min, 
98624438/107305082)bytes
INFO  [CompactionExecutor:69] 2018-01-22 12:59:54,135 
CompactionManager.java:1777 - Compaction interrupted: 
Compaction@ea0742f8-f3be-365d-a689-26ab346fdfb0(ruxitdb, Ts2Final01Min, 
49238516/72948781)bytes


No error etc. Does the "compaction interrupted" sound ok?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freist?dterstra?e 313

Re: Too many open files

[Jeff Jirsa]

Typically, long lived connections are better, so global.



-- 
Jeff Jirsa


> On Jan 22, 2018, at 3:28 AM, Andreou, Arys (Nokia - GR/Athens) 
>  wrote:
> 
> It turns out it was a mistake in the client’s implementation.
> The session was created for each request but it was shut down, so all the 
> connections were left open.
> I only needed to execute a cluste.shutdown() once the request was over.
>  
> I do have a follow up question though.
> Is it better to have a global session object or to create it and shut it down 
> for every request?
>  
>  
> From: n...@photonhost.com [mailto:n...@photonhost.com] On Behalf Of Nikolay 
> Mihaylov
> Sent: Monday, January 22, 2018 11:47 AM
> To: user@cassandra.apache.org
> Subject: Re: Too many open files
>  
> You can increase system open files,
> also if you compact, open files will go down.
>  
> On Mon, Jan 22, 2018 at 10:19 AM, Dor Laor  wrote:
> It's a high number, your compaction may run behind and thus
> many small sstables exist. However, you're also taking the
> number of network connection in the calculation (everything
> in *nix is a file). If it makes you feel better my laptop
> has 40k open files for Chrome..
>  
> On Sun, Jan 21, 2018 at 11:59 PM, Andreou, Arys (Nokia - GR/Athens) 
>  wrote:
> Hi,
>  
> I keep getting a “Last error: Too many open files” followed by a list of node 
> IPs.
> The output of “lsof -n|grep java|wc -l” is about 674970 on each node.
>  
> What is a normal number of open files?
>  
> Thank you.
>  
>  
>  

Re: ALTER default_time_to_live

[Alain RODRIGUEZ]

Hello Vlad,

I don't remember the context of the sentences, but I see no
incompatibilities. I believe the misunderstanding comes from the word
'compaction' that is a bit large. In the first sentence it is about
tombstone compactions. in sentence two it is about standard, regular
compaction according to the chosen compaction strategy. Both are slightly
different things:

A compaction strategy can be done compacting data like in those buckets you
mentioned but still do the tombstones compactions. Tombstone compactions
are done specially for tombstone eviction. Historically they are involving
only one sstable (in recent versions can be multiple SSTables), they are
ran with a lower priority than normal compactions and are not disabled
unless the 'tombstone_threshold' is set to 1 (subproperty of compaction,
table level).

And expiring tables does not involve any compactions.

I hope this is more clear now.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com


2018-01-18 19:21 GMT+00:00 Vlad :

> Hi, thanks for answer!
>
> I've read article about TWCS, and I don't understand how claim
>
> "When rows reach their TTL (10 minutes here), they turn into tombstones.
> Our table defines that tombstones can be purged 1 minute after they were
> created. If all rows are created with the same TTL, SSTables will get 100%
> droppable tombstones eventually and perform full SSTable deletion instead
> of purging tombstones through compaction."
>
> goes with
> "Once the major compaction for a time window is completed, no further
> compaction of the data will ever occur."
>
> In above example TTL is 10 minutes, but time window is only one. As far as
> I understand C* never compacts passed bucket. Does it check tombstones
> anyway?
>
>
> On Thursday, January 18, 2018 1:32 PM, Alain RODRIGUEZ 
> wrote:
>
>
> I set  default_time_to_live for existing table. Does it affect existing
> data?
>
>
> No, it sets a default TTL for the future writes (that is no guarantee, as
> it can be overwritten in any specific query).
>
> It seems data to be deleted, but after compaction, I don't see any disk
> space freed as expected
>
>
> Indeed tombstones are responsible for tombstones eviction, yet there are
> some conditions to respect to be able to remove the tombstones (for
> consistency reasons). I detailed this last year, and even though the
> content is a bit old, main principles are still true and the tuning options
> are still relevant.
>
> *About deletes and tombstones: *http://thelastpickle.com/blog/2016/
> 07/27/about-deletes-and-tombstones.html
>
> *tl;dr: *I would give a try to *unchecked_tombstone_compaction: true*.
> Maybe also consider using TWCS because of this "TTL is also ten days on
> one table and 100 days on other.". But I really recommend you to
> understand how this all work to act wisely. My guess can be wrong.
>
> *About TWCS*: http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2018-01-18 11:15 GMT+00:00 Vlad :
>
> Hi,
>
> I set  default_time_to_live for existing table. Does it affect existing
> data? It seems data to be deleted, but after compaction, I don't see any
> disk space freed as expected. Database has data for almost year, GC time is
> ten days, and TTL is also ten days on one table and 100 days on other.
>
>  Cassandra version 3.11.0
>
> Thanks.
>
>
>
>
>

Re: Cassandra 3.11 - nodetool cleanup - Compaction interrupted ...

[kurt greaves]

It's fine and intended behaviour. Upgradesstables also has the same effect.
Basically cleanup operates on all SSTables on a node (for each table) and
will cancel any in-progress compactions and instead run cleanup across
them, as you can't have two different compactions including the same file.
The compactions will be retriggered if necessary after cleanup has
completed.



On 22 January 2018 at 13:05, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello,
>
>
>
> when triggering a „nodetool cleanup“ with Cassandra 3.11, the nodetool
> call almost returns instantly and I see the following INFO log.
>
>
>
> INFO  [CompactionExecutor:54] 2018-01-22 12:59:53,903
> CompactionManager.java:1777 - Compaction interrupted:
> Compaction@fc9b0073-1008-3a07-aeb9-baf6f3cd0b1c(ruxitdb, Ts2Final05Min,
> 98624438/107305082)bytes
>
> INFO  [CompactionExecutor:69] 2018-01-22 12:59:54,135
> CompactionManager.java:1777 - Compaction interrupted:
> Compaction@ea0742f8-f3be-365d-a689-26ab346fdfb0(ruxitdb, Ts2Final01Min,
> 49238516/72948781)bytes
>
>
>
>
>
> No error etc. Does the “compaction interrupted” sound ok?
>
>
>
> Thanks,
>
> Thomas
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freist
> 
> ädterstra
> 
> ße 313
> 
>

RE: Slender Cassandra Cluster Project

[Kenneth Brotman]

Thanks Anthony!  I’ve made a note to include that information in the 
documentation. You’re right.  It won’t work as intended unless that is 
configured properly.

 

I’m also favoring a couple other guidelines for Slender Cassandra:

1.   SSD’s only, no spinning disks

2.   At least two cores per node

 

For AWS, I’m favoring the c3.large on Linux.  It’s available in these regions: 
US-East, US-West and US-West2.  The specifications are listed as:

· Two (2) vCPU’s

· 3.7 Gib Memory

· Two (2) 16 GB SSD’s

· Moderate I/O

 

It’s going to be hard to beat the inexpensive cost of operating a Slender 
Cluster on demand in the cloud – and it fits a lot of the use cases well:  

 

· For under a $100 a month, in current pricing for EC2 instances, you 
can operate an eighteen (18) node Slender Cluster for five (5) hours a day, ten 
(10) days a month.  That’s fine for demonstrations, teaching or experiments 
that last half a day or less. 

· For under $20, you can have that Slender Cluster up all day long, up 
to ten (10) hours, for whatever demonstrations or experiments you want it for.

 

As always, feedback is encouraged.

 

Thanks,

 

Kenneth Brotman

 

From: Anthony Grasso [mailto:anthony.gra...@gmail.com] 
Sent: Sunday, January 21, 2018 3:57 PM
To: user
Subject: Re: Slender Cassandra Cluster Project

 

Hi Kenneth,

 

Fantastic idea!

 

One thing that came to mind from my reading of the proposed setup was rack 
awareness of each node. Given that the proposed setup contains three DCs, I 
assume that each node will be made rack aware? If not, consider defining three 
racks for each DC and placing two nodes in each rack. This will ensure that all 
the nodes in a single rack contain at most one replica of the data.

 

Regards,

Anthony

 

On 17 January 2018 at 11:24, Kenneth Brotman  
wrote:

Sure.  That takes the project from awesome to 10X awesome.  I absolutely would 
be willing to do that.  Thanks Kurt!

 

Regarding your comment on the keyspaces, I agree.  There should be a few simple 
examples one way or the other that can be duplicated and observed, and then an 
example to duplicate and play with that has a nice real world mix, with some 
keyspaces that replicate over only a subset of DC’s and some that replicate to 
all DC’s.

 

Kenneth Brotman 

 

From: kurt greaves [mailto:k...@instaclustr.com] 
Sent: Tuesday, January 16, 2018 1:31 PM
To: User
Subject: Re: Slender Cassandra Cluster Project

 

Sounds like a great idea. Probably would be valuable to add to the official 
docs as an example set up if you're willing.

 

Only thing I'd add is that you should have keyspaces that replicate over only a 
subset of DC's, plus one/some replicated to all DC's

 

On 17 Jan. 2018 03:26, "Kenneth Brotman"  wrote:

I’ve begun working on a reference project intended to provide guidance on 
configuring and operating a modest Cassandra cluster of about 18 nodes suitable 
for the economic study, demonstration, experimentation and testing of a 
Cassandra cluster.

 

The slender cluster would be designed to be as inexpensive as possible while 
still using real world hardware in order to lower the cost to those with 
limited initial resources. Sorry no Raspberry Pi’s for this project.  

 

There would be an on-premises version and a cloud version.  Guidance would be 
provided on configuring the cluster, on demonstrating key Cassandra behaviors, 
on files sizes, capacity to use with the Slender Cassandra Cluster, and so on.

 

Why about eighteen nodes? I tried to figure out what the minimum number of 
nodes needed for Cassandra to be Cassandra is?  Here were my considerations:

 

• A user wouldn’t run Cassandra in just one data center; so at 
least two datacenters.

• A user probably would want a third data center available for 
analytics.

• There needs to be enough nodes for enough parallelism to observe 
Cassandra’s distributed nature.

• The cluster should have enough nodes that one gets a sense of the 
need for cluster wide management tools to do things like repairs, snapshots and 
cluster monitoring.

• The cluster should be able to demonstrate a RF=3 with local 
quorum.  If replicated in all three data centers, one write would impact half 
the 18 nodes, 3 datacenters X 3 nodes per data center = 9 nodes of 18 nodes  If 
replicated in two of the data centers, one write would still impact one third 
of the 18 nodes, 2 DC’s X 3 nodes per DC = 6 of the 18 nodes.  

 

So eighteen seems like the minimum number of nodes needed.  That’s six nodes in 
each of three data centers.

 

Before I get too carried away with this project, I’m looking for some feedback 
on whether this project would indeed be helpful to others? Also, should the 
project be changed in any way?

 

It’s always a pleasure to connect with the Cassandra users’ community.  Thanks 
for all the hard work, the expertise, the civi