Too many open files with Cassandra 1.2.11

[Oleg Dulin]

Got this error:

WARN [Thread-8] 2013-10-29 02:58:24,565 CustomTThreadPoolServer.java 
(line 122) Transport error occurred during acceptance of message.
2 org.apache.thrift.transport.TTransportException: 
java.net.SocketException: Too many open files
3 at 
org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:109) 

4 at 
org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:36) 

5 at 
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) 

6 at 
org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:110) 

7 at 
org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:111) 



I haven't seen this since 1.0 days. 1.1.11 had it all fixed I thought.

ulimit outputs "unlimited"

What could cause this ?

Any help is greatly apprecaited.

--
Regards,
Oleg Dulin
http://www.olegdulin.com

RE: Read repair

[Baskar Duraikannu]

Aaron

Rack1 goes down and some writes happen in quorum against rack 2 and 3. Hinted 
handoff is set to 30 mins. After couple of hours rack1 comes back and rack2 
goes down. Hinted handoff will play but will not cover all of the writes 
because of 30 min setting. Now for rows inserted for about 1 hour and 30 mins, 
there is no quorum until failed rack comes back up.

Hope this explains the scenario.

From: Aaron Morton
Sent: ‎10/‎28/‎2013 2:42 AM
To: Cassandra User
Subject: Re: Read repair

> As soon as it came back up, due to some human error, rack1 goes down. Now for 
> some rows it is possible that Quorum cannot be established.
Not sure I follow here.

if the first rack has come up I assume all nodes are available, if you then 
lose a different rack I assume you have 2/3 of the nodes available and would be 
able to achieve a QUORUM.

> Just to minimize the issues, we are thinking of running read repair manually 
> every night.
If you are reading and writing at QUORUM and the cluster does not have a QUORUM 
of nodes available writes will not be processed. During reads any mismatch 
between the data returned from the nodes will be detected and resolved before 
returning to the client.

Read Repair is an automatic process that reads from more nodes than necessary 
and resolves the differences in the back ground.

I would run nodetool repair / Anti Entropy as normal, once on every machine 
every gc_grace_seconds. If you have a while rack fail for run repair on the 
nodes in the rack if you want to get it back to consistency quickly. The need 
to do that depends on the config for Hinted Handoff, read_repair_chance, 
Consistency level, the write load, and (to some degree) the number of nodes. If 
you want to be extra safe just run it.

Cheers

-
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 26/10/2013, at 2:54 pm, Baskar Duraikannu  
wrote:

> We are thinking through the deployment architecture for our Cassandra 
> cluster.  Let us say that we choose to deploy data across three racks.
>
> If let us say that one rack power went down for 10 mins and then it came 
> back. As soon as it came back up, due to some human error, rack1 goes down. 
> Now for some rows it is possible that Quorum cannot be established. Just to 
> minimize the issues, we are thinking of running read repair manually every 
> night.
>
> Is this a good idea? How often do you perform read repair on your cluster?
>

Too many open files (Cassandra 2.0.1)

[Pieter Callewaert]

Hi,

I've noticed some nodes in our cluster are dying after some period of time.

WARN [New I/O server boss #17] 2013-10-29 12:22:20,725 Slf4JLogger.java (line 
76) Failed to accept a connection.
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
at 
org.jboss.netty.channel.socket.nio.NioServerBoss.process(NioServerBoss.java:100)
at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at 
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

And other exceptions related to the same cause.
Now, as we use the Cassandra package, the nofile limit is raised to 10.
To double check if this correct:

root@de-cass09 ~ # cat /proc/18332/limits
Limit Soft Limit   Hard Limit   Units
...
Max open files10   10   files
...

Now I check how many files are open:
root@de-cass09 ~ # lsof -n -p 18332 | wc -l
100038

This seems an awful a lot for size tiered compaction... ?
Now I noticed when I checked the list, a (deleted) file passed a lot

...
java18332 cassandra 4704r   REG8,1  10911921661 2147483839 
/data1/mapdata040/hos/mapdata040-hos-jb-7648-Data.db (deleted)
java18332 cassandra 4705r   REG8,1  10911921661 2147483839 
/data1/mapdata040/hos/mapdata040-hos-jb-7648-Data.db (deleted)
...

Actually, if I count specific for this file:
root@de-cass09 ~ # lsof -n -p 18332 | grep mapdata040-hos-jb-7648-Data.db | wc 
-l
52707

Other nodes are around a total of 350 files open... Any idea why this nofiles 
is so high ?

The first exceptions I see is this:
WARN [New I/O worker #8] 2013-10-29 12:09:34,440 Slf4JLogger.java (line 76) 
Unexpected exception in the selector loop.
java.lang.NullPointerException
at 
sun.nio.ch.EPollArrayWrapper.setUpdateEvents(EPollArrayWrapper.java:178)
at sun.nio.ch.EPollArrayWrapper.add(EPollArrayWrapper.java:227)
at sun.nio.ch.EPollSelectorImpl.implRegister(EPollSelectorImpl.java:164)
at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:133)
at 
java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:209)
at 
org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:151)
at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
at 
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:724)

Several minutes later I get Too many open files.

Specs:
12-node cluster with Ubuntu 12.04 LTS, Cassandra 2.0.1 (datastax packages), 
using JBOD of 2 disks.
JNA enabled.

Any suggestions?

Kind regards,
Pieter Callewaert

[Description: cid:image003.png@01CD9CE5.CE5A2330]

   Pieter Callewaert
   Web & IT engineer

   Web:   www.be-mobile.be
   Email: pieter.callewa...@be-mobile.be
   Tel:  + 32 9 330 51 80



<>

Re: Query a datacenter

[srmore]

Thanks Rob that helps !


On Fri, Oct 25, 2013 at 7:34 PM, Robert Coli  wrote:

> On Fri, Oct 25, 2013 at 2:47 PM, srmore  wrote:
>
>> I don't know whether this is possible but was just curious, can you query
>> for the data in the remote datacenter with a CL.ONE ?
>>
>
> A coordinator at CL.ONE picks which replica(s) to query based in large
> part on the dynamic snitch. If your remote data center has a lower badness
> score from the perspective of the dynamic snitch, a CL.ONE request might go
> there.
>
> 1.2.11 adds [1] a LOCAL_ONE consistencylevel which does the opposite of
> what you are asking, restricting CL.ONE from going cross-DC.
>
>
>> There could be a case where one might not have a QUORUM and would like to
>> read the most recent  data which includes the data from the other
>> datacenter. AFAIK to reliably read the data from other datacenter we only
>> have CL.EACH_QUORUM.
>>
>
> Using CL.QUORUM requires a QUORUM number of responses, it does not care
> from which data center those responses come.
>
>
>> Also, is there a way one can control how frequently the data is
>> replicated across the datacenters ?
>>
>
> Data centers don't really exist in this context [2], so your question is
> "can one control how frequently data is replicated between replicas" and
> the answer is "no." All replication always goes to every replica.
>
> =Rob
> [1] https://issues.apache.org/jira/browse/CASSANDRA-6202
> [2] this is slightly glib/reductive/inaccurate, but accurate enough for
> the purposes of this response.
>

Re: CQL & selecting individual items from a map

[Keith Freeman]

There's some rationale here: 
http://mail-archives.apache.org/mod_mbox/cassandra-user/201305.mbox/%3CCAENxBwx6pcSA=cWn=dkw_52k5odw5f3xigj-zn_4bwfth+4...@mail.gmail.com%3E


And I'm sure part of the reason is the 64k size limit: maps (and sets 
and lists) are limited to 64k total size 
(http://wiki.apache.org/cassandra/CassandraLimitations), so it wouldn't 
be very read-efficient to load individual elements.


On 10/28/2013 08:03 PM, Liam Stewart wrote:
I was wondering if anybody could explain the rationale behind 
disallowing selection of individual elements from a map in CQL and why 
an entire map must be retrieved at once when items are stored as 
distinct columns? Are there any plans to allow individual selection?


--
Liam Stewart :: liam.stew...@gmail.com 

IllegalStateException when bootstrapping new nodes

[Cyril Scetbon]

Hi,

We added yesterday 4 nodes in one of our data-centers but met an exception on 2 
of them :

java.lang.IllegalStateException: unable to find sufficient sources for 
streaming range (6358673204990085385,6363273598121398976]
at 
org.apache.cassandra.dht.RangeStreamer.getRangeFetchMap(RangeStreamer.java:205)
at 
org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:129)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at 
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:890)
at 
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:659)
at 
org.apache.cassandra.service.StorageService.initServer(StorageService.java:514)
at 
org.apache.cassandra.service.StorageService.initServer(StorageService.java:411)
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:278)
at 
org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:311)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at 
org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)

We didn't find the reason why it didn't work but we are wondering if it's a 
BUG. We are using Cassandra 1.2.2 and we resolved the issue with a rolling 
restart of other nodes in the same DC and maybe with some luck... We don't know 
if it's important but we also added the first 2 new nodes added to the topology 
file before we successfully bootstrapped the last 2 nodes. 

FYI, A few weeks ago we added 4 new nodes in another DC with the same version 
of Cassandra without any issue.

Here is the output log with DEBUG level enabled 
http://www.sendspace.com/file/7p81lz

Thanks
-- 
Cyril SCETBON

Re: [RELEASE] Apache Cassandra 2.0.2 released

[Chris Burroughs]

On 10/28/2013 06:20 AM, Sylvain Lebresne wrote:

[2]:http://goo.gl/uEtkmb  (NEWS.txt)


https://wiki.apache.org/cassandra/Metrics has been updated with a 
reference to the new Configurable metrics reporting.

RE: Too many open files (Cassandra 2.0.1)

[Pieter Callewaert]

Investigated a bit more:


-I can reproduce it, happened already on several nodes when I do some 
stress testing (5 select's spread over multiple threads)

-Unexpected exception in the selector loop. Seems not related with the 
Too many open files, it just happens.

-It's not socket related.

-Using Oracle Java(TM) SE Runtime Environment (build 1.7.0_40-b43)

-Using multiple data directories (maybe related ?)

I'm stuck at the moment, I don't know If I should try DEBUG log because it will 
be too much information?

Kind regards,
Pieter Callewaert

[Description: cid:image003.png@01CD9CE5.CE5A2330]

   Pieter Callewaert
   Web & IT engineer

   Web:   www.be-mobile.be
   Email: pieter.callewa...@be-mobile.be
   Tel:  + 32 9 330 51 80



From: Pieter Callewaert [mailto:pieter.callewa...@be-mobile.be]
Sent: dinsdag 29 oktober 2013 13:40
To: user@cassandra.apache.org
Subject: Too many open files (Cassandra 2.0.1)

Hi,

I've noticed some nodes in our cluster are dying after some period of time.

WARN [New I/O server boss #17] 2013-10-29 12:22:20,725 Slf4JLogger.java (line 
76) Failed to accept a connection.
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
at 
org.jboss.netty.channel.socket.nio.NioServerBoss.process(NioServerBoss.java:100)
at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at 
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

And other exceptions related to the same cause.
Now, as we use the Cassandra package, the nofile limit is raised to 10.
To double check if this correct:

root@de-cass09 ~ # cat /proc/18332/limits
Limit Soft Limit   Hard Limit   Units
...
Max open files10   10   files
...

Now I check how many files are open:
root@de-cass09 ~ # lsof -n -p 18332 | wc -l
100038

This seems an awful a lot for size tiered compaction... ?
Now I noticed when I checked the list, a (deleted) file passed a lot

...
java18332 cassandra 4704r   REG8,1  10911921661 2147483839 
/data1/mapdata040/hos/mapdata040-hos-jb-7648-Data.db (deleted)
java18332 cassandra 4705r   REG8,1  10911921661 2147483839 
/data1/mapdata040/hos/mapdata040-hos-jb-7648-Data.db (deleted)
...

Actually, if I count specific for this file:
root@de-cass09 ~ # lsof -n -p 18332 | grep mapdata040-hos-jb-7648-Data.db | wc 
-l
52707

Other nodes are around a total of 350 files open... Any idea why this nofiles 
is so high ?

The first exceptions I see is this:
WARN [New I/O worker #8] 2013-10-29 12:09:34,440 Slf4JLogger.java (line 76) 
Unexpected exception in the selector loop.
java.lang.NullPointerException
at 
sun.nio.ch.EPollArrayWrapper.setUpdateEvents(EPollArrayWrapper.java:178)
at sun.nio.ch.EPollArrayWrapper.add(EPollArrayWrapper.java:227)
at sun.nio.ch.EPollSelectorImpl.implRegister(EPollSelectorImpl.java:164)
at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:133)
at 
java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:209)
at 
org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:151)
at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
at 
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:724)

Several minutes later I get Too many open files.

Specs:
12-node cluster with Ubuntu 12.04 LTS, Cassandra 2.0.1 (datastax packages), 
using JBOD of 2 disks.
JNA enabled.

Any suggestions?

Kind regards,
Pieter Callewaert

[Description: cid:image003.png@01CD9CE5.CE5A2330]

   Pieter Callewaert
   Web & IT engineer

   Web:   www.be-mobile.be
   Email: pieter.callewa...@be-mobile.be
   Tel:  + 32 9 330 51 80



<>

Re: IllegalStateException when bootstrapping new nodes

[Cyril Scetbon]

Sorry but as the link is bad here is the good one : 
http://www.sendspace.com/file/7p81lz
-- 
Cyril SCETBON

On 29 Oct 2013, at 15:38, Cyril Scetbon  wrote:

> Hi,
> 
> We added yesterday 4 nodes in one of our data-centers but met an exception on 
> 2 of them :
> 
> java.lang.IllegalStateException: unable to find sufficient sources for 
> streaming range (6358673204990085385,6363273598121398976]
>   at 
> org.apache.cassandra.dht.RangeStreamer.getRangeFetchMap(RangeStreamer.java:205)
>   at 
> org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:129)
>   at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
>   at 
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:890)
>   at 
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:659)
>   at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:514)
>   at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:411)
>   at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:278)
>   at 
> org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:311)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)
> 
> We didn't find the reason why it didn't work but we are wondering if it's a 
> BUG. We are using Cassandra 1.2.2 and we resolved the issue with a rolling 
> restart of other nodes in the same DC and maybe with some luck... We don't 
> know if it's important but we also added the first 2 new nodes added to the 
> topology file before we successfully bootstrapped the last 2 nodes. 
> 
> FYI, A few weeks ago we added 4 new nodes in another DC with the same version 
> of Cassandra without any issue.
> 
> Here is the output log with DEBUG level enabled 
> http://www.sendspace.com/file/7p81lz
> 
> Thanks
> -- 
> Cyril SCETBON
> 

Re: Too many open files (Cassandra 2.0.1)

[Jon Haddad]

In general, my understanding is that memory mapped files use a lot of open file 
handlers.  We raise all our DBs to unlimited open files.

On Oct 29, 2013, at 8:30 AM, Pieter Callewaert  
wrote:

> Investigated a bit more:
>  
> -I can reproduce it, happened already on several nodes when I do some 
> stress testing (5 select’s spread over multiple threads)
> -Unexpected exception in the selector loop. Seems not related with 
> the Too many open files, it just happens.
> -It’s not socket related.
> -Using Oracle Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
> -Using multiple data directories (maybe related ?)
>  
> I’m stuck at the moment, I don’t know If I should try DEBUG log because it 
> will be too much information?
>  
> Kind regards,
> Pieter Callewaert
>  
> 
>Pieter Callewaert
>Web & IT engineer
>  
>Web:   www.be-mobile.be
>Email: pieter.callewa...@be-mobile.be
>Tel:  + 32 9 330 51 80
>  
>  
> From: Pieter Callewaert [mailto:pieter.callewa...@be-mobile.be] 
> Sent: dinsdag 29 oktober 2013 13:40
> To: user@cassandra.apache.org
> Subject: Too many open files (Cassandra 2.0.1)
>  
> Hi,
>  
> I’ve noticed some nodes in our cluster are dying after some period of time.
>  
> WARN [New I/O server boss #17] 2013-10-29 12:22:20,725 Slf4JLogger.java (line 
> 76) Failed to accept a connection.
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
> at 
> org.jboss.netty.channel.socket.nio.NioServerBoss.process(NioServerBoss.java:100)
> at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> at 
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
>  
> And other exceptions related to the same cause.
> Now, as we use the Cassandra package, the nofile limit is raised to 10.
> To double check if this correct:
>  
> root@de-cass09 ~ # cat /proc/18332/limits
> Limit Soft Limit   Hard Limit   Units
> …
> Max open files10   10   files
> …
>  
> Now I check how many files are open:
> root@de-cass09 ~ # lsof -n -p 18332 | wc -l
> 100038
>  
> This seems an awful a lot for size tiered compaction… ?
> Now I noticed when I checked the list, a (deleted) file passed a lot
>  
> …
> java18332 cassandra 4704r   REG8,1  10911921661 
> 2147483839 /data1/mapdata040/hos/mapdata040-hos-jb-7648-Data.db (deleted)
> java18332 cassandra 4705r   REG8,1  10911921661 
> 2147483839 /data1/mapdata040/hos/mapdata040-hos-jb-7648-Data.db (deleted)
> …
>  
> Actually, if I count specific for this file:
> root@de-cass09 ~ # lsof -n -p 18332 | grep mapdata040-hos-jb-7648-Data.db | 
> wc -l
> 52707
>  
> Other nodes are around a total of 350 files open… Any idea why this nofiles 
> is so high ?
>  
> The first exceptions I see is this:
> WARN [New I/O worker #8] 2013-10-29 12:09:34,440 Slf4JLogger.java (line 76) 
> Unexpected exception in the selector loop.
> java.lang.NullPointerException
> at 
> sun.nio.ch.EPollArrayWrapper.setUpdateEvents(EPollArrayWrapper.java:178)
> at sun.nio.ch.EPollArrayWrapper.add(EPollArrayWrapper.java:227)
> at 
> sun.nio.ch.EPollSelectorImpl.implRegister(EPollSelectorImpl.java:164)
> at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:133)
> at 
> java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:209)
> at 
> org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:151)
> at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
> at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
> at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
> at 
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at java.lang.Thread.run(Thread.java:724)
>  
> Several minutes later I get Too many open files.
>  
> Specs:
> 12-node cluster with Ubuntu 12.04 LTS, Cassandra 2.0.1 (datastax packages), 
> using JBOD of 2 disks.
> JNA enabled.
>  
> Any suggestions?
>  
> Kind regards,
> Pieter Callewaert
>  
> 
>Pieter Callewaert
>Web & IT engineer
>  
>Web:   www.be-mobile.be
>   

OpsCenter not connecting to Cluster

[Nigel LEACH]

Cassandra 2.0.1
OpsCenter 3.2.2
Java 1.7.0_25
RHEL 6.4

This is a new test cluster with just three nodes, two seed nodes, SSL turned 
off, and GossipingPropertyFileSnitch. Pretty much out of the box environment, 
with both Cassandra and OpsCenter via DataStax yum repository.

Cassandra seems fine, and OpsCenter is installed on a seed node. The OpsCenter 
gui comes up, but is unable to connect to the cluster, I get this error

INFO: Starting factory 

INFO:  will retry in 2 
seconds
DEBUG: Problem while pinging node: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/opscenterd/ThriftService.py", 
line 157, in checkThriftConnection
UserError: User aborted connection: Shutdown requested.
WARN: ProcessingError while calling CreateClusterConfController: Unable to 
connect to cluster
INFO: Stopping factory 
 

Not getting very far troubleshooting it, any clues would be much appreciated. 
Should I try installing the OpsCenter agent manually maybe?

Many Thanks, Nigel





___
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorised copying, 
disclosure or distribution of the material in this e-mail is prohibited.

Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional 
disclosures.

Re: IllegalStateException when bootstrapping new nodes

[Robert Coli]

On Tue, Oct 29, 2013 at 7:38 AM, Cyril Scetbon wrote:

> We didn't find the reason why it didn't work but we are wondering if it's
> a BUG. We are using Cassandra 1.2.2 and we resolved the issue with a
> rolling restart of other nodes in the same DC and maybe with some luck...
> We don't know if it's important but we also added the first 2 new nodes
> added to the topology file before we successfully bootstrapped the last 2
> nodes.
>

While this is not directly responsive to your question, I suggest upgrading
from 1.2.2 to at least 1.2.9 ASAP. 1.2.2 contains significant bugs.

=Rob

Re: Example Cassandra 2.0 trigger classes?

[Robert Coli]

On Mon, Oct 28, 2013 at 7:24 PM, Jacob Rhoden  wrote:

> 1. Before I slog my way through the documentation on using triggers? Is
> anyone aware of any real world usage sample code or blogs with examples on
> this?
>

I feel compelled to mention that currently, triggers are explicitly
"experimental" so any "real world" usage should be strongly constrained,
and non-production.

=Rob

RE: OpsCenter not connecting to Cluster

[Pieter Callewaert]

Hi Nigel,

I've currently have a simular problem. However, it has only been reproduced on 
Ubuntu...
Are you using hsha as rpc_server_type?

http://stackoverflow.com/questions/19633980/adding-cluster-error-creating-cluster-call-to-cluster-configs-timed-out
 is a guy with the same problem, showing how to reproduce.
I know the ppl of datastax are now investigating this, but no fix yet...

Kind regards,
Pieter Callewaert

-Original Message-
From: Nigel LEACH [mailto:nigel.le...@uk.bnpparibas.com] 
Sent: dinsdag 29 oktober 2013 18:24
To: user@cassandra.apache.org
Subject: OpsCenter not connecting to Cluster

Cassandra 2.0.1
OpsCenter 3.2.2
Java 1.7.0_25
RHEL 6.4

This is a new test cluster with just three nodes, two seed nodes, SSL turned 
off, and GossipingPropertyFileSnitch. Pretty much out of the box environment, 
with both Cassandra and OpsCenter via DataStax yum repository.

Cassandra seems fine, and OpsCenter is installed on a seed node. The OpsCenter 
gui comes up, but is unable to connect to the cluster, I get this error

INFO: Starting factory 

INFO:  will retry in 2 
seconds
DEBUG: Problem while pinging node: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/opscenterd/ThriftService.py", 
line 157, in checkThriftConnection
UserError: User aborted connection: Shutdown requested.
WARN: ProcessingError while calling CreateClusterConfController: Unable to 
connect to cluster
INFO: Stopping factory 
 

Not getting very far troubleshooting it, any clues would be much appreciated. 
Should I try installing the OpsCenter agent manually maybe?

Many Thanks, Nigel





___
This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and delete this e-mail. Any unauthorised copying, 
disclosure or distribution of the material in this e-mail is prohibited.

Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional 
disclosures.

Cassandra 1.1.6 - New node bootstrap not completing

[Narendra Sharma]

We had a cluster of 4 nodes in AWS. The average load on each node was
approx 750GB. We added 4 new nodes. It is now more than 30 hours and the
node is still in JOINING mode.
Specifically I am analyzing the one with IP 10.3.1.29. There is no
compaction or streaming or index building happening.

$ ./nodetool ring
Note: Ownership information does not include topology, please specify a
keyspace.
Address DC  RackStatus State   LoadOwns
   Token

   148873535527910577765226390751398592512
10.3.1.179  datacenter1 rack1   Up Normal  740.41 GB
25.00%  0
10.3.1.29   datacenter1 rack1   Up Joining 562.49 GB
0.00%   21267647932558653966460912964485513215
10.3.1.175  datacenter1 rack1   Up Normal  755.7 GB
 25.00%  42535295865117307932921825928971026431
10.3.1.30   datacenter1 rack1   Up Joining 565.68 GB
0.00%   63802943797675961899382738893456539648
10.3.1.177  datacenter1 rack1   Up Normal  754.18 GB
25.00%  85070591730234615865843651857942052863
10.3.1.135  datacenter1 rack1   Up Normal  95.97 GB
 20.87%  120580289963820081458352857409882669785
10.3.1.178  datacenter1 rack1   Up Normal  747.53 GB
4.13%   127605887595351923798765477786913079295
10.3.1.24   datacenter1 rack1   Up Joining 522.09 GB
0.00%   148873535527910577765226390751398592512
$ ./nodetool netstats
Mode: JOINING
Not sending any streams.
 Nothing streaming from /10.3.1.177
 Nothing streaming from /10.3.1.179
Pool NameActive   Pending  Completed
Commandsn/a 0 82
Responses   n/a 0   40135123
$ ./nodetool compactionStats
pending tasks: 0
Active compaction remaining time :n/a
$ ./nodetool info
Token: 21267647932558653966460912964485513215
Gossip active: true
Thrift active: false
Load : 562.49 GB
Generation No: 1382981644
Uptime (seconds) : 90340
Heap Memory (MB) : 9298.59 / 13272.00
Data Center  : datacenter1
Rack : rack1
Exceptions   : 2
Key Cache: size 104857584 (bytes), capacity 104857584 (bytes),
187373 hits, 94709046 requests, 0.002 recent hit rate, 14400 save period in
seconds
Row Cache: size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests,
NaN recent hit rate, 0 save period in seconds


The 2 Exceptions in info output are the ones that were logged when I
stopped index build to let bootstrap complete faster.

Any clue whats wrong and where should I look for to further analyze the
issue? I haven't restarted the Cassandra process. I am afraid the node will
start bootstrap again if I restart the node.

Thanks,
Naren



-- 
Narendra Sharma
Software Engineer
*http://www.aeris.com*
*http://narendrasharma.blogspot.com/*

Re: Cassandra 1.1.6 - New node bootstrap not completing

[Robert Coli]

On Tue, Oct 29, 2013 at 11:45 AM, Narendra Sharma  wrote:

> We had a cluster of 4 nodes in AWS. The average load on each node was
> approx 750GB. We added 4 new nodes. It is now more than 30 hours and the
> node is still in JOINING mode.
> Specifically I am analyzing the one with IP 10.3.1.29. There is no
> compaction or streaming or index building happening.
>

If your cluster has RF>2, you are bootstrapping two nodes into the same
range simultaneously. That is not supported. [1,2] The node you are having
the problem with is in the range that is probably overlapping.

If I were you I would :

1) stop all "Joining" nodes and wipe their state including system keyspace
2) optionally "removetoken" any nodes which remain in cluster gossip state
after stopping
3) re-start/bootstrap them one at a time, waiting for each to complete
bootstrapping before starting the next  one
4) (unrelated) Upgrade from 1.1.6 to the head of 1.1.x ASAP.

=Rob
[1] https://issues.apache.org/jira/browse/CASSANDRA-2434
[2]
https://issues.apache.org/jira/browse/CASSANDRA-2434?focusedCommentId=13091851&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13091851

not even number of keys per CFs in fully balanced cluster with random partitioner

[Piavlo]

 Hi,

There is a 12 node cluster , still stuck on 1.0.8.
All nodes in the cluster ring are balanced.
Using random partitioner.
All CFs use compression.
Data size on nodes varies from 40G to 75G.
This variance is not due to the bigger nodes having more uncompacted 
sstables than others.
Most biggest CFs have exact same row keys, just store different data, so 
data for same same key should end up on same node for these CFs.
The keys estimate for each of these biggest CF on the nodes with larger 
data size  is almost twice larger than key estimate on the nodes with 
smallest data size, thus proportional to the data size on the node. 
These CFs have about 50-100 millions for rows per node.


I can't understand how statistically it's possible that with random 
partitioner some nodes have x2 more keys than others with 50-100 
millions of keys per node.

Any ideas how it's possible?
Anything else I can check?

tnx
Alex

Re: not even number of keys per CFs in fully balanced cluster with random partitioner

[Robert Coli]

On Tue, Oct 29, 2013 at 4:09 PM, Piavlo  wrote:

> I can't understand how statistically it's possible that with random
> partitioner some nodes have x2 more keys than others with 50-100 millions
> of keys per node.
>

1) you have very high row variance and really really bad luck with the
distribution of your super-huge rows
OR
2) you have repaired some nodes and not others, and have dramatically
over-repaired the repaired ones, possibly due to f/e TTL + repair issues in
1.0.x

The simple way to see if #2 is the case is to run a major compaction on a
good and a bad node. They should end up the same size.

There are operational implications to doing a major compaction, however, so
you might want to investigate the use of checksstablegarbage [1].

=Rob
 [1] https://github.com/cloudian/support-tools

Recovering from mystical cassandra errors.

[Jacob Rhoden]

Hi Guys,

Im testing out Cassandra with a 2.0.1 install, just like with 1.2.8 I last 
tested, from time to time I am coming across errors that prevent Cassandra 
starting up.

For example: Yesterday I installed this instance, created and removed some 
“tables” using cql3 and inserted/updated/truncated some tables. Overnight, not 
sure why, cassandra stopped running and won’t start up again:

 INFO 10:34:37,776 Finished reading 
/cassandra/commitlog/CommitLog-3-1382935041799.log
 INFO 10:34:37,776 Replaying /cassandra/commitlog/CommitLog-3-1383089534960.log
 INFO 10:34:37,796 Finished reading 
/cassandra/commitlog/CommitLog-3-1383089534960.log
 INFO 10:34:37,796 Skipped 11 mutations from unknown (probably removed) CF with 
id 941b25fd-703f-3013-aadd-40b18cbcd9cd
ERROR 10:34:37,797 Exception encountered during startup
java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
at 
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411)
at 
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400)
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.blockForWrites(CommitLogReplayer.java:105)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:147)
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299)
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Caused by: java.util.concurrent.ExecutionException: 
java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at 
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407)
... 7 more

Normally I just delete the files and restart the node. If you are in a 
production environment, how do you deal with this type of issue?

(I suspect there may be some sort of bug with replaying commit logs in 2.0.1, 
probably associated with replaying a commit log after you have deleted a table, 
and recreated with a different definition.)

Best regards,
Jacob

Re: not even number of keys per CFs in fully balanced cluster with random partitioner

[Robert Coli]

On Tue, Oct 29, 2013 at 4:39 PM, Robert Coli  wrote:

> On Tue, Oct 29, 2013 at 4:09 PM, Piavlo  wrote:
>
>> I can't understand how statistically it's possible that with random
>> partitioner some nodes have x2 more keys than others with 50-100 millions
>> of keys per node.
>>
>
> 1) you have very high row variance and really really bad luck with the
> distribution of your super-huge rows
>

Oh, herp, you specified that you have a number-of-keys difference. So it
can't be this one.

But..

First, that number of keys is the number of keys *SEEN BY COMPACTION SINCE
NODE STARTUP* so different runtimes can skew numbers.

Second, that number of keys is the number of times the key has been seen in
all sstables. A single row key with fragments in 5 sstables = "5".

Have you added nodes to the cluster and not run a cleanup compaction?

=Rob

Re: Recovering from mystical cassandra errors.

[Robert Coli]

On Tue, Oct 29, 2013 at 4:40 PM, Jacob Rhoden  wrote:

> Im testing out Cassandra with a 2.0.1 install [and having this issue where
> nodes won't start after being stopped]
>

Someone else having what looks like this same issue (in C* 2.0.2 iirc) came
into #cassandra yesterday.

I encourage you to :

1) search the apache cassandra JIRA for an existing ticket on this problem
2) if one does not exist, file a ticket including the error log and as much
info to help reproduce the issue as possible


> Normally I just delete the files and restart the node. If you are in a
> production environment, how do you deal with this type of issue?
>

By not running X.Y.Z in production where Z<= 5.

https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/

But seriously, in general people nuke node state until the node starts, and
then replace/repair the node.


> (I suspect there may be some sort of bug with replaying commit logs in
> 2.0.1, probably associated with replaying a commit log after you have
> deleted a table, and recreated with a different definition.)
>

This problem exists in general when you DROP and reCREATE tables and
keyspaces.

The workaround is to use TRUNCATE and then ALTER instead of DROP and CREATE.

https://issues.apache.org/jira/browse/CASSANDRA-4857
which references
https://issues.apache.org/jira/browse/CASSANDRA-4221

=Rob

ReadCount change rate is different across nodes

[Daning Wang]

We are running 1.2.5 on 8 nodes(256 tokens). all the nodes are running on
same type of machine. and db size is about same. but recently we checked
ReadCount stats through jmx, and found that some nodes got  3 times change
rate(we have calculated the changes per minute)  than others.

We are using hector on client side, and clients are connecting to all the
servers, we checked open connections on each server, the numbers are about
same.

What could cause this problem, how to debug this?


Thanks in advance,

Daning

heap issues - looking for advices on gc tuning

[Piavlo]

Hi,

Below I try to give a full picture to the problem I'm facing.

This is a 12 node cluster, running on ec2 with m2.xlarge instances (17G 
ram , 2 cpus).

Cassandra version is 1.0.8
Cluster normally having between 3000 - 1500 reads per second (depends on 
time of the day) and 1700 - 800 writes per second- according to Opscetner.

RF=3, now row caches are used.

Memory relevant  configs from cassandra.yaml:
flush_largest_memtables_at: 0.85
reduce_cache_sizes_at: 0.90
reduce_cache_capacity_to: 0.75
commitlog_total_space_in_mb: 4096

relevant JVM options used are:
-Xms8000M -Xmx8000M -Xmn400M
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
-XX:MaxTenuringThreshold=1

-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly"

Now what happens is that with these settings after cassandra process 
restart, the GC it working fine at the beginning, and heap used looks like a
saw with perfect teeth, eventually the teeth size start to diminish 
until the teeth become not noticable, and then cassandra starts to spend 
lot's of CPU time
doing gc. It takes about 2 weeks until for such cycle , and then I need 
to restart cassandra process to improve performance.
During all this time there are no memory  related messages in cassandra 
system.log, except a "GC for ParNew: little above 200ms" once in a while.


Things i've already done trying to reduce this eventual heap pressure.
1) reducing bloom_filter_fp_chance  resulting in reduction from ~700MB 
to ~280MB total per node based on all Filter.db files on the node.
2) reducing key cache sizes, and dropping key_caches for CFs which do no 
not have many reads

3) the heap size was increased from 7000M to 8000M
All these have not really helped , just the increase from 7000M to 
8000M, helped in increase the cycle till excessive gc from ~9 days to 
~14 days.


I've tried to graph overtime the data that is supposed to be in heap vs 
actual heap size, by summing up all CFs bloom filter sizes + all CFs key 
cache capacities multipled by average key size + all CFs memtables data 
size reported (i've overestimated the data size a bit on purpose to be 
on the safe size).
Here is a link to graph showing last 2 day metrics for a node which 
could not effectively do GC, and then cassandra process was restarted.

http://awesomescreenshot.com/0401w5y534
You can clearly see that before and after restart, the size of data that 
is in supposed to be in heap, is the same pretty much the same,

which makes me think that I really need is GC tunning.

Also I suppose that this is not due to number of total keys each node 
has , which is between 300 - 200 milions keys for all CF key estimates 
summed on a code.
The nodes have datasize between 75G to 45G  accordingly to milions of 
keys. And all nodes are starting to have having GC heavy load after 
about 14 days.
Also the excessive GC and heap usage are not affected by load which 
varies depending on time of the day (see read/write rates at the 
beginning of the mail).
So again based on this , I assume this is not due to large number of 
keys or too much load on the cluster,  but due to a pure GC 
misconfiguration issue.


Things I remember that I've tried for GC tunning:
1) Changing -XX:MaxTenuringThreshold=1 to values like 8 - did not help.
2) Adding  -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing 
-XX:CMSIncrementalDutyCycleMin=0
  -XX:CMSIncrementalDutyCycle=10 
-XX:ParallelGCThreads=2 JVM_OPTS -XX:ParallelCMSThreads=1

this actually made things worse.
3) Adding -XX:-XX-UseAdaptiveSizePolicy -XX:SurvivorRatio=8 - did not help.

Also since it takes like 2 weeks to verify that changing GC setting did 
not help, the process is painfully slow to try all the possibilities :)

I'd highly appreciate any help and hints on the GC tunning.

tnx
Alex

Re: Cassandra 1.1.6 - New node bootstrap not completing

[Narendra Sharma]

Thanks Robert.

I didn't realize that some of the keyspaces (not all and esp. the biggest
one I was focusing on) had RF > 2. I wasted 3 days on it. Thanks again for
the pointers. I will try again and share the results.


On Wed, Oct 30, 2013 at 12:28 AM, Robert Coli  wrote:

> On Tue, Oct 29, 2013 at 11:45 AM, Narendra Sharma <
> narendra.sha...@gmail.com> wrote:
>
>> We had a cluster of 4 nodes in AWS. The average load on each node was
>> approx 750GB. We added 4 new nodes. It is now more than 30 hours and the
>> node is still in JOINING mode.
>> Specifically I am analyzing the one with IP 10.3.1.29. There is no
>> compaction or streaming or index building happening.
>>
>
> If your cluster has RF>2, you are bootstrapping two nodes into the same
> range simultaneously. That is not supported. [1,2] The node you are having
> the problem with is in the range that is probably overlapping.
>
> If I were you I would :
>
> 1) stop all "Joining" nodes and wipe their state including system keyspace
> 2) optionally "removetoken" any nodes which remain in cluster gossip state
> after stopping
> 3) re-start/bootstrap them one at a time, waiting for each to complete
> bootstrapping before starting the next  one
> 4) (unrelated) Upgrade from 1.1.6 to the head of 1.1.x ASAP.
>
> =Rob
> [1] https://issues.apache.org/jira/browse/CASSANDRA-2434
> [2]
> https://issues.apache.org/jira/browse/CASSANDRA-2434?focusedCommentId=13091851&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13091851
>



-- 
Narendra Sharma
Software Engineer
*http://www.aeris.com*
*http://narendrasharma.blogspot.com/*