Ya, that happens when some operation throws a time out or any other sort of operation (connection refuse, etc). There is a failback logic that will try to discover all the nodes within the Cluster (not only the ones you configured) in order to reach the cluster and execution the operation.
Have you seen the log of Cassandra? You might be having a problem on your client when a Compactation is kicked. Doing so, all your connections to the node will be much slower. This may introduce a a file handling problem, since your client will continue to stack up connections against Cassandra. If you see that this is the problem that you are having, see how you can manage to raise such limitation to a higher value that will let you get through Cassandra compactation. Kani On Thu, Dec 16, 2010 at 2:48 PM, Germán Kondolf <german.kond...@gmail.com>wrote: > Indeed Hector has a connection pool behind it, I think it uses 50 > connectios per node. > But also uses a node to discover the others, I assume that, as I saw > connections from my app to nodes that I didn't configure in Hector. > > So, you may check the fds in OS level to see if there is a bottleneck > there. > > On Thu, Dec 16, 2010 at 2:39 PM, Amin Sakka, Novapost > <amin.sa...@novapost.fr> wrote: > > > > I'm using a unique client instance (using Hector) and a unique connection > to > > cassandra. > > For each insertion I'm using a new mutator and then I release it. > > I have 473 sstable "Data.db", the average size of each is 30Mo. > > > > > > > > 2010/12/16 Ryan King <r...@twitter.com> > >> > >> Are you creating a new connection for each row you insert (and if so > >> are you closing it)? > >> > >> -ryan > >> > >> On Wed, Dec 15, 2010 at 8:13 AM, Amin Sakka, Novapost > >> <amin.sa...@novapost.fr> wrote: > >> > Hello, > >> > I'm using cassandra 0.7.0 rc1, a single node configuration, > replication > >> > factor 1, random partitioner, 2 GO heap size. > >> > I ran my hector client to insert 5.000.000 rows but after a couple of > >> > hours, > >> > the following Exception occurs : > >> > > >> > WARN [main] 2010-12-15 16:38:53,335 CustomTThreadPoolServer.java > (line > >> > 104) > >> > Transport error occurred during acceptance of message. > >> > org.apache.thrift.transport.TTransportException: > >> > java.net.SocketException: > >> > Too many open files > >> > at > >> > > >> > > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124) > >> > at > >> > > >> > > org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:67) > >> > at > >> > > >> > > org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:38) > >> > at > >> > > >> > > org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) > >> > at > >> > > >> > > org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:98) > >> > at > >> > > >> > > org.apache.cassandra.thrift.CassandraDaemon.start(CassandraDaemon.java:120) > >> > at > >> > > >> > > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:229) > >> > at > >> > > >> > > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134) > >> > Caused by: java.net.SocketException: Too many open files > >> > at java.net.PlainSocketImpl.socketAccept(Native Method) > >> > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) > >> > at java.net.ServerSocket.implAccept(ServerSocket.java:453) > >> > at java.net.ServerSocket.accept(ServerSocket.java:421) > >> > at > >> > > >> > > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119) > >> > > >> > When I try to restart Cassandra, I have the following exception : > >> > > >> > ERROR 16:42:26,573 Exception encountered during startup. > >> > java.lang.ArithmeticException: / by zero > >> > at > >> > > >> > > org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:233) > >> > at > >> > > >> > > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:284) > >> > at > >> > > >> > > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:200) > >> > at > >> > > >> > > org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:225) > >> > at > >> > > >> > > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:449) > >> > at > >> > > >> > > org.apache.cassandra.db.ColumnFamilyStore.addIndex(ColumnFamilyStore.java:306) > >> > at > >> > > >> > > org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:246) > >> > at > >> > > >> > > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:449) > >> > at > >> > > >> > > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:437) > >> > at org.apache.cassandra.db.Table.initCf(Table.java:341) > >> > at org.apache.cassandra.db.Table.<init>(Table.java:283) > >> > at org.apache.cassandra.db.Table.open(Table.java:114) > >> > at > >> > > >> > > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:138) > >> > at > >> > > >> > > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55) > >> > at > >> > > >> > > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:216) > >> > at > >> > > >> > > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134) > >> > > >> > I am looking for advice on how to debug this. > >> > > >> > Thanks, > >> > -- > >> > > >> > Amin > >> > > >> > > >> > > >> > > >> > > > > > > > > > -- > > Amin > > > > > > > > > > > > -- > //GK > german.kond...@gmail.com > // sites > http://twitter.com/germanklf > http://www.facebook.com/germanklf > http://ar.linkedin.com/in/germankondolf >
