Hi All,
I have a 2 node cluster(32gb ram/8cpu) running 3.0.10 and I created 50
keyspaces in it. Each keyspace has 25 CF. Column count in each CF ranges
between 5 to 30.
I am getting few issues once keyspace count reaches ~50.
*Issue 1:*
When I try to use cqlsh, I get timeout.
*$ cqlsh `hostname -i`*
*Connection error: ('Unable to connect to any servers', {'10.0.20.220':
OperationTimedOut('errors=None, last_host=None',)})*
If I increase connect timeout, I am able to access cluster through cqlsh
*$ cqlsh --connect-timeout 20 `hostname -i //this works fine*
*Question: *Does C* reads some schema/metadata on calling cqlsh, which is
causing timeout with large number of keyspaces?
*Issue 2:*
If I create keyspaces which have 3 large CF(each having around 2500 cols),
then I start to see schema agreement timeout in my logs. I have set schema
agreement timeout to 30 seconds in driver.
*2016-12-13 08:37:02.733 | gbd-std-01 | WARN | cluster2-worker-194 |
com.datastax.driver.core.Cluster | Error while waiting for schema agreement*
*Question:* Can a single C* cluster of 5 nodes(32gb/8cpu each) support upto
500 keyspaces each having 25 CFs. What kind of issues I can expect?
*Issue 3:*
I am creating keyspaces and CFs through datastax driver. I see following
exception in my log after reaching *~50 keyspaces.*
*Question: *What is the effect of below exception?
2016-12-19 13:55:35.615 | gbd-std-01 | ERROR | cluster1-worker-147 |
*com.datastax.driver.core.ControlConnection
| [Control connection] Unexpected error while refreshing schema*
*java.util.concurrent.ExecutionException:
com.datastax.driver.core.exceptions.OperationTimedOutException:
[gbd-cass-20.ec2-east1.hidden.com/10.0.20.220
<http://gbd-cass-20.ec2-east1.hidden.com/10.0.20.220>] Operation timed out*
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[com.google.guava.guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[com.google.guava.guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[com.google.guava.guava-18.0.jar:na]
at com.datastax.driver.core.SchemaParser.get(SchemaParser.java:467)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.SchemaParser.access$400(SchemaParser.java:30)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.SchemaParser$V3SchemaParser.fetchSystemRows(SchemaParser.java:632)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:56)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:341)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:306)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Cluster$Manager$SchemaRefreshRequestDeliveryCallback$1.runMayThrow(Cluster.java:2570)
[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32)
[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: com.datastax.driver.core.exceptions.OperationTimedOutException: [
gbd-cass-20.ec2-east1.hidden.com/10.0.20.220] Operation timed out
at
com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:209)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1260)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:581)
~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:655)
~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367)
~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final]
... 1 common frames omitted
2016-12-19 13:55:39.885 | gbd-std-01 | ERROR | cluster2-worker-124 |
*com.datastax.driver.core.ControlConnection
| [Control connection] Unexpected error while refreshing schema*
*java.util.concurrent.ExecutionException:
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout
during read query at consistency ONE (1 responses were required but only 0
replica responded)*
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[com.google.guava.guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[com.google.guava.guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[com.google.guava.guava-18.0.jar:na]
at com.datastax.driver.core.SchemaParser.get(SchemaParser.java:467)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.SchemaParser.access$400(SchemaParser.java:30)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.SchemaParser$V3SchemaParser.fetchSystemRows(SchemaParser.java:632)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:56)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:341)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:306)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Cluster$Manager$SchemaRefreshRequestDeliveryCallback$1.runMayThrow(Cluster.java:2570)
[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32)
[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Best Regards,
Saumitra
