Hi,
Question: Does C* reads some schema/metadata on calling cqlsh, which is causing
timeout with large number of keyspaces?
A lot ). cqlsh reads schemas, cluster topology, each node tokens, etc. You can
just capture TCP port 9042 (unless you use SSL) and view all negotiation
between cqlsh and node.
Question: Can a single C* cluster of 5 nodes(32gb/8cpu each) support upto 500
keyspaces each having 25 CFs. What kind of issues I can expect?
You have 500*25 = 12500 tables, it's huge number. Each CF takes at least 1M of
heap memory. So it needs 12G heap only for starting usage. Make test on one-two
node cluster.
Question: What is the effect of below exception?
Is keyspaces created despite exception or no?
Best regards, Vladimir Yudovin,
Winguzone - Cloud Cassandra Hosting
---- On Mon, 19 Dec 2016 10:24:20 -0500 Saumitra S
<saumitra.srivast...@gmail.com> wrote ----
Hi All,
I have a 2 node cluster(32gb ram/8cpu) running 3.0.10 and I created 50
keyspaces in it. Each keyspace has 25 CF. Column count in each CF ranges
between 5 to 30.
I am getting few issues once keyspace count reaches ~50.
Issue 1:
When I try to use cqlsh, I get timeout.
$ cqlsh `hostname -i`
Connection error: ('Unable to connect to any servers', {'10.0.20.220':
OperationTimedOut('errors=None, last_host=None',)})
If I increase connect timeout, I am able to access cluster through cqlsh
$ cqlsh --connect-timeout 20 `hostname -i //this works fine
Question: Does C* reads some schema/metadata on calling cqlsh, which is causing
timeout with large number of keyspaces?
Issue 2:
If I create keyspaces which have 3 large CF(each having around 2500 cols), then
I start to see schema agreement timeout in my logs. I have set schema agreement
timeout to 30 seconds in driver.
2016-12-13 08:37:02.733 | gbd-std-01 | WARN | cluster2-worker-194 |
com.datastax.driver.core.Cluster | Error while waiting for schema agreement
Question: Can a single C* cluster of 5 nodes(32gb/8cpu each) support upto 500
keyspaces each having 25 CFs. What kind of issues I can expect?
Issue 3:
I am creating keyspaces and CFs through datastax driver. I see following
exception in my log after reaching ~50 keyspaces.
Question: What is the effect of below exception?
2016-12-19 13:55:35.615 | gbd-std-01 | ERROR | cluster1-worker-147 |
com.datastax.driver.core.ControlConnection | [Control connection] Unexpected
error while refreshing schema
java.util.concurrent.ExecutionException:
com.datastax.driver.core.exceptions.OperationTimedOutException:
[gbd-cass-20.ec2-east1.hidden.com/10.0.20.220] Operation timed out
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[com.google.guava.guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[com.google.guava.guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[com.google.guava.guava-18.0.jar:na]
at com.datastax.driver.core.SchemaParser.get(SchemaParser.java:467)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.SchemaParser.access$400(SchemaParser.java:30)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.SchemaParser$V3SchemaParser.fetchSystemRows(SchemaParser.java:632)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:56)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:341)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:306)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Cluster$Manager$SchemaRefreshRequestDeliveryCallback$1.runMayThrow(Cluster.java:2570)
[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32)
[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: com.datastax.driver.core.exceptions.OperationTimedOutException:
[gbd-cass-20.ec2-east1.hidden.com/10.0.20.220] Operation timed out
at
com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:209)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1260)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:581)
~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:655)
~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final]
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367)
~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final]
... 1 common frames omitted
2016-12-19 13:55:39.885 | gbd-std-01 | ERROR | cluster2-worker-124 |
com.datastax.driver.core.ControlConnection | [Control connection] Unexpected
error while refreshing schema
java.util.concurrent.ExecutionException:
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout
during read query at consistency ONE (1 responses were required but only 0
replica responded)
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[com.google.guava.guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[com.google.guava.guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[com.google.guava.guava-18.0.jar:na]
at com.datastax.driver.core.SchemaParser.get(SchemaParser.java:467)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.SchemaParser.access$400(SchemaParser.java:30)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.SchemaParser$V3SchemaParser.fetchSystemRows(SchemaParser.java:632)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:56)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:341)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:306)
~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Cluster$Manager$SchemaRefreshRequestDeliveryCallback$1.runMayThrow(Cluster.java:2570)
[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32)
[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Best Regards,
Saumitra
