>I want to dig deeper into what all things happen in C* at time of CF creation
It starts somewhere in MigrationManager.announceNewColumnFamily function, I guess. >imitation of number of keyspaces which can be created. Actually it's CF limitation, not keyspaces. >if you can also point me to the this 1MB per CF thingy, it would be great. Look at http://www.mail-archive.com/user@cassandra.apache.org/msg46359.html, CASSANDRA-5935, CASSANDRA-2252 In source look at SlabAllocator.REGION_SIZE definition. Best regards, Vladimir Yudovin, Winguzone - Cloud Cassandra Hosting ---- On Mon, 19 Dec 2016 14:10:37 -0500 Saumitra S <saumitra.srivast...@gmail.com> wrote ---- Hi Vladimir, Thanks for the response. When I see "com.datastax.driver.core.ControlConnection" exceptions, I see that keyspaces and CF are created. But when I create CF with large number of columns(2400 cols) quickly one after the other(with 2 seconds gap between CREATE TABLE queries), I get schema agreement timeout errors ( com.datastax.driver.core.Cluster | Error while waiting for schema agreement). This happens even with a clean slate(empty data directory), just after creating 4 keyspaces. Timeout is set to 30 seconds. Please note that CREATE TABLE queries are NOT fired in parallel. I wait for 1 query to complete(with schema agreement) before firing another one. I want to dig deeper into what all things happen in C* at time of CF creation to understand more about the limitation of number of keyspaces which can be created. Can you please point me to the corresponding source code? Specifically if you can also point me to the this 1MB per CF thingy, it would be great. Best Regards, Saumitra On Mon, Dec 19, 2016 at 11:41 PM, Vladimir Yudovin <vla...@winguzone.com> wrote: Hi, Question: Does C* reads some schema/metadata on calling cqlsh, which is causing timeout with large number of keyspaces? A lot ). cqlsh reads schemas, cluster topology, each node tokens, etc. You can just capture TCP port 9042 (unless you use SSL) and view all negotiation between cqlsh and node. Question: Can a single C* cluster of 5 nodes(32gb/8cpu each) support upto 500 keyspaces each having 25 CFs. What kind of issues I can expect? You have 500*25 = 12500 tables, it's huge number. Each CF takes at least 1M of heap memory. So it needs 12G heap only for starting usage. Make test on one-two node cluster. Question: What is the effect of below exception? Is keyspaces created despite exception or no? Best regards, Vladimir Yudovin, Winguzone - Cloud Cassandra Hosting ---- On Mon, 19 Dec 2016 10:24:20 -0500 Saumitra S <saumitra.srivast...@gmail.com> wrote ---- Hi All, I have a 2 node cluster(32gb ram/8cpu) running 3.0.10 and I created 50 keyspaces in it. Each keyspace has 25 CF. Column count in each CF ranges between 5 to 30. I am getting few issues once keyspace count reaches ~50. Issue 1: When I try to use cqlsh, I get timeout. $ cqlsh `hostname -i` Connection error: ('Unable to connect to any servers', {'10.0.20.220': OperationTimedOut('errors=None, last_host=None',)}) If I increase connect timeout, I am able to access cluster through cqlsh $ cqlsh --connect-timeout 20 `hostname -i //this works fine Question: Does C* reads some schema/metadata on calling cqlsh, which is causing timeout with large number of keyspaces? Issue 2: If I create keyspaces which have 3 large CF(each having around 2500 cols), then I start to see schema agreement timeout in my logs. I have set schema agreement timeout to 30 seconds in driver. 2016-12-13 08:37:02.733 | gbd-std-01 | WARN | cluster2-worker-194 | com.datastax.driver.core.Cluster | Error while waiting for schema agreement Question: Can a single C* cluster of 5 nodes(32gb/8cpu each) support upto 500 keyspaces each having 25 CFs. What kind of issues I can expect? Issue 3: I am creating keyspaces and CFs through datastax driver. I see following exception in my log after reaching ~50 keyspaces. Question: What is the effect of below exception? 2016-12-19 13:55:35.615 | gbd-std-01 | ERROR | cluster1-worker-147 | com.datastax.driver.core.ControlConnection | [Control connection] Unexpected error while refreshing schema java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.OperationTimedOutException: [gbd-cass-20.ec2-east1.hidden.com/10.0.20.220] Operation timed out at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[com.google.guava.guava-18.0.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[com.google.guava.guava-18.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-18.0.jar:na] at com.datastax.driver.core.SchemaParser.get(SchemaParser.java:467) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.SchemaParser.access$400(SchemaParser.java:30) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.SchemaParser$V3SchemaParser.fetchSystemRows(SchemaParser.java:632) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:56) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:341) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:306) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.Cluster$Manager$SchemaRefreshRequestDeliveryCallback$1.runMayThrow(Cluster.java:2570) [com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) [com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] Caused by: com.datastax.driver.core.exceptions.OperationTimedOutException: [gbd-cass-20.ec2-east1.hidden.com/10.0.20.220] Operation timed out at com.datastax.driver.core.DefaultResultSetFuture.onTimeout(DefaultResultSetFuture.java:209) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1260) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:581) ~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:655) ~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final] at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367) ~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final] ... 1 common frames omitted 2016-12-19 13:55:39.885 | gbd-std-01 | ERROR | cluster2-worker-124 | com.datastax.driver.core.ControlConnection | [Control connection] Unexpected error while refreshing schema java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[com.google.guava.guava-18.0.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[com.google.guava.guava-18.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[com.google.guava.guava-18.0.jar:na] at com.datastax.driver.core.SchemaParser.get(SchemaParser.java:467) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.SchemaParser.access$400(SchemaParser.java:30) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.SchemaParser$V3SchemaParser.fetchSystemRows(SchemaParser.java:632) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:56) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:341) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:306) ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.Cluster$Manager$SchemaRefreshRequestDeliveryCallback$1.runMayThrow(Cluster.java:2570) [com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) [com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] Best Regards, Saumitra
