Sebastian,Thank you so much for providing detailed explanation. I still have
some questions and I need to provide some clarifications
1. We do not have code that is creating the tables dynamically. All DDL
operations are done through Datastax DevCenter tool. When you say schema to
settle, do you means we provide proper consistency level? I don't think there
is a provision to do that in tool. Or I can change the SYSTEM KEYSPACE
definition of replication factor equal to number of nodes?
2. In the steps described below for correcting this problem - when you say move
data from old directory to new, do you mean move the .db file? It will override
the current file right?
3. Do we have to rename the directory name to remove CFID i.e. just column
family name without CFID? After that, update the System table as well? Naidu
Saladi
From: Sebastian Estevez <sebastian.este...@datastax.com>
To: user@cassandra.apache.org; Saladi Naidu <naidusp2...@yahoo.com>
Sent: Friday, July 10, 2015 5:25 PM
Subject: Re: DROP Table
#1 The cause of this problem is a CREATE TABLE statement collision. Do not
generate tables dynamically from multiple clients, even with IF NOT EXISTS.
First thing you need to do is fix your code so that this does not happen. Just
create your tables manually from cqlsh allowing time for the schema to settle.
#2 Here's the fix:
1) Change your code to not automatically re-create tables (even with IF NOT
EXISTS).
2) Run a rolling restart to ensure schema matches across nodes. Run nodetool
describecluster around your cluster. Check that there is only one schema
version.
ON EACH NODE:3) Check your filesystem and see if you have two directories for
the table in question in the data directory.
If THERE ARE TWO OR MORE DIRECTORIES:4)Identify from schema_column_families
which cf ID is the "new" one (currently in use).
cqlsh -e "select * from system.schema_column_families"|grep <table name>
5) Move the data from the "old" one to the "new" one and remove the old
directory.
6) If there are multiple "old" ones repeat 5 for every "old" directory.
7) run nodetool refresh
IF THERE IS ONLY ONE DIRECTORY:
No further action is needed.
All the best,
Sebastián EstévezSolutions Architect | 954 905 8615 |
sebastian.este...@datastax.com
DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any size.
With more than 500 customers in 45 countries, DataStax is the database
technology and transactional backbone of choice for the worlds most innovative
companies such as Netflix, Adobe, Intuit, and eBay.
On Fri, Jul 10, 2015 at 12:15 PM, Saladi Naidu <naidusp2...@yahoo.com> wrote:
My understanding is that Cassandra File Structure follows below naming
convention
/cassandra/data/ <key-spaces> <table>
Whereas our file structure is as below, each table has multiple names and when
we drop tables and recreate these directories remain. Also when we dropped the
table one node was down, when it came back, we tried to do Nodetool repair and
repair kept failing referring to CFID error listed below
drwxr-xr-x. 16 cass cass 4096 May 24 06:49 ../drwxr-xr-x. 4 cass cass 4096 Jul
2 11:09application_by_user-e0eec95019a211e58b954ffc8e9bfaa6/drwxr-xr-x. 2
cass cass 4096 Jun 25
10:15application_info-4dba2bf0054f11e58b954ffc8e9bfaa6/drwxr-xr-x. 4 cass cass
4096 Jul 2 11:09application_info-a0ee65d019a311e58b954ffc8e9bfaa6/drwxr-xr-x.
4 cass cass 4096 Jul 2
11:09configproperties-228ea2e0c13811e4aa1d4ffc8e9bfaa6/drwxr-xr-x. 4 cass cass
4096 Jul 2 11:09user_activation-95d005f019a311e58b954ffc8e9bfaa6/drwxr-xr-x.
3 cass cass 4096 Jun 25
10:16user_app_permission-9fddcd62ffbe11e4a25a45259f96ec68/drwxr-xr-x. 4 cass
cass 4096 Jul 2
11:09user_credential-86cfff1019a311e58b954ffc8e9bfaa6/drwxr-xr-x. 4 cass cass
4096 Jul 2 11:09user_info-2fa076221b1011e58b954ffc8e9bfaa6/drwxr-xr-x. 2 cass
cass 4096 Jun 25 10:15user_info-36028c00054f11e58b954ffc8e9bfaa6/drwxr-xr-x. 3
cass cass 4096 Jun 25
10:15user_info-fe1d7b101a5711e58b954ffc8e9bfaa6/drwxr-xr-x. 4 cass cass 4096
Jun 25 10:16user_role-9ed0ca30ffbe11e4b71d09335ad2d5a9/
WARN [Thread-2579] 2015-07-02 16:02:27,523 IncomingTcpConnection.java:91
-UnknownColumnFamilyException reading from socket;
closingorg.apache.cassandra.db.UnknownColumnFamilyException:Couldn't
findcfId=218e3c90-1b0e-11e5-a34b-d7c17b3e318a
atorg.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)~[apache-cassandra-2.1.2.jar:2.1.2]
atorg.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)~[apache-cassandra-2.1.2.jar:2.1.2]
atorg.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)~[apache-cassandra-2.1.2.jar:2.1.2]
atorg.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)~[apache-cassandra-2.1.2.jar:2.1.2]
atorg.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)~[apache-cassandra-2.1.2.jar:2.1.2]
Naidu Saladi
