Thank you Mikhail. I'll take a look at your utility. On Jul 13, 2015 5:30 PM, "Mikhail Strebkov" <streb...@gmail.com> wrote:
> Hi Saladi, > > Recently I faced a similar problem, I had a lot of CFs to fix, so I wrote > this: https://github.com/kluyg/cassandra-schema-fix > I think it can be useful to you. > > Kind regards, > Mikhail > > On Mon, Jul 13, 2015 at 11:51 AM, Saladi Naidu <naidusp2...@yahoo.com> > wrote: > >> Sebastian, >> Thank you so much for providing detailed explanation. I still have some >> questions and I need to provide some clarifications >> >> 1. We do not have code that is creating the tables dynamically. All DDL >> operations are done through Datastax DevCenter tool. When you say schema to >> settle, do you means we provide proper consistency level? I don't think >> there is a provision to do that in tool. Or I can change the SYSTEM >> KEYSPACE definition of replication factor equal to number of nodes? >> >> 2. In the steps described below for correcting this problem - when you >> say move data from old directory to new, do you mean move the .db file? It >> will override the current file right? >> >> 3. Do we have to rename the directory name to remove CFID i.e. just >> column family name without CFID? After that, update the System table as >> well? >> >> >> Naidu Saladi >> >> ------------------------------ >> *From:* Sebastian Estevez <sebastian.este...@datastax.com> >> *To:* user@cassandra.apache.org; Saladi Naidu <naidusp2...@yahoo.com> >> *Sent:* Friday, July 10, 2015 5:25 PM >> *Subject:* Re: DROP Table >> >> #1 The cause of this problem is a CREATE TABLE statement collision. Do >> *not* generate tables dynamically from multiple clients, even with IF >> NOT EXISTS. First thing you need to do is fix your code so that this does >> not happen. Just create your tables manually from cqlsh allowing time for >> the schema to settle. >> >> #2 Here's the fix: >> >> 1) *Change your code to not automatically re-create tables (even with IF >> NOT EXISTS).* >> >> 2) Run a rolling restart to ensure schema matches across nodes. Run >> nodetool describecluster around your cluster. Check that there is only one >> schema version. >> >> ON EACH NODE: >> 3) Check your filesystem and see if you have two directories for the >> table in question in the data directory. >> >> If THERE ARE TWO OR MORE DIRECTORIES: >> 4)Identify from schema_column_families which cf ID is the "new" one >> (currently in use). >> >> cqlsh -e "select * from system.schema_column_families"|grep <table name> >> >> >> 5) Move the data from the "old" one to the "new" one and remove the old >> directory. >> >> 6) If there are multiple "old" ones repeat 5 for every "old" directory. >> >> 7) run nodetool refresh >> >> IF THERE IS ONLY ONE DIRECTORY: >> >> No further action is needed. >> >> All the best, >> >> [image: datastax_logo.png] <http://www.datastax.com/> >> Sebastián Estévez >> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com >> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image: >> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] >> <https://twitter.com/datastax> [image: g+.png] >> <https://plus.google.com/+Datastax/about> >> <http://feeds.feedburner.com/datastax> >> >> <http://cassandrasummit-datastax.com/> >> >> DataStax is the fastest, most scalable distributed database technology, >> delivering Apache Cassandra to the world’s most innovative enterprises. >> Datastax is built to be agile, always-on, and predictably scalable to any >> size. With more than 500 customers in 45 countries, DataStax is the >> database technology and transactional backbone of choice for the worlds >> most innovative companies such as Netflix, Adobe, Intuit, and eBay. >> >> >> >> On Fri, Jul 10, 2015 at 12:15 PM, Saladi Naidu <naidusp2...@yahoo.com> >> wrote: >> >> My understanding is that Cassandra File Structure follows below naming >> convention >> >> /cassandra/*data/ <key-spaces> <table>* >> >> >> >> Whereas our file structure is as below, each table has multiple names and >> when we drop tables and recreate these directories remain. Also when we >> dropped the table one node was down, when it came back, we tried to do >> Nodetool repair and repair kept failing referring to CFID error listed >> below >> >> >> drwxr-xr-x. 16 cass cass 4096 May 24 06:49 ../ >> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 >> application_by_user-e0eec95019a211e58b954ffc8e9bfaa6/ >> drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15 application_info- >> 4dba2bf0054f11e58b954ffc8e9bfaa6/ >> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 >> application_info-a0ee65d019a311e58b954ffc8e9bfaa6/ >> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 >> configproperties-228ea2e0c13811e4aa1d4ffc8e9bfaa6/ >> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 >> user_activation-95d005f019a311e58b954ffc8e9bfaa6/ >> drwxr-xr-x. 3 cass cass 4096 Jun 25 10:16 >> user_app_permission-9fddcd62ffbe11e4a25a45259f96ec68/ >> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 >> user_credential-86cfff1019a311e58b954ffc8e9bfaa6/ >> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09 >> user_info-2fa076221b1011e58b954ffc8e9bfaa6/ >> drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15 >> user_info-36028c00054f11e58b954ffc8e9bfaa6/ >> drwxr-xr-x. 3 cass cass 4096 Jun 25 10:15 >> user_info-fe1d7b101a5711e58b954ffc8e9bfaa6/ >> drwxr-xr-x. 4 cass cass 4096 Jun 25 10:16 >> user_role-9ed0ca30ffbe11e4b71d09335ad2d5a9/ >> >> >> WARN [Thread-2579] 2015-07-02 16:02:27,523 IncomingTcpConnection.java:91 >> - UnknownColumnFamilyException reading from socket; closing >> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find >> cfId=218e3c90-1b0e-11e5-a34b-d7c17b3e318a >> at >> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164) >> ~[apache-cassandra-2.1.2.jar:2.1.2] >> at >> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97) >> ~[apache-cassandra-2.1.2.jar:2.1.2] >> at >> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) >> ~[apache-cassandra-2.1.2.jar:2.1.2] >> at >> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) >> ~[apache-cassandra-2.1.2.jar:2.1.2] >> at >> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) >> ~[apache-cassandra-2.1.2.jar:2.1.2] >> at >> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) >> ~[apache-cassandra-2.1.2.jar:2.1.2] >> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) >> ~[apache-cassandra-2.1.2.jar:2.1.2] >> at >> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) >> ~[apache-cassandra-2.1.2.jar:2.1.2] >> at >> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) >> ~[apache-cassandra-2.1.2.jar:2.1.2] >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) >> ~[apache-cassandra-2.1.2.jar:2.1.2] >> >> >> Naidu Saladi >> >> >> >> >> >
