Hello Mark Second, any ideas what could be creating bottlenecks for schema alteration?
I am not too sure what could be going on to make things that long, but about the corrupted data, I've seen it before. Here are some thoughts around schema changes and finding the bottlenecks: Ideally, use commands to make sure there is no disagreement before moving with the next schema change. Worst case, *slow down *the path of the changes. The schema changes queries are not designed to be performant not to be used asynchronously or so many times in a row. Thus they should and cannot be used as standard queries hammering the cluster, as of now and imho. For design reasons, it's not good to run asynchronous / fast path 'alter table' queries. More information about current design issues and incoming improvement is available on Jira: https://issues.apache.org/jira/browse/CASSANDRA-9449, https://issues.apache.org/jira/browse/CASSANDRA-9425, https://issues.apache.org/jira/browse/CASSANDRA-10699, ... You might still improve speed if you want to find the bottleneck, use monitoring dashboards or nodetool: - 'nodetool describecluster' - show node's schema version. - 'watch -d nodetool tpstats' --> look for pending / dropped operations - Check your GC performances if you see a lot of/big GC pauses (maybe https://gceasy.io might help you there). - Check the logs (for warn/error - missing columns or mismatching schema) and the system.schema_column_families to dig deeper and see what each nodes have as a source of truth. I hope you'll find some clues you can investigate further with one of those ^. Also, 'nodetool resetlocalschema' could help if some nodes are stuck with an old schema version: http://cassandra.apache.org/doc/latest/tools/nodetool/resetlocalschema.html. Often a rolling restart also does the trick. If you would need more specific help it would be good to share the version you are using so we know where to look at and also to have an idea of the number of nodes could help. Some of this stuff is shown here: https://docs.datastax.com/en/cql/3.3/cql/cql_using/useCreateTableCollisionFix.html. The existence of this kind of document show that this is a common issue and it even says clearly: 'Dynamic schema creation or updates can cause schema collision resulting in errors.' So for know, I would move slowly, probably automating a check of the output of 'nodetool describecluster' to make sure the schema was spread before going for the next mutation. the number of "alter table" statements was quite large (300+). > I must admit I don't know the 'alter table' path well internally, but I think this is a lot of changes and that it's not designed to happen quickly. Slow it down and add control procedure in the path I would say. First, is cqlsh the best way to handle these types of loads? Yes, I see no problem with that. It could also be through any other Cassandra client. Maybe they would be faster. I never had to do so many changes at once :). You can give python or Java a try for this work I guess. In that case use synchronous requests and automate checks I would say, to definitely stay away of race conditions / data corruption. > Our DBAs report that even under normal conditions they send alter table > statements in small chunks or else the will see load times of 20-45 minutes. I also often noticed schema changes take some time, but I did not mind much. Maybe the comments above that should hopefully keep you away from race conditions or the use of some other client (Java instead of cqlsh let's say) might help. I guess you could give it a try. I would definitely start by reading more (Jira/doc/code) about how Cassandra perform those changes, if I had to do this kind of batch of changes, because schema changes do not seem to be as safe and efficient as most of Cassandra internals are nowadays (for mainstream features, not counting MVs, Indexes, triggers, etc). This common feature that is to make multiple changes to your data model quickly should be handled with care and understanding in Cassandra and for now I would say. I hope some of the above might be useful to you, C*heers, ----------------------- Alain Rodriguez - al...@thelastpickle.com France / Spain The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com Le jeu. 9 mai 2019 à 13:40, Mark Bidewell <mbide...@gmail.com> a écrit : > I am doing post-mortem on an issue with our cassandra cluster. One of our > tables became corrupt and had to be restored via a backup. The table > schema has been undergoing active development, so the number of "alter > table" statements was quite large (300+). Currently, we use cqlsh to do > schema loads. During the restore, the schema load alone took about 4 hours. > > Our DBAs report that even under normal conditions they send alter table > statements in small chunks or else the will see load times of 20-45 minutes. > > My question is two part. First, is cqlsh the best way to handle these > types of loads? Second, any ideas what could be creating bottlenecks for > schema alteration? > > Thanks! > > -- > Mark Bidewell > http://www.linkedin.com/in/markbidewell >
