Hi Bohdan, Carlos, Could you try some jvm tool to find out which thread are allocating memory or gc? maybe the migration stage thread..
BTW, is your cluster under high load while dropping table? As far as I remember, in older c* version, it applies the schema mutation in memory, ie. DROP, then flush all schema info into sstable, then reads all on disk schema into memory (5k tables info + related column info).. > You also might need to increase the node count if you're resource constrained. More nodes won't help and most probably make it worse due to coordination. Zhao Yang On Fri, 21 Apr 2017 at 21:10 Bohdan Tantsiura <bohdan...@gmail.com> wrote: > Hi, > > Problem is still not solved. Does anybody have any idea what to do with it? > > Thanks > > 2017-04-20 15:05 GMT+03:00 Bohdan Tantsiura <bohdan...@gmail.com>: > >> Thanks Carlos, >> >> In each keyspace we also have 11 MVs. >> >> It is impossible to reduce number of tables now. Long GC Pauses take >> about one minute. But why it takes so much time and how that can be fixed? >> >> Each node in cluster has 128GB RAM, so resources are not constrained now >> >> Thanks >> >> 2017-04-20 13:18 GMT+03:00 Carlos Rolo <r...@pythian.com>: >> >>> You have 4800 Tables in total? That is a lot of tables, plus MVs? or MVs >>> are already considered in the 60*80 account? >>> >>> I would recommend to reduce the table number. Other thing is that you >>> need to check your log file for GC Pauses, and how long those pauses take. >>> >>> You also might need to increase the node count if you're resource >>> constrained. >>> >>> Regards, >>> >>> Carlos Juzarte Rolo >>> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP >>> >>> Pythian - Love your data >>> >>> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: >>> *linkedin.com/in/carlosjuzarterolo >>> <http://linkedin.com/in/carlosjuzarterolo>* >>> Mobile: +351 918 918 100 <+351%20918%20918%20100> >>> www.pythian.com >>> >>> On Thu, Apr 20, 2017 at 11:10 AM, Bohdan Tantsiura <bohdan...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> We are using cassandra 3.10 in a 10 nodes cluster with replication = 3. >>>> MAX_HEAP_SIZE=64GB on all nodes, G1 GC is used. We have about 60 keyspaces >>>> with about 80 tables in each keyspace. We had to delete three tables and >>>> two materialized views from each keyspace. It began to take more and more >>>> time for each next keyspace (for some keyspaces it took about 30 minutes) >>>> and then failed with "Cannot achieve consistency level ALL". After >>>> restarting the same repeated. It seems that cassandra hangs on GC. How that >>>> can be solved? >>>> >>>> Thanks >>>> >>> >>> >>> -- >>> >>> >>> >>> >> >
