Hi
We were trying to overcome OOM crashes.
Fair enough :-).
We changed settings to default on one node. GC times became about two times
> smaller on that node.
>
That's encouraging! Looks like even if the number of tables is really high,
there is still space for optimization. Have you made the
Hi,
> Why did you move from defaults that much?
We were trying to overcome OOM crashes.
> Would you consider giving defaults a try on a canary node and monitor /
compare GC times to other nodes?
We changed settings to default on one node. GC times became about two times
smaller on that node.
> W
Hi,
A few comments:
Long GC Pauses take about one minute
>
This is huge. About JVM config, I haven't played much with G1GC, but the
following seems to be a bad idea according to comments:
## Main G1GC tunable: lowering the pause target will lower throughput and
vise versa.
## 200ms is the JVM d
Thanks Alain,
> Or is it on happening during drop table actions?
Some other schema changes (e.g. adding columns to tables) also takes too
much time.
Link to complete set of GC options: https://pastebin.com/4qyENeyu
> Have you had a look at logs, mainly errors and warnings?
In logs I found warnin
Hi
> Long GC Pauses take about one minute. But why it takes so much time and
> how that can be fixed?
This is very long. Looks like you are having a major issue, and it is not
just about dropping tables... Or is it on happening during drop table
actions? Knowing the complete set of GC options i
Thanks Zhao Yang,
> Could you try some jvm tool to find out which thread are allocating
memory or gc? maybe the migration stage thread..
I use Cassandra Cluster Manager to locally reproduce the issue. I tried to
use VisualVM to find out which threads are allocating memory, but VisualVM
does not s
Hi Bohdan, Carlos,
Could you try some jvm tool to find out which thread are allocating memory
or gc? maybe the migration stage thread..
BTW, is your cluster under high load while dropping table?
As far as I remember, in older c* version, it applies the schema mutation
in memory, ie. DROP, then f
Hi,
Problem is still not solved. Does anybody have any idea what to do with it?
Thanks
2017-04-20 15:05 GMT+03:00 Bohdan Tantsiura :
> Thanks Carlos,
>
> In each keyspace we also have 11 MVs.
>
> It is impossible to reduce number of tables now. Long GC Pauses take about
> one minute. But why it
Thanks Carlos,
In each keyspace we also have 11 MVs.
It is impossible to reduce number of tables now. Long GC Pauses take about
one minute. But why it takes so much time and how that can be fixed?
Each node in cluster has 128GB RAM, so resources are not constrained now
Thanks
2017-04-20 13:18
You have 4800 Tables in total? That is a lot of tables, plus MVs? or MVs
are already considered in the 60*80 account?
I would recommend to reduce the table number. Other thing is that you need
to check your log file for GC Pauses, and how long those pauses take.
You also might need to increase th
Hi,
We are using cassandra 3.10 in a 10 nodes cluster with replication = 3.
MAX_HEAP_SIZE=64GB on all nodes, G1 GC is used. We have about 60 keyspaces
with about 80 tables in each keyspace. We had to delete three tables and
two materialized views from each keyspace. It began to take more and more
11 matches
Mail list logo
