Huge size of system.batches table after dropping an incomplete Materialized View
Hi there, Version :- Cassandra 3.0.7 I attempted to create a Materialized View on a certain table and it failed with never-ending WARN message "Mutation of bytes is too large for the maximum size of ". "nodetool stop VIEW_BUILD" also did not help. That seems to be a result of https://issues.apache.org/jira/browse/CASSANDRA-11670 which is fixed in newer versions. So I tried dropping the view and that generated error messages like following :- ERROR [CompactionExecutor:632] [Timestamp] Keyspace.java:475 - Attempting to mutate non-existant table 7c2e1c40-b82b-11e6-9d20-4b0190661423 (keyspace_name.view_name) I performed an incremental repair of the table on which view was created and a rolling restart to stop these errors. Now I see huge size of system.batches table on one of the nodes. It seems related to issues mentioned above since last modification timestamps of the sstable files inside system/batches is same as when I tried to drop the MV. Some insight and suggestions regarding it will be very helpful. I will like to know if i can safely truncate the table, rm the files or any other approach to clean it up? Thanks.
Re: Huge size of system.batches table after dropping an incomplete Materialized View
I cannot tell you were these errors like "Attempting to mutate ..." come from but under certain circumstances all view mutations are stored in batches, so the batchlog can grow insanely large. I don't see why a repair should help you in this situation. I guess what you want is to recreate the table. 1. You should not repair MVs directly. The current design is to only repairs the base table - though it's not properly documented. Repairing MVs can create inconsistent states. Only repairing the base tables wont. 2. A repair does only repair data and won't fix schema-issues 3. A repair of a base table that contains an MV is incredibly slow if the state is very inconsistent (which is probably the case in your situation) What to do? - If you don't care about the data of the MV, you of course can delete all SSTables (when CS is stopped) and all data will be gone. But I don't know if it helps. - If you are 100% sure that no other batch logs are going on, you could also truncate the system.batches, otherwise your log may be flooded with "non-existant table" things if the batch log is replayed. It is annoying but should not harm anyone. => Start over, try to drop and create the MV. Watch out for logs referring to schema changes and errors Side note: I'd recommend not to use MVs (yet) if you don't have an "inside" understanding of them or "know what you are doing". They can have a very big impact on your cluster performance in some situations and are not generally considered as stable yet. 2017-01-22 18:42 GMT+01:00 Vinci : > Hi there, > > Version :- Cassandra 3.0.7 > > I attempted to create a Materialized View on a certain table and it failed > with never-ending WARN message "Mutation of bytes is too large for > the maximum size of ". > > "nodetool stop VIEW_BUILD" also did not help. > > That seems to be a result of https://issues.apache.org/ > jira/browse/CASSANDRA-11670 which is fixed in newer versions. > > So I tried dropping the view and that generated error messages like > following :- > > ERROR [CompactionExecutor:632] [Timestamp] Keyspace.java:475 - Attempting > to mutate non-existant table 7c2e1c40-b82b-11e6-9d20-4b0190661423 > (keyspace_name.view_name) > > I performed an incremental repair of the table on which view was created > and a rolling restart to stop these errors. > > Now I see huge size of system.batches table on one of the nodes. It seems > related to issues mentioned above since last modification timestamps of the > sstable files inside system/batches is same as when I tried to drop the MV. > > Some insight and suggestions regarding it will be very helpful. I will > like to know if i can safely truncate the table, rm the files or any other > approach to clean it up? > > Thanks. > -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer
Dropped messages on random nodes.
Hello there, We have a 100 nodes ish cluster, I find that there are dropped messages on random nodes in the cluster, which caused error spikes and P99 latency spikes as well. I tried to figure out the cause. I do not see any obvious bottleneck in the cluster, the C* nodes still have plenty of cpu idle/disk io. But I do see some suspicious gossip events around that time, not sure if it's related. 2017-01-21_16:43:56.71033 WARN 16:43:56 [GossipTasks:1]: Not marking nodes down due to local pause of 13079498815 > 50 2017-01-21_16:43:56.85532 INFO 16:43:56 [ScheduledTasks:1]: MUTATION messages were dropped in last 5000 ms: 65 for internal timeout and 10895 for cross node timeout 2017-01-21_16:43:56.85533 INFO 16:43:56 [ScheduledTasks:1]: READ messages were dropped in last 5000 ms: 33 for internal timeout and 7867 for cross node timeout 2017-01-21_16:43:56.85534 INFO 16:43:56 [ScheduledTasks:1]: Pool Name Active Pending Completed Blocked All Time Blocked 2017-01-21_16:43:56.85534 INFO 16:43:56 [ScheduledTasks:1]: MutationStage 128 47794 1015525068 0 0 2017-01-21_16:43:56.85535 2017-01-21_16:43:56.85535 INFO 16:43:56 [ScheduledTasks:1]: ReadStage 64 20202 450508940 0 0 Any suggestions? Thanks! -- Dikang
Re: Dropped messages on random nodes.
Btw, the C* version is 2.2.5, with several backported patches. On Sun, Jan 22, 2017 at 10:36 PM, Dikang Gu wrote: > Hello there, > > We have a 100 nodes ish cluster, I find that there are dropped messages on > random nodes in the cluster, which caused error spikes and P99 latency > spikes as well. > > I tried to figure out the cause. I do not see any obvious bottleneck in > the cluster, the C* nodes still have plenty of cpu idle/disk io. But I do > see some suspicious gossip events around that time, not sure if it's > related. > > 2017-01-21_16:43:56.71033 WARN 16:43:56 [GossipTasks:1]: Not marking > nodes down due to local pause of 13079498815 > 50 > 2017-01-21_16:43:56.85532 INFO 16:43:56 [ScheduledTasks:1]: MUTATION > messages were dropped in last 5000 ms: 65 for internal timeout and 10895 > for cross node timeout > 2017-01-21_16:43:56.85533 INFO 16:43:56 [ScheduledTasks:1]: READ messages > were dropped in last 5000 ms: 33 for internal timeout and 7867 for cross > node timeout > 2017-01-21_16:43:56.85534 INFO 16:43:56 [ScheduledTasks:1]: Pool Name >Active Pending Completed Blocked All Time Blocked > 2017-01-21_16:43:56.85534 INFO 16:43:56 [ScheduledTasks:1]: MutationStage > 128 47794 1015525068 0 0 > 2017-01-21_16:43:56.85535 > 2017-01-21_16:43:56.85535 INFO 16:43:56 [ScheduledTasks:1]: ReadStage >64 20202 450508940 0 0 > > Any suggestions? > > Thanks! > > -- > Dikang > > -- Dikang
Re: JVM state determined to be unstable. Exiting forcefully. what is Java Stability Inspector ?? why it is stopping DSE?
Hi Pranay, i seems that your data is unevenly distributed across the cluster with respect your insertion frequency.Please restructure your partition key Thanks On Fri, Jan 20, 2017 at 6:49 AM, Pranay akula wrote: > what i have observed is 2-3 old gen GC's in 1-2 mins before OOM which i > rarely see and seen hinted handoffs get accumulated on nodes which went > down, and Mutation drops as well. > > i really don't know how to analyse hprof file is there any guide or blog > that can help me how to analyse it ?? our cluster has 2 DC's each DC with > 18 nodes each and 12 GB Heap and 4 GB new Heap. > > > Thanks > Pranay. > > On Thu, Jan 19, 2017 at 8:19 AM, Alain RODRIGUEZ > wrote: > >> Hi Pranay, >> >> what can be the reason for this >> >> >> It can be due to a JVM / GC misconfiguration or to some abnormal activity >> in Cassandra. Often, GC issues are a consequences and not the root cause of >> an issue in Cassandra. >> >> >>> how to debug that ?? >> >> how to fine grain why on those particular nodes this is happening when >>> these nodes are serving same requests like rest of the cluster ?? >> >> >> You can enable GC logs on those nodes (use the cassandra-env.sh file to >> do so) and have a look at what's happening there. Also you can have a look >> at the system.log files (search for warning or errors - WARN / ERROR) and >> at "nodetool tpstats". I like to use this last command as follow "watch -d >> nodetool tpstats" to see variations. >> >> Having pending or dropped threads is likely to increase the GC activity. >> As well as having wide rows, many tomstones and some other cases. >> >> So to determine why this is happening, could you share your hardware >> specs, the way JVM / GC is configured (cassandra-env.sh) and let us know >> how nodes are handling threads and about any relevant infrmation that might >> be appearing in the logs. >> >> You can investigate the heap dump as well (I believe you can do this >> using Eclipse Memory Analyzer - MAT). >> >> C*heers, >> --- >> Alain Rodriguez - @arodream - al...@thelastpickle.com >> France >> >> The Last Pickle - Apache Cassandra Consulting >> http://www.thelastpickle.com >> >> 2017-01-19 14:00 GMT+01:00 Pranay akula : >> >>> From last few days i am seeing on some of the nodes in cassandra cluster >>> DSE is getting shutdown due to the error below and i need to kill Java >>> process and restart DSE service. >>> >>> I have cross checked reads and writes and compactions nothing looks >>> suspicious, but i am seeing full Gc pause on these server just before the >>> issue happening. what can be the reason for this how to debug that ?? how >>> to fine grain why on those particular nodes this is happening when these >>> nodes are serving same requests like rest of the cluster ?? >>> >>> Is this happening because of Full Gc is not getting performed properly, >>> we using G1GC and DSE 4.8.3 >>> >>> >>> ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100 >>> JVMStabilityInspector.java:117 - JVM state determined to be unstable. >>> Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap space >>> >>> at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74] >>> >>> at >>> org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112) >>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>> >>> at org.apache.cassandra.db.Deleti >>> onInfo.copy(DeletionInfo.java:104) ~[cassandra-all-2.1.13.1131.ja >>> r:2.1.13.1131] >>> >>> at org.apache.cassandra.db.Atomic >>> BTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217) >>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>> >>> at org.apache.cassandra.db.Memtable.put(Memtable.java:210) >>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>> >>> at org.apache.cassandra.db.Column >>> FamilyStore.apply(ColumnFamilyStore.java:1230) >>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>> >>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) >>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>> >>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) >>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>> >>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) >>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>> >>> at org.apache.cassandra.db.Mutati >>> onVerbHandler.doVerb(MutationVerbHandler.java:54) >>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>> >>> at org.apache.cassandra.net.Messa >>> geDeliveryTask.run(MessageDeliveryTask.java:64) >>> ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] >>> >>> at java.util.concurrent.Executors >>> $RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_74] >>> >>> at org.apache.cassandra.concurren >>> t.AbstractTracingAwareExecutorService$FutureTask.run(Abstrac >>> tTracingAwareExecutorService.java:164) ~[cassandra-all-2.1.13.1131.ja >>> r:2.1.13.1131] >>> >>> at org.apache.cassan
