So no it did not fill in a minute, but ton's of header files were written in a minute (is that normal, I assume these are marker files which get written when memtables are flushed. The actual data files have been around for the last 24 hours? Somehow this all seems connected to "reintroduce node" exercise I went thru on Saturday. Somehow can this get worse with cleanup, hinted hand off, etc....
When does the actual commit-data file get deleted. The flush interval on all my memtables is 60 minutes Thanks On Sun, Aug 21, 2011 at 8:43 AM, Anand Somani <meatfor...@gmail.com> wrote: > Hi, > > 7.4, 3 node cluster, RF=3 > > Load has not changed much, on 2 of the 3 nodes the commit log filled up in > less than a minute (did not give a chance to recover). Now have been running > this cluster for abt 2-3 months without any problem. At this point I do not > see any unusual load (continue to investigate). The commit log has never > taken more than 20% before this!!! > > The only changes was a re-intro of a node couple days ago with a bunch of > cleanup on each node. Has anybody seen anything like this? > > I am starting the entire cluster again, anyway hints and things I should > watch for (besides commit -log). I have backed up the commit logs and logs, > so any tips on how to do a analysis on this would help. > > Thanks > Anand >
