Thanks for update. I could see lot of io waits. This causing Gc and mutation drops . But as i mentioned we do not have high load for now. Hint replays are creating such high disk I/O. compactionstats show very high hint bytes like 780gb around. Is this normal?
Just mentioning we are using flash disks. In such case, if i run truncatehints , will it remove or decrease size of hints bytes in compaction stats. I can trigger repair therafter. Please let me know if any recommendation on same. Also , table which we dumped from kafka which created this much hints and compaction pendings is also dropped today. Because we have to redump table again once cluster is stable. Regards, Varun Sent from my iPhone > On 16-May-2017, at 6:59 PM, Nitan Kainth <ni...@bamlabs.com> wrote: > > Yes but it means data has to be replicated using repair. > > Hints are out come of unhealthy nodes, focus on finding why you have mutation > drops, is it node, io or network etc. ideally you shouldn't see increasing > hints all the time. > > Sent from my iPhone > >> On May 16, 2017, at 7:58 AM, varun saluja <saluj...@gmail.com> wrote: >> >> Hi Nitan, >> >> Thanks for response. >> >> Yes, I could see mutation drops and increase count in system.hints. Is there >> any way , i can proceed to truncate hints like using nodetool truncatehints. >> >> >> Regards, >> Varun Saluja >> >>> On 16 May 2017 at 17:52, Nitan Kainth <ni...@bamlabs.com> wrote: >>> Do you see mutation drops? >>> Select count from system.hints; is it increasing? >>> >>> Sent from my iPhone >>> >>>> On May 16, 2017, at 5:52 AM, varun saluja <saluj...@gmail.com> wrote: >>>> >>>> Hi Experts, >>>> >>>> We are facing issue on production cluster. Compaction on system.hint table >>>> is running from last 2 days. >>>> >>>> >>>> pending tasks: 1 >>>> compaction type keyspace table completed total >>>> unit progress >>>> Compaction system hints 20623021829 877874092407 >>>> bytes 2.35% >>>> Active compaction remaining time : 0h27m15s >>>> >>>> >>>> Active compaction remaining time shows in minutes. But, this is job is >>>> running like indefinitely. >>>> >>>> We have 3 node cluster V 2.1.7. And we ran write intensive job last week >>>> on particular table. >>>> Compaction on this table finished but hint table size is growing >>>> continuously. >>>> >>>> Can someone Please help me. >>>> >>>> >>>> Thanks & Regards, >>>> Varun Saluja >>>> >>
