You can control compaction with nodetool compactionthroughput but it will just slow down compaction and give resources for application, however it's not a fix.
Sent from my iPhone > On May 16, 2017, at 9:15 AM, varun saluja <saluj...@gmail.com> wrote: > > Thanks Nitan. > Appreciate your help. > > Can anyone suggest parameter change or something which can help in this > situation. > > Regards, > Varun > > Sent from my iPhone > >> On 16-May-2017, at 7:31 PM, Nitan Kainth <ni...@bamlabs.com> wrote: >> >> If target table is dropped then you can remove its hints but there could be >> more hints from other table. If it has tables of your interest , then I >> won't comment on truncating hints. >> >> Size of hints depends on Kafka load , looks like you had overloaded the >> cluster during data load and not hints are just recovering from it. I would >> say wait until cluster comes to normal state. May be some other expert can >> suggest an alternate. >> >> Sent from my iPhone >> >>> On May 16, 2017, at 8:50 AM, varun saluja <saluj...@gmail.com> wrote: >>> >>> Thanks for update. >>> I could see lot of io waits. This causing Gc and mutation drops . >>> But as i mentioned we do not have high load for now. Hint replays are >>> creating such high disk I/O. >>> compactionstats show very high hint bytes like 780gb around. Is this normal? >>> >>> Just mentioning we are using flash disks. >>> >>> In such case, if i run truncatehints , will it remove or decrease size of >>> hints bytes in compaction stats. I can trigger repair therafter. >>> Please let me know if any recommendation on same. >>> >>> Also , table which we dumped from kafka which created this much hints and >>> compaction pendings is also dropped today. Because we have to redump table >>> again once cluster is stable. >>> >>> Regards, >>> Varun >>> >>> Sent from my iPhone >>> >>>> On 16-May-2017, at 6:59 PM, Nitan Kainth <ni...@bamlabs.com> wrote: >>>> >>>> Yes but it means data has to be replicated using repair. >>>> >>>> Hints are out come of unhealthy nodes, focus on finding why you have >>>> mutation drops, is it node, io or network etc. ideally you shouldn't see >>>> increasing hints all the time. >>>> >>>> Sent from my iPhone >>>> >>>>> On May 16, 2017, at 7:58 AM, varun saluja <saluj...@gmail.com> wrote: >>>>> >>>>> Hi Nitan, >>>>> >>>>> Thanks for response. >>>>> >>>>> Yes, I could see mutation drops and increase count in system.hints. Is >>>>> there any way , i can proceed to truncate hints like using nodetool >>>>> truncatehints. >>>>> >>>>> >>>>> Regards, >>>>> Varun Saluja >>>>> >>>>>> On 16 May 2017 at 17:52, Nitan Kainth <ni...@bamlabs.com> wrote: >>>>>> Do you see mutation drops? >>>>>> Select count from system.hints; is it increasing? >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>>> On May 16, 2017, at 5:52 AM, varun saluja <saluj...@gmail.com> wrote: >>>>>>> >>>>>>> Hi Experts, >>>>>>> >>>>>>> We are facing issue on production cluster. Compaction on system.hint >>>>>>> table is running from last 2 days. >>>>>>> >>>>>>> >>>>>>> pending tasks: 1 >>>>>>> compaction type keyspace table completed total >>>>>>> unit progress >>>>>>> Compaction system hints 20623021829 >>>>>>> 877874092407 bytes 2.35% >>>>>>> Active compaction remaining time : 0h27m15s >>>>>>> >>>>>>> >>>>>>> Active compaction remaining time shows in minutes. But, this is job is >>>>>>> running like indefinitely. >>>>>>> >>>>>>> We have 3 node cluster V 2.1.7. And we ran write intensive job last >>>>>>> week on particular table. >>>>>>> Compaction on this table finished but hint table size is growing >>>>>>> continuously. >>>>>>> >>>>>>> Can someone Please help me. >>>>>>> >>>>>>> >>>>>>> Thanks & Regards, >>>>>>> Varun Saluja >>>>>>> >>>>>
