In Cassandra versions up to 3.0, hints are stored within a table, where the partition key is the host ID of the server for which the hints are stored.
In such a data model, accumulating 800GB of hints is almost certain to cause very wide rows, which will in turn cause GC pressure when you attempt to read the hints for delivery. This will cause GC pauses, which will cause hints to fail to be delivered, which will cause more hints to be stored. This is bad. In 3.0, hints were rewritten to work around this design flaw. In 2.1, your most likely corrective course is to use 'nodetool truncatehints' on all servers, followed by 'nodetool repair' to deliver the data you lost by truncating the hints. NOTE: this is ONLY safe if you wrote with a consistency level stronger than CL:ANY. If you wrote this data with CL:ANY, you may lose data if you truncate hints. - Jeff On 2017-05-16 06:50 (-0700), varun saluja <saluj...@gmail.com> wrote: > Thanks for update. > I could see lot of io waits. This causing Gc and mutation drops . > But as i mentioned we do not have high load for now. Hint replays are > creating such high disk I/O. > compactionstats show very high hint bytes like 780gb around. Is this normal? > > Just mentioning we are using flash disks. > > In such case, if i run truncatehints , will it remove or decrease size of > hints bytes in compaction stats. I can trigger repair therafter. > Please let me know if any recommendation on same. > > Also , table which we dumped from kafka which created this much hints and > compaction pendings is also dropped today. Because we have to redump table > again once cluster is stable. > > Regards, > Varun > > Sent from my iPhone > > > On 16-May-2017, at 6:59 PM, Nitan Kainth <ni...@bamlabs.com> wrote: > > > > Yes but it means data has to be replicated using repair. > > > > Hints are out come of unhealthy nodes, focus on finding why you have > > mutation drops, is it node, io or network etc. ideally you shouldn't see > > increasing hints all the time. > > > > Sent from my iPhone > > > >> On May 16, 2017, at 7:58 AM, varun saluja <saluj...@gmail.com> wrote: > >> > >> Hi Nitan, > >> > >> Thanks for response. > >> > >> Yes, I could see mutation drops and increase count in system.hints. Is > >> there any way , i can proceed to truncate hints like using nodetool > >> truncatehints. > >> > >> > >> Regards, > >> Varun Saluja > >> > >>> On 16 May 2017 at 17:52, Nitan Kainth <ni...@bamlabs.com> wrote: > >>> Do you see mutation drops? > >>> Select count from system.hints; is it increasing? > >>> > >>> Sent from my iPhone > >>> > >>>> On May 16, 2017, at 5:52 AM, varun saluja <saluj...@gmail.com> wrote: > >>>> > >>>> Hi Experts, > >>>> > >>>> We are facing issue on production cluster. Compaction on system.hint > >>>> table is running from last 2 days. > >>>> > >>>> > >>>> pending tasks: 1 > >>>> compaction type keyspace table completed total > >>>> unit progress > >>>> Compaction system hints 20623021829 877874092407 > >>>> bytes 2.35% > >>>> Active compaction remaining time : 0h27m15s > >>>> > >>>> > >>>> Active compaction remaining time shows in minutes. But, this is job is > >>>> running like indefinitely. > >>>> > >>>> We have 3 node cluster V 2.1.7. And we ran write intensive job last > >>>> week on particular table. > >>>> Compaction on this table finished but hint table size is growing > >>>> continuously. > >>>> > >>>> Can someone Please help me. > >>>> > >>>> > >>>> Thanks & Regards, > >>>> Varun Saluja > >>>> > >> > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
