Try sstablescrub on the node where it is showing corrupted data.
On Fri, Aug 11, 2023 at 8:38 AM Joe Obernberger <
joseph.obernber...@gmail.com> wrote:
> Finally found a message on another node that seem relevant:
>
> INFO [CompactionExecutor:7413] 2023-08-11 11:36:22,397
> CompactionTask.java:1
Finally found a message on another node that seem relevant:
INFO [CompactionExecutor:7413] 2023-08-11 11:36:22,397
CompactionTask.java:164 - Compacting
(d30b64ba-385c-11ee-8e74-edf5512ad115)
[/data/3/cassandra/data/doc/extractedmetadata-25c210e0ada011ebade9fdc1d34336d3/nb-97958-big-Data.db:le
What logs of /172.16.20.16:7000 say when repair failed. It indicates
"validation failed". Can you check system.log for /172.16.20.16:7000 and
see what they say. Looks like you have some issue with *doc/origdoc,
probably some corrupt sstable. *Try to run repair for individual table and
see for wh
Thank you. I've tried:
nodetool repair --full
nodetool repair -pr
They all get to 57% on any of the nodes, and then fail. Interestingly
the debug log only has INFO - there are no errors.
[2023-08-07 14:02:09,828] Repair command #6 failed with error
Incremental repair session 83dc17d0-354c-11e
Quick drive-by observation:
> Did not get replies from all endpoints.. Check the
> logs on the repair participants for further details
> dropping message of type HINT_REQ due to error
> org.apache.cassandra.net.AsyncChannelOutputPlus$FlushException: The
> channel this output stream was writing to
Can you please try to do nodetool describecluster from every node of the
cluster?
One time I noticed issue when nodetool status shows all nodes UN but
describecluster was not.
Thanks
Surbhi
On Fri, Aug 4, 2023 at 8:59 AM Joe Obernberger
wrote:
> Hi All - been using reaper to do repairs, but it
Hi All - been using reaper to do repairs, but it has hung. I tried to run:
nodetool repair -pr
on each of the nodes, but they all fail with some form of this error:
error: Repair job has failed with the error message: Repair command #521
failed with error Did not get replies from all endpoints.
