Re: Bursts of Thrift threads make cluster unresponsive

scribed happened, or is the > order with which you presented them the order you notice things going > wrong? > > On Thu, Jun 27, 2019 at 1:29 PM Dmitry Simonov > wrote: > >> Thanks for your reply! >> >> > Have you tried increasing concurrent reads until you se

Re: Bursts of Thrift threads make cluster unresponsive

.path=transport.scope=Native-Transport-Requests.name=TotalBlockedTasks.Count > > > > On Thu, Jun 27, 2019 at 1:52 AM Dmitry Simonov > wrote: > >> Hello! >> >> We've met several times the following problem. >> >> Cassandra cluster (5 nodes) becomes un

Bursts of Thrift threads make cluster unresponsive

odes of cluster. I cannot tie this problem with increased load from clients ("read rate" does't increase during the problem). Also looks like there is no problem with disks (I/O latencies are OK). Could anybody please give some advice in further troubleshooting? -- Best Regards, Dmitry Simonov

cqlsh COPY ... TO ... doesn't work if one node down

Errno None] Tried connecting to [('10.0.0.47', 9042)]. Last error: timed out 2018-06-29 19:21:40,064 Error attempting to reconnect to 10.0.0.47, scheduling retry in 512.0 seconds: [Errno None] Tried connecting to [('10.0.0.47', 9042)]. Last error: timed out :1:(4, 'Interrupted system call') IOError: IOError: IOError: IOError: IOError: -- Best Regards, Dmitry Simonov

Re: Network problems during repair make it hang on "Wait for validation to complete"

In the previous message, I have pasted source code from cassandra 2.2.8 by mistake. Re-checked for 2.2.11 source. These lines are the same. 2018-06-21 2:49 GMT+05:00 Dmitry Simonov : > Hello! > > Using Cassandra 2.2.11, I observe behaviour, that is very similar to > https://issue

Network problems during repair make it hang on "Wait for validation to complete"

issues) during repair may also lead to the same problem. Is it a known issue? -- Best Regards, Dmitry Simonov

Re: Many SSTables only on one node

tained in range (-9223372036854775808,-9223372036854775808], mutating repairedAt instead of anticompacting Does it means that anti-compaction is not the cause? 2018-04-05 18:01 GMT+05:00 Evelyn Smith : > It might not be what cause it here. But check your logs for > anti-compactions. > > > On 5 Apr 20

Re: Many SSTables only on one node

of pending compactions and see if it prevents the issue > from happening again. > > Cheers, > > > On Thu, Apr 5, 2018 at 11:33 AM Dmitry Simonov > wrote: > >> Hi, Alexander! >> >> SizeTieredCompactionStrategy is used for all CFs in problematic keyspace. >&g

Re: Many SSTables only on one node

> Hi Dmitry, > > could you tell us which compaction strategy that table is currently using ? > Also, what is the compaction max throughput and is auto-compaction > correctly enabled on that node ? > > Did you recently run repair ? > > Thanks, > > On Thu, Apr 5, 2018 a

Many SSTables only on one node

s, on other nodes: 3 sstable. Data size in this keyspace was not very big ~100-200Mb per node. There is no such problem with other CFs of that keyspace. nodetool compact solved the issue as a quick-fix. But I'm wondering, what was the cause? How prevent it from repeating? -- Best Regar

Re: "READ messages were dropped ... for internal timeout" after big amount of writes

ge I/O wait at some > point, blocked proc etc) > > > > > On 16 March 2018 at 07:33, Dmitry Simonov wrote: > >> Hello! >> >> We are experiencing problems with Cassandra 2.2.8. >> There is a cluster with 3 nodes. >> Problematic keyspace has RF=3 a

"READ messages were dropped ... for internal timeout" after big amount of writes

;batch_mutate" queries at all, only reads! There are no GC warnings about long pauses Could you please help troubleshooting the issue? -- Best Regards, Dmitry Simonov