Hi, FYI: I created a bug ticket since I think the behavior is just not right. https://issues.apache.org/jira/browse/CASSANDRA-15138
Thanks, Hiro On Mon, May 13, 2019 at 10:58 AM Hiroyuki Yamada <mogwa...@gmail.com> wrote: > Hi, > > Should I post a bug ? > It doesn't seem to be an expected behavior, > so I think it should be at least documented somewhere. > > Thanks, > Hiro > > > On Fri, Apr 26, 2019 at 3:17 PM Hiroyuki Yamada <mogwa...@gmail.com> > wrote: > >> Hello, >> >> Thank you for some feedbacks. >> >> >Ben >> Thank you. >> I've tested with lower concurrency in my side, the issue still occurs. >> We are using 3 x T3.xlarge instances for C* and small and separate >> instance for the client program. >> But if we tried with 1 host with 3 C* nodes, the issue didn't occur. >> >> > Alok >> We also thought so and tested with hints disabled, but it doesn't make >> any difference. (the issue still occurs) >> >> Thanks, >> Hiro >> >> >> >> >> On Fri, Apr 26, 2019 at 8:19 AM Alok Dwivedi < >> alok.dwiv...@instaclustr.com> wrote: >> >>> Could it be related to hinted hand offs being stored in Node1 and then >>> attempted to be replayed in Node2 when it comes back causing more load as >>> new mutations are also being applied from cassandra-stress at same time? >>> >>> Alok Dwivedi >>> Senior Consultant >>> https://www.instaclustr.com/ >>> >>> >>> >>> >>> On 26 Apr 2019, at 09:04, Ben Slater <ben.sla...@instaclustr.com> wrote: >>> >>> In the absence of anyone else having any bright ideas - it still sounds >>> to me like the kind of scenario that can occur in a heavily overloaded >>> cluster. I would try again with a lower load. >>> >>> What size machines are you using for stress client and the nodes? Are >>> they all on separate machines? >>> >>> Cheers >>> Ben >>> >>> --- >>> >>> >>> *Ben Slater**Chief Product Officer* >>> >>> <https://www.instaclustr.com/platform/> >>> >>> <https://www.facebook.com/instaclustr> >>> <https://twitter.com/instaclustr> >>> <https://www.linkedin.com/company/instaclustr> >>> >>> Read our latest technical blog posts here >>> <https://www.instaclustr.com/blog/>. >>> >>> This email has been sent on behalf of Instaclustr Pty. Limited >>> (Australia) and Instaclustr Inc (USA). >>> >>> This email and any attachments may contain confidential and legally >>> privileged information. If you are not the intended recipient, do not copy >>> or disclose its content, but please reply to this email immediately and >>> highlight the error to the sender and then immediately delete the message. >>> >>> >>> On Thu, 25 Apr 2019 at 17:26, Hiroyuki Yamada <mogwa...@gmail.com> >>> wrote: >>> >>>> Hello, >>>> >>>> Sorry again. >>>> We found yet another weird thing in this. >>>> If we stop nodes with systemctl or just kill (TERM), it causes the >>>> problem, >>>> but if we kill -9, it doesn't cause the problem. >>>> >>>> Thanks, >>>> Hiro >>>> >>>> On Wed, Apr 24, 2019 at 11:31 PM Hiroyuki Yamada <mogwa...@gmail.com> >>>> wrote: >>>> >>>>> Sorry, I didn't write the version and the configurations. >>>>> I've tested with C* 3.11.4, and >>>>> the configurations are mostly set to default except for the >>>>> replication factor and listen_address for proper networking. >>>>> >>>>> Thanks, >>>>> Hiro >>>>> >>>>> On Wed, Apr 24, 2019 at 5:12 PM Hiroyuki Yamada <mogwa...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hello Ben, >>>>>> >>>>>> Thank you for the quick reply. >>>>>> I haven't tried that case, but it does't recover even if I stopped >>>>>> the stress. >>>>>> >>>>>> Thanks, >>>>>> Hiro >>>>>> >>>>>> On Wed, Apr 24, 2019 at 3:36 PM Ben Slater < >>>>>> ben.sla...@instaclustr.com> wrote: >>>>>> >>>>>>> Is it possible that stress is overloading node 1 so it’s not >>>>>>> recovering state properly when node 2 comes up? Have you tried running >>>>>>> with >>>>>>> a lower load (say 2 or 3 threads)? >>>>>>> >>>>>>> Cheers >>>>>>> Ben >>>>>>> >>>>>>> --- >>>>>>> >>>>>>> >>>>>>> *Ben Slater* >>>>>>> *Chief Product Officer* >>>>>>> >>>>>>> >>>>>>> <https://www.facebook.com/instaclustr> >>>>>>> <https://twitter.com/instaclustr> >>>>>>> <https://www.linkedin.com/company/instaclustr> >>>>>>> >>>>>>> Read our latest technical blog posts here >>>>>>> <https://www.instaclustr.com/blog/>. >>>>>>> >>>>>>> This email has been sent on behalf of Instaclustr Pty. Limited >>>>>>> (Australia) and Instaclustr Inc (USA). >>>>>>> >>>>>>> This email and any attachments may contain confidential and legally >>>>>>> privileged information. If you are not the intended recipient, do not >>>>>>> copy >>>>>>> or disclose its content, but please reply to this email immediately and >>>>>>> highlight the error to the sender and then immediately delete the >>>>>>> message. >>>>>>> >>>>>>> >>>>>>> On Wed, 24 Apr 2019 at 16:28, Hiroyuki Yamada <mogwa...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I faced a weird issue when recovering a cluster after two nodes are >>>>>>>> stopped. >>>>>>>> It is easily reproduce-able and looks like a bug or an issue to fix, >>>>>>>> so let me write down the steps to reproduce. >>>>>>>> >>>>>>>> === STEPS TO REPRODUCE === >>>>>>>> * Create a 3-node cluster with RF=3 >>>>>>>> - node1(seed), node2, node3 >>>>>>>> * Start requests to the cluster with cassandra-stress (it continues >>>>>>>> until the end) >>>>>>>> - what we did: cassandra-stress mixed cl=QUORUM duration=10m >>>>>>>> -errors ignore -node node1,node2,node3 -rate threads\>=16 >>>>>>>> threads\<=256 >>>>>>>> * Stop node3 normally (with systemctl stop) >>>>>>>> - the system is still available because the quorum of nodes is >>>>>>>> still available >>>>>>>> * Stop node2 normally (with systemctl stop) >>>>>>>> - the system is NOT available after it's stopped. >>>>>>>> - the client gets `UnavailableException: Not enough replicas >>>>>>>> available for query at consistency QUORUM` >>>>>>>> - the client gets errors right away (so few ms) >>>>>>>> - so far it's all expected >>>>>>>> * Wait for 1 mins >>>>>>>> * Bring up node2 >>>>>>>> - The issue happens here. >>>>>>>> - the client gets ReadTimeoutException` or WriteTimeoutException >>>>>>>> depending on if the request is read or write even after the node2 is >>>>>>>> up >>>>>>>> - the client gets errors after about 5000ms or 2000ms, which are >>>>>>>> request timeout for write and read request >>>>>>>> - what node1 reports with `nodetool status` and what node2 >>>>>>>> reports >>>>>>>> are not consistent. (node2 thinks node1 is down) >>>>>>>> - It takes very long time to recover from its state >>>>>>>> === STEPS TO REPRODUCE === >>>>>>>> >>>>>>>> Is it supposed to happen ? >>>>>>>> If we don't start cassandra-stress, it's all fine. >>>>>>>> >>>>>>>> Some workarounds we found to recover the state are the followings: >>>>>>>> * Restarting node1 and it recovers its state right after it's >>>>>>>> restarted >>>>>>>> * Setting lower value in dynamic_snitch_reset_interval_in_ms (to >>>>>>>> 60000 >>>>>>>> or something) >>>>>>>> >>>>>>>> I don't think either of them is a really good solution. >>>>>>>> Can anyone explain what is going on and what is the best way to make >>>>>>>> it not happen or recover ? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Hiro >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>>>>>>> For additional commands, e-mail: user-h...@cassandra.apache.org >>>>>>>> >>>>>>>> >>>
