Application team confirmed that they are * not* referencing the dropped MVs
anywhere for reading or writing
On Tue, 18 Feb 2020 at 22:25, Surbhi Gupta wrote:
> So should upgrading to 3.11.1 will solve this issue?
>
> On Tue, 18 Feb 2020 at 22:18, Surbhi Gupta
> wrote:
>
>> Thanks Eric...
>>
>>
>
> So should upgrading to 3.11.1 will solve this issue?
>
Upgrading off 3.11.0 will prevent nodes going down as a result of the hint
replay bug in CASSANDRA-13696, yes. But I'd recommend upgrading to the
latest C* 3.11.6 unless you have a very specific reason for upgrading to
3.11.1 which was rel
So should upgrading to 3.11.1 will solve this issue?
On Tue, 18 Feb 2020 at 22:18, Surbhi Gupta wrote:
> Thanks Eric...
>
> On Tue, 18 Feb 2020 at 22:06, Erick Ramirez
> wrote:
>
>> Just to add to my above point because here we are dropping MV not a
>>> regular table.
>>> And MV does read befor
Thanks Eric...
On Tue, 18 Feb 2020 at 22:06, Erick Ramirez
wrote:
> Just to add to my above point because here we are dropping MV not a
>> regular table.
>> And MV does read before write , Is this the reason we are seeing the
>> below message? Trying to understand
>>
>> WARN [HintsDispatcher:67
>
> Just to add to my above point because here we are dropping MV not a
> regular table.
> And MV does read before write , Is this the reason we are seeing the below
> message? Trying to understand
>
> WARN [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 -
> Failed to read a h
Just to add to my above point because here we are dropping MV not a regular
table.
And MV does read before write , Is this the reason we are seeing the below
message? Trying to understand
WARN [HintsDispatcher:6737] 2020-02-18 14:22:24,932
HintsReader.java:237 - Failed to read a hint for /10.X.X.
I'm not sure I understand your last response. Was there a question in there
somewhere? Cheers!
>
Hi Eric,
As per https://issues.apache.org/jira/browse/CASSANDRA-13696 , this issue
happens even with write traffic
"I did more investigation today. Seems it's more serious than I thought.
Even there's no down node, "drop table" + write traffic, will trigger the
problem."
Thanks
Surbhi
On Tue, 1
>
> We are Cassandra 3.11.0 unfortunately :(
>
Oh, right. That's why the hint read failure is causing the node to
shutdown. We've at least identified that. I was worried that there was
another bug we didn't know about. Cheers!
>
We tested this is dev and test before dropping in production but did not
see this issue in dev/test .
I am yet to hear from application team.
However if it was application read was happening from the dropped MV then
we would have caught this error in dev/test itself.
And we are running Cassandra 3.
We are Cassandra 3.11.0 unfortunately :(
On Tue, 18 Feb 2020 at 19:41, Erick Ramirez
wrote:
> Clearly the hint error invoked the fs error handler - probably incorrectly
>> - which shut down the db. That’s not ok and deserves a JIRA.
>>
>
> It's supposed to have been fixed by CASSANDRA-13696 in 3
>
> Clearly the hint error invoked the fs error handler - probably incorrectly
> - which shut down the db. That’s not ok and deserves a JIRA.
>
It's supposed to have been fixed by CASSANDRA-13696 in 3.0.15/3.11.1 but
I'm waiting for Surbhi to confirm the exact C* version. Cheers!
Reading from a non-existent table shouldn’t crash the database
Clearly the hint error invoked the fs error handler - probably incorrectly -
which shut down the db. That’s not ok and deserves a JIRA.
If dropping a table causes a hint checksum error then it needs to be fixed.
>> On Feb 18, 2020
Thanks Eric, Let me go back to the app team
On Tue, Feb 18, 2020 at 6:49 PM Erick Ramirez
wrote:
> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>>
>
> Which exact version of C* is it again?
>
>> WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
>> Incomin
>
> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>
Which exact version of C* is it again?
> WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
> socket; closing
>
This is expected
Thanks Jonathan, it still helps...
Anyone knows the solution?
On Tue, 18 Feb 2020 at 18:08, Jonathan Koppenhofer
wrote:
> Forensics are gone at this point, so I can't verify exact errors, but
> wanted to mention we had seen something similar to corroborate your
> experience and warn others.
>
>
Forensics are gone at this point, so I can't verify exact errors, but
wanted to mention we had seen something similar to corroborate your
experience and warn others.
The version would have been 3.0.15 or 3.11.3 as that is what we were
deploying on our clusters at the time. I think it was more like
Jonathan,
As per https://issues.apache.org/jira/browse/CASSANDRA-13696 the issue,
Digest mismatch Exception if hints file has UnknownColumnFamily, is fixed
for 3.0.15 , did you still faced this issue on 3.0.15 ?
Thanks
Surbhi
On Tue, 18 Feb 2020 at 17:40, Jonathan Koppenhofer
wrote:
> I believ
I believe we had something similar happen on 3.0.15 a while back. We had a
cluster that created mass havoc by creating MVs on a large existing
dataset. We thought we had stabilized the cluster, but saw similar issues
as you when we dropped the MVs.
We interpreted our errors to mean that we should
We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
So we had to drop 7 MVs in production, as soon as we dropped the first
Materialized View, our cluster became unstable and app started giving 100%
error, what we noticed:
1. As soon as MV was dropped , cluster became unstable and
Thanks Eric ...
This is helpful...
On Wed, 12 Feb 2020 at 17:46, Erick Ramirez
wrote:
> There shouldn't be any negative impact from dropping MVs and there's
> certainly no risk to the base table if that is your concern. All it will do
> is remove all the data in the respective views plus drop a
There shouldn't be any negative impact from dropping MVs and there's
certainly no risk to the base table if that is your concern. All it will do
is remove all the data in the respective views plus drop any pending view
mutations from the batch log. If anything, you should see some performance
gain
22 matches
Mail list logo
