Reading from a non-existent table shouldn’t crash the database
Clearly the hint error invoked the fs error handler - probably incorrectly -
which shut down the db. That’s not ok and deserves a JIRA.
If dropping a table causes a hint checksum error then it needs to be fixed.
>> On Feb 18, 2020, at 6:49 PM, Erick Ramirez <erick.rami...@datastax.com>
>> wrote:
>
>> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>
> Which exact version of C* is it again?
>> WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
>> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
>> socket; closing
>
> This is expected because your app is trying to read the MV you just dropped.
>
>> ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 -
>> Attempting to mutate non-existant table 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
>
> Any view update transactions which are already in-flight will fail. Again,
> this is expected since the MV updates are not synchronous with the base table
> updates.
>
>> WARN [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 -
>> Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
>
> As above, this is expected since the MV updates which are already in the
> queue will fail to apply because the MV got dropped.
>
>> WARN [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 -
>> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c -
>> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file
>> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
>
> This is also expected. It won't be able to read the hint for a non-existent
> MV.
>
> So where to from here? The symptoms you described indicate that you haven't
> updated your app before you dropped the MVs. That is key here -- your app
> shouldn't be issuing reads for the MV since your intention is to drop it.
> What I think happened is that the nodes were backed up with read requests for
> the MV which no longer exists.
>
> Can you confirm that you have changed your application so it is not supposed
> to issue reads to the MV? You shouldn't drop MVs if you're still going to
> issue read requests from them just like you shouldn't drop tables your app is
> still expected to access. I hope that makes sense. Cheers!
