Hi, I have finally had time to test this a bit. And the issue seems to be there with Cassandra 3.11.13 as well as 3.0.27. Hinted handoff from these versions to version 4.0.6 does not seem to work.
The way I tested this was by setting up a 2 node cluster, 1 node in each of dc1 and dc2. Install version 3.11.13 (or 3.0.27) on both of them and confirm that hinted handoff is working by taking one node down and then write 100k lines with cassandra-stress. Take it back up and see that the data shows up in the DC that was down, the hints files disappear from the node that was written to and the system.log shows that hinted handoff was completed. After that I truncate the table that was written to and upgrade one node to 4.0.6. Repeat the process above only to see that the node gets no data when it gets back up, the hint files are not deleted on the node that was written to and there is no mention of hinted handoff in the logs. Is this supposed to work? And if not, how is it recommended to upgrade production clusters without significant data inconsistencies? Regards Morten Iversen On Tue, 2022-08-23 at 11:35 -0400, Jim Shaw wrote: > Is it over max hint window ? if over, better to do a full repair. > check table system.hints, do you see rows ? > > As I remember, during upgrade, transactions will store in hints until > other cluster have done upgrade, so for safety, change default 3 > hours hint window to long time just before starting upgrade. > > Regards, > > Jim > > > On Tue, Aug 23, 2022 at 8:33 AM Morten A. Iversen via user > <user@cassandra.apache.org> wrote: > > Hi, > > > > We are currently in the process of upgrading our environment from > > 3.0.27 to 4.0.4. However I see some issues with hints not being > > sent > > from v3 nodes to v4 nodes. > > > > We have a test environment with 2DCs, we are currently writing to > > DC1 > > and DC2 have been upgraded from version 3.0.27 -> 4.0.4 > > > > After the upgrade I see that all the v3 nodes in DC1 have hint > > files > > stored for all the upgraded nodes in DC2. These files where created > > while the nodes in DC2 was upgraded, however they do not disappear > > and > > there is nothing in the logs mentioning anything about these files. > > > > Does this mean that data written while a node is being upgraded is > > not > > replicated to that node? > > > > Hint related configs: > > hinted_handoff_enabled: true > > max_hint_window_in_ms: 172800000 # 48 > > hourshinted_handoff_throttle_in_kb: 10240 > > max_hints_delivery_threads: 8 > > hints_directory: /var/lib/cassandra/hints > > hints_flush_period_in_ms: 10000 > > max_hints_file_size_in_mb: 12 > > > > Regards > > Morten
