This deserves a JIRA ticket please. (I assume the sending host is randomly choosing the bad IP and blocking on it for some period of time, causing other tasks to pile up, but it should be investigated as a regression).
On Tue, Jun 7, 2022 at 7:52 AM Gil Ganz <gilg...@gmail.com> wrote: > Yes, I know the issue with the peers table, we had it in different > clusters, in this case it appears the cause of the problem was indeed a bad > ip in the seed list. > After removing it from all nodes and reloading seeds, running a rolling > restart does not cause any gossip issues, and in general the number of > gossip pending tasks is 0 all the time, vs jumping to 2-5 pending tasks > every once in a while before this change. > > Interesting that this bad ip didn't cause an issue in 3.11.9, I guess > something in the way gossip works in c*4 made it so it caused a real issue > after the upgrade. > > On Tue, Jun 7, 2022 at 12:04 PM Bowen Song <bo...@bso.ng> wrote: > >> Regarding the "ghost IP", you may want to check the system.peers_v2 table >> by doing "select * from system.peers_v2 where peer = '123.456.789.012';" >> >> I've seen this (non-)issue many times, and I had to do "delete from >> system.peers_v2 where peer=..." to fix it, as on our client side, the >> Python cassandra-driver, reads the token ring information from this table >> and uses it for routing requests. >> On 07/06/2022 05:22, Gil Ganz wrote: >> >> Only errors I see in the logs prior to gossip pending issue are things >> like this >> >> INFO [Messaging-EventLoop-3-32] 2022-06-02 20:29:44,833 >> NoSpamLogger.java:92 - /X:7000->/Y:7000-URGENT_MESSAGES-[no-channel] failed >> to connect >> io.netty.channel.AbstractChannel$AnnotatedConnectException: >> finishConnect(..) failed: No route to host: /Y:7000 >> Caused by: java.net.ConnectException: finishConnect(..) failed: No route >> to host >> at >> io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) >> at io.netty.channel.unix.Socket.finishConnect(Socket.java:251) >> at >> io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:673) >> at >> io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650) >> at >> io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530) >> at >> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470) >> at >> io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) >> at >> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >> at >> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) >> at java.lang.Thread.run(Thread.java:748) >> >> Remote ip mentioned here is an ip that is appearing in the seed list >> (there are 20 other valid ip addresses in the seed clause), but it's no >> longer a valid ip, it's an old ip of an existing server (it's not in the >> peers table). I will try to reproduce the issue with this this ip removed >> from seed list >> >> >> On Mon, Jun 6, 2022 at 9:39 PM C. Scott Andreas <sc...@paradoxica.net> >> wrote: >> >>> Hi Gil, thanks for reaching out. >>> >>> Can you check Cassandra's logs to see if any uncaught exceptions are >>> being thrown? What you described suggests the possibility of an uncaught >>> exception being thrown in the Gossiper thread, preventing further tasks >>> from making progress; however I'm not aware of any open issues in 4.0.4 >>> that would result in this. >>> >>> Would be eager to investigate immediately if so. >>> >>> – Scott >>> >>> On Jun 6, 2022, at 11:04 AM, Gil Ganz <gilg...@gmail.com> wrote: >>> >>> >>> Hey >>> We have a big cluster (>500 nodes, onprem, multiple datacenters, most >>> with vnodes=32, but some with 128), that was recently upgraded from 3.11.9 >>> to 4.0.4. Servers are all centos 7. >>> >>> We have been dealing with a few issues related to gossip since : >>> 1 - The moment the last node in the cluster was up with 4.0.4, and all >>> nodes were in the same version, gossip pending tasks started to climb to >>> very high numbers (>1M) in all nodes in the cluster, and quickly the >>> cluster was practically down. Took us a few hours of stopping/starting up >>> nodes, and adding more nodes to the seed list, to finally get the cluster >>> back up. >>> 2 - We notice that pending gossip tasks go up to very high >>> numbers (50k), in random nodes in the cluster, without any meaningful event >>> that happened and it doesn't look like it will go down on its own. After a >>> few hours we restart those nodes and it goes back to 0. >>> 3 - Doing a rolling restart to a list of servers is now an issue, more >>> often then not, what will happen is one of the nodes we restart goes up >>> with gossip issues, and we need a 2nd restart to get the gossip pending >>> tasks to 0. >>> >>> Is there a known issue related to gossip in big clusters, in recent >>> versions? >>> Is there any tuning that can be done? >>> >>> Just to give a sense of how big the gossip information in this cluster, >>> "*nodetool >>> gossipinfo*" output size is ~300kb >>> >>> gil >>> >>> >>>
