Talking about telnet, a closer look at the sequence 'FF F4 FF FD' makes
me thinking about telnet commands code. Based on RFC 854
<https://www.rfc-editor.org/rfc/rfc854.html>, the sequence is IAC,
Interrupt Process, IAC, DO, which is basically the key sequence 'ctrl-c'
in telnet.
On 29/06/2023 12:42, Bowen Song wrote:
Did anyone connecting to the servers' storage port via telnet, nc
(netcat) or something similar? 218762506 is 0x0D0A0D0A, which is two
newlines.
On 29/06/2023 11:49, MyWorld wrote:
When checked in the source nodes, we got similar errors.
Forgot to mention, we also received below error message :
ERROR [Messaging-EventLoop-3-3] 2023-06-27 18:57:09,128
InboundConnectionInitiator.java:360 - Failed to properly handshake
with peer /10.127.2.10:58490 <http://10.127.2.10:58490>. Closing the
channel.
io.netty.handler.codec.DecoderException:
org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read
218762506, Expected -900387334
On Thu, Jun 29, 2023 at 2:57 PM Bowen Song via user
<user@cassandra.apache.org> wrote:
The expected value "-900387334" is the little endian decimal
representation of the PROTOCOL_MAGIC value 0xCA552DFA defined in
the net/Message.java
<https://github.com/apache/cassandra/blob/c579faa488ec156a59ed8e15dd6db55759b9c942/src/java/org/apache/cassandra/net/Message.java#L393>
file.
The read value "-720899" converts to hex is 0xFFF4FFFD, that's
not a valid TLS header which should start with 0x16, so I don't
think has anything to do with the server encryption related
options. It also does not look like a valid version number from
pre-4.0 Cassandra, so we can rule that out too. Since it's nether
a valid Cassandra 4.0+ magic, a TLS header nor a pre-4.0 version
number, I have reason to believe the connection was not initiated
by another Cassandra server for inter-node communication, but
from another program. Can you follow the source IP and port
number back to the originating host, and find out what is that
program? or indeed it was one of the servers in the cluster, not
something else, which could indicate a misconfiguration of the
firewall rules.
On 29/06/2023 01:26, MyWorld wrote:
Hi all,
We are currently using Apache cassandra 4.0.7 in our
environment. While adding a new node in the existing 3-node DC,
we found below error.
This error is observed multiple times when the node was in the
UJ (up and joining) state.
Our current server-to-server internode encryption settings are
default.
server_encryption_options:
internode_encryption: none
enable_legacy_ssl_storage_port: false
require_client_auth: false
require_endpoint_verification: false
Please help to debug the root cause of this error.
Is it a point to worry about or is it just a Warning issue?
Also, our API properties have received a few 5xx messages
"Operation timed out. received only 2 responses" during this
time(addition of new node), which we have not received when we
were on the 3.11.x version. What could be the possible reason?
However things are stable once the node comes to the UN state.
ERROR [Messaging-EventLoop-3-10] 2023-06-27 18:37:14,931
InboundConnectionInitiator.java:360 - Failed to properly
handshake with peer /x.x.x.x:35894. Closing the channel.
io.netty.handler.codec.DecoderException:
org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic:
Read -720899, Expected -900387334
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by:
org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic:
Read -720899, Expected -900387334
at
org.apache.cassandra.net.Message.validateLegacyProtocolMagic(Message.java:340)
at
org.apache.cassandra.net.HandshakeProtocol$Initiate.maybeDecode(HandshakeProtocol.java:167)
at
org.apache.cassandra.net.InboundConnectionInitiator$Handler.initiate(InboundConnectionInitiator.java:242)
at
org.apache.cassandra.net.InboundConnectionInitiator$Handler.decode(InboundConnectionInitiator.java:235)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
... 15 common frames omitted
Regards,
Ashish
