liuyk0109 opened a new issue #2574:
URL: https://github.com/apache/rocketmq/issues/2574
I build the namesrv and broker container myself and run them on k8s, and use
the dledger mode. They can start and look like normal.
But when i send a message with mqadmin, some exceptions were thrown.
I run the cluster on k8s namespace rocketmq, there is my broker config file:
n0:
```
brokerClusterName = RaftCluster
brokerName=RaftNode00
brokerIP1=rocketmq-broker-0.rocketmq-broker-service.rocketmq
brokerIP2=rocketmq-broker-0.rocketmq-broker-service.rocketmq
listenPort=10911
namesrvAddr=rocketmq-namesrv-0.rocketmq-namesrv-service:9876;rocketmq-namesrv-1.rocketmq-namesrv-service:9876;rocketmq-namesrv-2.rocketmq-namesrv-service:9876
storePathRootDir=/mountdir/rmqstore/n0
storePathCommitLog=/mountdir/rmqstore/n0/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n0-rocketmq-broker-0.rocketmq-broker-service:20911;n1-rocketmq-broker-1.rocketmq-broker-service:20911;n2-rocketmq-broker-2.rocketmq-broker-service:20911
## must be unique
dLegerSelfId=n0
sendMessageThreadPoolNums=16
autoCreateTopicEnable=false
```
n1:
```
brokerClusterName = RaftCluster
brokerName=RaftNode00
brokerIP1=rocketmq-broker-1.rocketmq-broker-service.rocketmq
brokerIP2=rocketmq-broker-1.rocketmq-broker-service.rocketmq
listenPort=10911
namesrvAddr=rocketmq-namesrv-0.rocketmq-namesrv-service:9876;rocketmq-namesrv-1.rocketmq-namesrv-service:9876;rocketmq-namesrv-2.rocketmq-namesrv-service:9876
storePathRootDir=/mountdir/rmqstore/n1
storePathCommitLog=/mountdir/rmqstore/n1/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n0-rocketmq-broker-0.rocketmq-broker-service:20911;n1-rocketmq-broker-1.rocketmq-broker-service:20911;n2-rocketmq-broker-2.rocketmq-broker-service:20911
## must be unique
dLegerSelfId=n1
sendMessageThreadPoolNums=16
autoCreateTopicEnable=false
```
n2:
```
brokerClusterName = RaftCluster
brokerName=RaftNode00
brokerIP1=rocketmq-broker-2.rocketmq-broker-service.rocketmq
brokerIP2=rocketmq-broker-2.rocketmq-broker-service.rocketmq
listenPort=10911
namesrvAddr=rocketmq-namesrv-0.rocketmq-namesrv-service:9876;rocketmq-namesrv-1.rocketmq-namesrv-service:9876;rocketmq-namesrv-2.rocketmq-namesrv-service:9876
storePathRootDir=/mountdir/rmqstore/n2
storePathCommitLog=/mountdir/rmqstore/n2/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n0-rocketmq-broker-0.rocketmq-broker-service:20911;n1-rocketmq-broker-1.rocketmq-broker-service:20911;n2-rocketmq-broker-2.rocketmq-broker-service:20911
## must be unique
dLegerSelfId=n2
sendMessageThreadPoolNums=16
autoCreateTopicEnable=false
```
The `/mountdir` is nfs pvc path.
Some logs in the n0 broker that I think are useful:
```
2021-01-08 14:30:47 INFO brokerOutApi_thread_1 - register broker[1]to name
server rocketmq-namesrv-1.rocketmq-namesrv-service:9876 OK
2021-01-08 14:30:47 INFO brokerOutApi_thread_4 - register broker[1]to name
server rocketmq-namesrv-0.rocketmq-namesrv-service:9876 OK
2021-01-08 14:30:47 INFO brokerOutApi_thread_2 - register broker[1]to name
server rocketmq-namesrv-2.rocketmq-namesrv-service:9876 OK
2021-01-08 14:30:47 ERROR DiskCheckScheduledThread1 - Error when measuring
disk space usage, file doesn't exist on this path:
/mountdir/rmqstore/n0/commitlog
2021-01-08 14:30:52 INFO BrokerControllerScheduledThread1 - Update slave
topic config from master,
rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:30:53 INFO BrokerControllerScheduledThread1 - Update slave
consumer offset from master,
rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:30:53 INFO BrokerControllerScheduledThread1 - Update slave
delay offset from master,
rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:30:53 INFO BrokerControllerScheduledThread1 - Update slave
Subscription Group from master,
rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:30:57 ERROR DiskCheckScheduledThread1 - Error when measuring
disk space usage, file doesn't exist on this path:
/mountdir/rmqstore/n0/commitlog
2021-01-08 14:31:04 INFO BrokerControllerScheduledThread1 - Update slave
consumer offset from master,
rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:31:04 INFO BrokerControllerScheduledThread1 - Update slave
delay offset from master,
rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
2021-01-08 14:31:06 ERROR StoreScheduledThread1 - Error when measuring disk
space usage, file doesn't exist on this path:
/mountdir/rmqstore/n0/dledger-n0/data
```
Some files can't find:
```
[root@hadoop-dn02 rocketmq]# ll rmqstore/n0/
-rw-r--r-- 1 root root 0 1月 8 14:30 abort
-rw-r--r-- 1 root root 4096 1月 8 15:40 checkpoint
drwxr-xr-x 2 root root 280 1月 8 15:40 config
drwxr-xr-x 2 root root 82 1月 8 15:40 dledger-n0
-rw-r--r-- 1 root root 4 1月 8 14:30 lock
[root@hadoop-dn02 rocketmq]# ll rmqstore/n0/dledger-n0/
-rw-r--r-- 1 root root 30 1月 8 15:41 checkpoint
-rw-r--r-- 1 root root 30 1月 8 15:41 checkpoint.bak
-rw-r--r-- 1 root root 25 1月 8 14:30 currterm
-rw-r--r-- 1 root root 23 1月 8 14:30 currterm.bak
```
Entry the broker n0 container and run some commands by mqadmin:
clusterList:
```
[root@rocketmq-broker-0 bin]# sh mqadmin clusterList -n
rocketmq-namesrv-0.rocketmq-namesrv-service.rocketmq:9876
RocketMQLog:WARN No appenders could be found for logger
(io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
#Cluster Name #Broker Name #BID #Addr
#Version #InTPS(LOAD) #OutTPS(LOAD) #PCWait(ms) #Hour
#SPACE
RaftCluster RaftNode00 0
rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911 V4_8_0
0.00(0,0ms) 0.00(0,0ms) 0 447246.58 -1.0000
RaftCluster RaftNode00 1
rocketmq-broker-0.rocketmq-broker-service.rocketmq:10911 V4_8_0
0.00(0,0ms) 0.00(0,0ms) 0 447246.58 -1.0000
RaftCluster RaftNode00 3
rocketmq-broker-2.rocketmq-broker-service.rocketmq:10911 V4_8_0
0.00(0,0ms) 0.00(0,0ms) 0 447246.58 -1.0000
```
updateTopic:
```
[root@rocketmq-broker-0 bin]# sh mqadmin updateTopic -n
rocketmq-namesrv-0.rocketmq-namesrv-service.rocketmq:9876 -c RaftCluster -t
lyktest
RocketMQLog:WARN No appenders could be found for logger
(io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
create topic to rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
success.
TopicConfig [topicName=lyktest, readQueueNums=8, writeQueueNums=8, perm=RW-,
topicFilterType=SINGLE_TAG, topicSysFlag=0, order=false]
```
topicList:
```
sh mqadmin topicList -n
rocketmq-namesrv-0.rocketmq-namesrv-service.rocketmq:9876
RocketMQLog:WARN No appenders could be found for logger
(io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
RaftCluster_REPLY_TOPIC
SCHEDULE_TOPIC_XXXX
RMQ_SYS_TRANS_HALF_TOPIC
lyktest
RaftCluster
BenchmarkTest
OFFSET_MOVED_EVENT
RaftNode00
SELF_TEST_TOPIC
```
Everything looks normal, but when i send a message:
```
sh mqadmin sendMessage -n
rocketmq-namesrv-0.rocketmq-namesrv-service.rocketmq:9876 -t lyktest -p hello
RocketMQLog:WARN No appenders could be found for logger
(io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
java.lang.RuntimeException: SendMessageCommand command failed
at
org.apache.rocketmq.tools.command.message.SendMessageCommand.execute(SendMessageCommand.java:128)
at
org.apache.rocketmq.tools.command.MQAdminStartup.main0(MQAdminStartup.java:140)
at
org.apache.rocketmq.tools.command.MQAdminStartup.main(MQAdminStartup.java:91)
Caused by: org.apache.rocketmq.client.exception.MQClientException: Send [3]
times, still failed, cost [207]ms, Topic: lyktest, BrokersSent: [RaftNode00,
RaftNode00, RaftNode00]
See http://rocketmq.apache.org/docs/faq/ for further details.
at
org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:665)
at
org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1343)
at
org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1289)
at
org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:325)
at
org.apache.rocketmq.tools.command.message.SendMessageCommand.execute(SendMessageCommand.java:124)
... 2 more
Caused by: org.apache.rocketmq.client.exception.MQBrokerException: CODE: 1
DESC: java.lang.NullPointerException,
org.apache.rocketmq.common.message.MessageExt.socketAddress2ByteBuffer(MessageExt.java:77)
BROKER: rocketmq-broker-1.rocketmq-broker-service.rocketmq:10911
For more information, please visit the url,
http://rocketmq.apache.org/docs/faq/
at
org.apache.rocketmq.client.impl.MQClientAPIImpl.processSendResponse(MQClientAPIImpl.java:666)
at
org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessageSync(MQClientAPIImpl.java:505)
at
org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:487)
at
org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:431)
at
org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendKernelImpl(DefaultMQProducerImpl.java:854)
at
org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:584)
... 6 more
```
It can't work.
What happened in my cluster and how can i fix it? Or it's bug in 4.8.0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
