wolf27w opened a new issue, #8255: URL: https://github.com/apache/rocketmq/issues/8255
### Before Creating the Bug Report - [X] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [X] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment 运行平台是kubernetes中部署的多主多从集群 ### RocketMQ version rocketmq集群版本4.9.7 ### JDK Version openjdk1.8 ### Describe the Bug 我在kubernetes中部署了一个rocketmq多主多从集群，在集群内部访问没有任何问题，但是我通过代理访问集群内部的rocketmq集群就会出现消息重复，线看一下我的集群  我的多主多从配置 broker-a.conf ``` brokerClusterName=DefaultCluster brokerName=broker-a brokerEnableProxy=true brokerId=0 namesrvAddr=mq-ns.ops.svc.cluster.local:9876 brokerIP1=192.168.6.93 defaultTopicQueueNums=4 #Whether to allow the broker to automatically create topics, it is recommended to open offline and close online autoCreateTopicEnable=true #Whether to allow the broker to automatically create a subscription group, it is recommended to start offline and close online autoCreateSubscriptionGroup=true listenPort=10911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 destroyMapedFileInterval=120000 redeleteHangedFileInterval=120000 diskMaxUsedSpaceRatio=88 maxMessageSize=65536 flushCommitLogLeastPages=4 flushConsumeQueueLeastPages=2 flushCommitLogThoroughInterval=10000 flushConsumeQueueThoroughInterval=60000 checkTransactionMessageEnable=false sendMessageThreadPoolNums=16 pullMessageThreadPoolNums=16 brokerRole=ASYNC_MASTER flushDiskType=ASYNC_FLUSH ``` broker-b.conf ``` brokerClusterName=DefaultCluster brokerName=broker-b brokerId=0 brokerEnableProxy=true brokerIP1=192.168.6.93 namesrvAddr=mq-ns.ops.svc.cluster.local:9876 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=10911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 destroyMapedFileInterval=120000 redeleteHangedFileInterval=120000 diskMaxUsedSpaceRatio=88 maxMessageSize=65536 flushCommitLogLeastPages=4 flushConsumeQueueLeastPages=2 flushCommitLogThoroughInterval=10000 flushConsumeQueueThoroughInterval=60000 checkTransactionMessageEnable=false sendMessageThreadPoolNums=16 pullMessageThreadPoolNums=16 brokerRole=ASYNC_MASTER flushDiskType=ASYNC_FLUSH ``` 先看一下我的nodeport配置  我的nginx代理配置，在93上面 ``` stream { upstream prod-eft-link-mq9876 { least_conn; server 192.168.6.237:30740; server 192.168.6.236:30740; } upstream prod-eft-link-mq10909 { least_conn; server 192.168.6.236:30532; server 192.168.6.237:32727; } upstream prod-eft-link-mq10911 { least_conn; server 192.168.6.236:31213; server 192.168.6.237:32742; } server { listen 9876;#9876 proxy_pass prod-eft-link-mq9876; } server { listen 10909;#10909 proxy_pass prod-eft-link-mq10909; } server { listen 10911; #10911 proxy_pass prod-eft-link-mq10911; } } ``` ### Steps to Reproduce 如果我在broker-a和broker-b的主节点里增加了brokerIP1=192.168.6.93配置，我就会消息里出现重复  如果我不在配置文件里增加brokerIP1=192.168.6.93，消息不重复了，但是我客户端通过nginx无法直接和rocketmq集群通信 ### What Did You Expect to See? 我希望rocketmq集群可以实现客户端通过代理连接到rocketmq集群，并且消息也不会重复 ### What Did You See Instead? 这里是我的yaml文件 ``` cat broker-a.yaml --- apiVersion: v1 data: broker.conf: |- brokerClusterName=DefaultCluster brokerName=broker-a brokerId=0 namesrvAddr=mq-ns.ops.svc.cluster.local:9876 defaultTopicQueueNums=4 #Whether to allow the broker to automatically create topics, it is recommended to open offline and close online autoCreateTopicEnable=true #Whether to allow the broker to automatically create a subscription group, it is recommended to start offline and close online autoCreateSubscriptionGroup=true listenPort=10911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 destroyMapedFileInterval=120000 redeleteHangedFileInterval=120000 diskMaxUsedSpaceRatio=88 maxMessageSize=65536 flushCommitLogLeastPages=4 flushConsumeQueueLeastPages=2 flushCommitLogThoroughInterval=10000 flushConsumeQueueThoroughInterval=60000 checkTransactionMessageEnable=false sendMessageThreadPoolNums=16 pullMessageThreadPoolNums=16 brokerRole=ASYNC_MASTER flushDiskType=ASYNC_FLUSH kind: ConfigMap metadata: creationTimestamp: null name: broker-a cat broker-a-s.yaml --- apiVersion: v1 data: broker.conf: |- brokerClusterName=DefaultCluster brokerName=broker-a brokerId=1 namesrvAddr=mq-ns.ops.svc.cluster.local:9876 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=10911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 destroyMapedFileInterval=120000 redeleteHangedFileInterval=120000 diskMaxUsedSpaceRatio=88 maxMessageSize=65536 flushCommitLogLeastPages=4 flushConsumeQueueLeastPages=2 flushCommitLogThoroughInterval=10000 flushConsumeQueueThoroughInterval=60000 checkTransactionMessageEnable=false sendMessageThreadPoolNums=16 pullMessageThreadPoolNums=16 brokerRole=SLAVE flushDiskType=ASYNC_FLUSH kind: ConfigMap metadata: creationTimestamp: null name: broker-a-s cat broker-b.yaml --- apiVersion: v1 data: broker.conf: |- brokerClusterName=DefaultCluster brokerName=broker-b brokerId=0 namesrvAddr=mq-ns.ops.svc.cluster.local:9876 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=10911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 destroyMapedFileInterval=120000 redeleteHangedFileInterval=120000 diskMaxUsedSpaceRatio=88 maxMessageSize=65536 flushCommitLogLeastPages=4 flushConsumeQueueLeastPages=2 flushCommitLogThoroughInterval=10000 flushConsumeQueueThoroughInterval=60000 checkTransactionMessageEnable=false sendMessageThreadPoolNums=16 pullMessageThreadPoolNums=16 brokerRole=ASYNC_MASTER flushDiskType=ASYNC_FLUSH kind: ConfigMap metadata: creationTimestamp: null name: broker-b cat broker-b-s.yaml --- apiVersion: v1 data: broker.conf: |- brokerClusterName=DefaultCluster brokerName=broker-b brokerId=1 namesrvAddr=mq-ns.ops.svc.cluster.local:9876 defaultTopicQueueNums=4 autoCreateTopicEnable=true autoCreateSubscriptionGroup=true listenPort=10911 deleteWhen=04 fileReservedTime=120 mapedFileSizeCommitLog=1073741824 mapedFileSizeConsumeQueue=300000 destroyMapedFileInterval=120000 redeleteHangedFileInterval=120000 diskMaxUsedSpaceRatio=88 maxMessageSize=65536 flushCommitLogLeastPages=4 flushConsumeQueueLeastPages=2 flushCommitLogThoroughInterval=10000 flushConsumeQueueThoroughInterval=60000 checkTransactionMessageEnable=false sendMessageThreadPoolNums=16 pullMessageThreadPoolNums=16 brokerRole=SLAVE flushDiskType=ASYNC_FLUSH kind: ConfigMap metadata: creationTimestamp: null name: broker-b-s ``` pod的yaml文件信息 ``` cat broker-a-pods.yaml apiVersion: v1 kind: Service metadata: labels: app: broker-a name: broker-a spec: # type: NodePort type: ClusterIP ports: - port: 10911 targetPort: 10911 name: broker-port - port: 10909 targetPort: 10909 name: broker-vip - port: 10912 targetPort: 10912 name: broker-hp selector: app: broker-a --- apiVersion: apps/v1 kind: StatefulSet metadata: name: broker-a spec: serviceName: broker-a replicas: 1 selector: matchLabels: app: broker-a template: metadata: labels: app: broker-a spec: # affinity: # #podAntiAffinity 反亲合性 # podAntiAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # - labelSelector: # matchExpressions: # - key: "app" # operator: In # values: # - broker-a # topologyKey: "kubernetes.io/hostname" containers: - name: broker-a image: apache/rocketmq:4.9.7 imagePullPolicy: IfNotPresent env: - name: TZ value: Asia/Shanghai - name: NAMESRV_ADDR value: "mq-ns.ops.svc.cluster.local:9876" - name: JAVA_OPT_EXT value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m" #command: ["sh","-c","mqbroker -c /home/broker.conf"] command: - /bin/sh args: - mqbroker - '-c' - /home/broker.conf volumeMounts: - name: rocketmq-data mountPath: /home/rocketmq/logs - name: rocketmq-data mountPath: /home/rocketmq/store - name: broker-config mountPath: /home/broker.conf subPath: broker.conf readinessProbe: tcpSocket: port: 10911 initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 20 volumes: - name: broker-config configMap: name: broker-a #启动依赖 initContainers: - name: init-mq-ns image: busybox:latest command: ['sh', '-c', "until nslookup mq-ns.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mq-ns; sleep 2; done"] volumeClaimTemplates: - metadata: name: rocketmq-data annotations: volume.beta.kubernetes.io/storage-class: "nfs-storage" spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi cat broker-a-s-pods.yaml apiVersion: v1 kind: Service metadata: labels: app: broker-a-s name: broker-a-s spec: # type: NodePort type: ClusterIP ports: - port: 10911 targetPort: 10911 name: broker-port - port: 10909 targetPort: 10909 name: broker-vip - port: 10912 targetPort: 10912 name: broker-hp selector: app: broker-a-s --- apiVersion: apps/v1 kind: StatefulSet metadata: name: broker-a-s spec: serviceName: broker-a-s replicas: 1 selector: matchLabels: app: broker-a-s template: metadata: labels: app: broker-a-s spec: # affinity: # #podAntiAffinity 反亲合性 # podAntiAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # - labelSelector: # matchExpressions: # - key: "app" # operator: In # values: # - broker-a-s # topologyKey: "kubernetes.io/hostname" containers: - name: broker-a-s image: apache/rocketmq:4.9.7 #下面使用的是用官方地址自行构建的镜像，推荐直接使用这个 #image: apacherocketmq/rocketmq:4.8.0 imagePullPolicy: IfNotPresent env: - name: TZ value: Asia/Shanghai - name: NAMESRV_ADDR value: "mq-ns.ops.svc.cluster.local:9876" - name: JAVA_OPT_EXT value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m" #command: ["sh","-c","mqbroker -c /home/broker.conf"] command: - /bin/sh args: - mqbroker - '-c' - /home/broker.conf volumeMounts: - name: rocketmq-data mountPath: /home/rocketmq/logs - name: rocketmq-data mountPath: /home/rocketmq/store - name: broker-config mountPath: /home/broker.conf subPath: broker.conf readinessProbe: tcpSocket: port: 10911 initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 20 volumes: - name: broker-config configMap: name: broker-a-s #启动依赖 initContainers: - name: init-mq-ns image: busybox:latest command: ['sh', '-c', "until nslookup mq-ns.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mq-ns; sleep 2; done"] volumeClaimTemplates: - metadata: name: rocketmq-data annotations: volume.beta.kubernetes.io/storage-class: "nfs-storage" spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi cat broker-b-pods.yaml apiVersion: v1 kind: Service metadata: labels: app: broker-b name: broker-b spec: # type: NodePort type: ClusterIP ports: - port: 10911 targetPort: 10911 name: broker-port - port: 10909 targetPort: 10909 name: broker-vip - port: 10912 targetPort: 10912 name: broker-hp selector: app: broker-b --- apiVersion: apps/v1 kind: StatefulSet metadata: name: broker-b spec: serviceName: broker-b replicas: 1 selector: matchLabels: app: broker-b template: metadata: labels: app: broker-b spec: # affinity: # #podAntiAffinity 反亲合性 # podAntiAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # - labelSelector: # matchExpressions: # - key: "app" # operator: In # values: # - broker-b # topologyKey: "kubernetes.io/hostname" containers: - name: broker-b image: apache/rocketmq:4.9.7 #下面使用的是用官方地址自行构建的镜像，推荐直接使用这个 #image: apacherocketmq/rocketmq:4.8.0 imagePullPolicy: IfNotPresent env: - name: TZ value: Asia/Shanghai - name: NAMESRV_ADDR value: "mq-ns.ops.svc.cluster.local:9876" - name: JAVA_OPT_EXT value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m" #command: ["sh","-c","mqbroker -c /home/broker.conf"] command: - /bin/sh args: - mqbroker - '-c' - /home/broker.conf volumeMounts: - name: rocketmq-data mountPath: /home/rocketmq/logs - name: rocketmq-data mountPath: /home/rocketmq/store - name: broker-config mountPath: /home/broker.conf subPath: broker.conf readinessProbe: tcpSocket: port: 10911 initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 20 volumes: - name: broker-config configMap: name: broker-b #启动依赖 initContainers: - name: init-mq-ns image: busybox:latest command: ['sh', '-c', "until nslookup mq-ns.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mq-ns; sleep 2; done"] volumeClaimTemplates: - metadata: name: rocketmq-data annotations: volume.beta.kubernetes.io/storage-class: "nfs-storage" spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi cat broker-b-s-pods.yaml apiVersion: v1 kind: Service metadata: labels: app: broker-b-s name: broker-b-s spec: # type: NodePort type: ClusterIP ports: - port: 10911 targetPort: 10911 name: broker-port - port: 10909 targetPort: 10909 name: broker-vip - port: 10912 targetPort: 10912 name: broker-hp selector: app: broker-b-s --- apiVersion: apps/v1 kind: StatefulSet metadata: name: broker-b-s spec: serviceName: broker-b-s replicas: 1 selector: matchLabels: app: broker-b-s template: metadata: labels: app: broker-b-s spec: # affinity: # #podAntiAffinity 反亲合性 # podAntiAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # - labelSelector: # matchExpressions: # - key: "app" # operator: In # values: # - broker-b-s # topologyKey: "kubernetes.io/hostname" containers: - name: broker-b-s image: apache/rocketmq:4.9.7 #下面使用的是用官方地址自行构建的镜像，推荐直接使用这个 #image: apacherocketmq/rocketmq:4.8.0 imagePullPolicy: IfNotPresent env: - name: TZ value: Asia/Shanghai - name: NAMESRV_ADDR value: "mq-ns.ops.svc.cluster.local:9876" - name: JAVA_OPT_EXT value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m" #command: ["sh","-c","mqbroker -c /home/broker.conf"] command: - /bin/sh args: - mqbroker - '-c' - /home/broker.conf volumeMounts: - name: rocketmq-data mountPath: /home/rocketmq/logs - name: rocketmq-data mountPath: /home/rocketmq/store - name: broker-config mountPath: /home/broker.conf subPath: broker.conf readinessProbe: tcpSocket: port: 10911 initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 20 volumes: - name: broker-config configMap: name: broker-b-s #启动依赖 initContainers: - name: init-mq-ns image: busybox:latest command: ['sh', '-c', "until nslookup mq-ns.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mq-ns; sleep 2; done"] volumeClaimTemplates: - metadata: name: rocketmq-data annotations: volume.beta.kubernetes.io/storage-class: "nfs-storage" spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi cat mq-console.yaml apiVersion: v1 kind: Service metadata: labels: app: mq-console name: mq-cs spec: type: ClusterIP ports: - port: 8080 targetPort: 8080 name: console-port selector: app: mq-console --- apiVersion: apps/v1 kind: Deployment metadata: name: mq-console spec: replicas: 1 selector: matchLabels: app: mq-console template: metadata: labels: app: mq-console spec: # affinity: # #podAntiAffinity 反亲合性 # podAntiAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # - labelSelector: # matchExpressions: # - key: "app" # operator: In # values: # - mq-console # topologyKey: "kubernetes.io/hostname" containers: - name: mq-console image: gguilin/rocketmq-console-ng imagePullPolicy: IfNotPresent env: - name: JAVA_OPTS value: "-Drocketmq.namesrv.addr=mq-ns.ops.svc.cluster.local:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false" readinessProbe: tcpSocket: port: 8080 initialDelaySeconds: 15 #启动依赖 initContainers: - name: init-mq-ns image: busybox:latest command: ['sh', '-c', "until nslookup mq-ns.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mq-ns; sleep 2; done"] cat mq-nmserv.yaml apiVersion: v1 kind: Service metadata: labels: app: mq-namesrv name: mq-ns spec: type: ClusterIP ports: - port: 9876 targetPort: 9876 name: namesrv-port selector: app: mq-namesrv --- apiVersion: apps/v1 kind: StatefulSet metadata: name: mq-namesrv spec: serviceName: mq-namesrv replicas: 1 selector: matchLabels: app: mq-namesrv template: metadata: labels: app: mq-namesrv spec: # affinity: # #podAntiAffinity 反亲合性 # podAntiAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # - labelSelector: # matchExpressions: # - key: "app" # operator: In # values: # - mq-namesrv # topologyKey: "kubernetes.io/hostname" containers: - name: mq-namesrv image: apache/rocketmq:4.9.7 imagePullPolicy: IfNotPresent env: - name: TZ value: Asia/Shanghai - name: JAVA_OPT_EXT value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m" #command: ["sh","-c","mqnamesrv"] command: - /bin/sh args: - mqnamesrv readinessProbe: tcpSocket: port: 9876 initialDelaySeconds: 15 timeoutSeconds: 5 periodSeconds: 20 ``` ### Additional Context 正常在rocketmq的集群就应该可以在集群外部可以接受访问的，而不是只是集群内部访问 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
