syhleo opened a new issue, #8468:
URL: https://github.com/apache/rocketmq/issues/8468
### Before Creating the Enhancement Request
- [X] I have confirmed that this should be classified as an enhancement
rather than a bug/feature.
### Summary
The solution provides an extensible message grayscale solution for
implementing message grayscale publishing that supports RocketMQ 4.x and
RocketMQ 5.x. It is compatible with RocketMQ 5.x's POP consumption mode and
RocketMQ 4.x's Push consumption mode. No matter the client-rebalance or
server-rebalance, the solution can realize the message gray level in a
lightweight way.
### Motivation
At present, many business systems control the impact of version release
through the gray scale release process, and almost all service releases will
strictly go gray scale release first, and then go online after verification.
gRPC and HTTP calls during GRay-scale publishing can be precisely controlled
through microservice governance and gateway request headers.
However, if the message gray scale scheme is not implemented, even if
the current link is a gray scale link, the message transmission is still
uncontrollable, may go to the gray cluster monitoring partition, may also enter
the online cluster monitoring partition. Therefore, when it comes to consuming
logic changes, developers need to incorporate a lot of compatibility logic into
their code. Even so, these logics only ensure that new traffic does not affect
the line, but they do not guarantee that gray-scale traffic precisely enters
the gray-scale consumer client, so that strict gray-scale verification cannot
be performed.
the industry lacks a perfect message gray scale solution, the existing
MQ gray scale solution can not completely solve the problem of message
isolation and switching interface. Therefore, it is particularly important to
design a lightweight message gray scale scheme that is easy to access and
controllable to upgrade.
### Describe the Solution You'd Like
1. Add client configuration: Add `enableGraySwitch` and `grayTag`.
`enableGraySwitch` Indicates whether the grayscale message is enabled. grayTag
indicates whether the client is a grayscale client. `grayTag` takes effect only
when `enableGraySwitch` is enabled. The solution adds two configuration items
that make it easy for the RocketMQ consumer to determine whether a client is a
grayscale client.
2. Use the clientId generation mechanism: If the client is a grayscale
client (`enableGraySwitch` and `grayTag` are both true), its clientId will
contain the `@gray` identifier.
3. Producer message sending strategy: The producer client decides to send
the message to grayscale partition or non-grayscale partition according to
whether there is a grayscale identifier. For this purpose, the
`SelectMessageQueueByGray` policy is implemented specifically to ensure that
producers send messages accurately to the specified `MessageQueue` when they
are sent.
4. The new weight balance strategy: to design a new weight balance strategy,
expand the `AllocateMessageQueueAveragely`, compatible RocketMQ 4 x and 5. X
version different versions, and different consumption patterns (such as
POP/Push). The core idea is that during rebalancing, grayscale clients are
distinguished from non-grayscale clients by whether or not there is a `@gray`
identifier in the `clientId` string, ensuring that grayscale clients consume
from grayscale queues and non-grayscale clients consume from non-grayscale
queues.
5. Compatible with POP consumption mode: The program also takes into account
POP consumption mode. However, the scheme will be divided into gray scale queue
and non-gray scale queue according to gray scale identification. The feature of
POP consumption mode is still retained, that is, multiple POP Normal consumers
can consume the same Normal Queue. Similarly, multiple POP Gray consumers can
consume the same Gray Queue. Ensure that every consumer client can consume
messages.
6. Switch connection processing: When all gray consumer clients do not exist
(for example, the gray verification on the consumer side is published online,
or the gray consumer side is abnormally offline), the message will be taken
over by other normal consumer clients immediately to ensure that the message
will not be lost.
### Describe Alternatives You've Considered
/
### Additional Context
This solution is lightweight, easy to access and upgrade controllable
message grayscale scheme. Compatible with RocketMQ4.x and RocketMQ5.x versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
