(rocketmq-eventbridge) branch feat/runtime-manager updated: feat:add test case.
This is an automated email from the ASF dual-hosted git repository.
shenlin pushed a commit to branch feat/runtime-manager
in repository https://gitbox.apache.org/repos/asf/rocketmq-eventbridge.git
The following commit(s) were added to refs/heads/feat/runtime-manager by this
push:
new 2da769b feat:add test case.
2da769b is described below
commit 2da769bb2a91b0bea5726ba43fbe5620e1498d1c
Author: 2011shenlin <2011shen...@gmail.com>
AuthorDate: Wed Jan 17 16:39:54 2024 +0800
feat:add test case.
---
.../ClusterSelectorService.java} | 14 ++--
.../runtime/manager/cluster/ClusterService.java| 3 +-
.../manager/dispatch/RunnerTaskDispatcher.java | 31 +++--
...nnerTaskRebalance.java => WorkerRebalance.java} | 2 +-
.../{ClusterWorkerScale.java => ClusterScale.java} | 4 +-
.../runtime/manager/task/RunnerTaskService.java| 9 +++
.../adapter/runtime/manager/watch/WatchWorker.java | 3 +-
.../WorkerLoadService.java}| 11 ++--
.../WorkerSelectorService.java}| 26 +---
.../WorkerInstanceRepositoryOnK8STest.java | 73 +
rocketmq_eventbridge.mv.db | Bin 81920 -> 77824 bytes
11 files changed, 147 insertions(+), 29 deletions(-)
diff --git
a/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/task/RunnerTaskService.java
b/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/cluster/ClusterSelectorService.java
similarity index 73%
copy from
adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/task/RunnerTaskService.java
copy to
adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/cluster/ClusterSelectorService.java
index 67254db..6ba2766 100644
---
a/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/task/RunnerTaskService.java
+++
b/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/cluster/ClusterSelectorService.java
@@ -15,14 +15,18 @@
* limitations under the License.
*/
-package org.apache.rocketmq.eventbridge.adapter.runtime.manager.task;
+package org.apache.rocketmq.eventbridge.adapter.runtime.manager.cluster;
-import java.util.List;
+import org.apache.rocketmq.eventbridge.adapter.runtime.manager.task.RunnerTask;
+import org.springframework.stereotype.Service;
-public class RunnerTaskService {
+@Service
+public class ClusterSelectorService {
+public Cluster selectCluster(RunnerTask runnerTask) {
+return selectDefaultCluster();
+}
-List listRunnerTask(String runnerName) {
+public Cluster selectDefaultCluster() {
return null;
}
-
}
\ No newline at end of file
diff --git
a/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/cluster/ClusterService.java
b/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/cluster/ClusterService.java
index 7ac59ca..ef27461 100644
---
a/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/cluster/ClusterService.java
+++
b/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/cluster/ClusterService.java
@@ -38,10 +38,11 @@ public class ClusterService {
return true;
}
-public Cluster getCluster(String clusterName) {
+public Cluster getCluster(long clusterId) {
return null;
}
+
public List listCluster() {
return clusterRepository.listCluster();
}
diff --git
a/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/dispatch/RunnerTaskDispatcher.java
b/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/dispatch/RunnerTaskDispatcher.java
index 605d38f..696265b 100644
---
a/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/dispatch/RunnerTaskDispatcher.java
+++
b/adapter/runtime-manager/src/main/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/dispatch/RunnerTaskDispatcher.java
@@ -17,16 +17,39 @@
package org.apache.rocketmq.eventbridge.adapter.runtime.manager.dispatch;
+import javax.annotation.Resource;
+import org.apache.rocketmq.eventbridge.adapter.runtime.manager.cluster.Cluster;
+import
org.apache.rocketmq.eventbridge.adapter.runtime.manager.cluster.ClusterSelectorService;
+import org.apache.rocketmq.eventbridge.adapter.runtime.manager.task.RunnerTask;
+import
org.apache.rocketmq.eventbridge.adapter.runtime.manager.task.RunnerTaskService;
import org.apache.rocketmq.eventbridge.adapter.runtime.manager.worker.Worker;
+import
org.apache.rocketmq.eventbridge.adapter.runtime.manager.worker.WorkerSelectorService;
public class RunnerTaskDispatcher {
-
(rocketmq-eventbridge) branch feat/runtime-manager updated: feat:add WorkerInstanceRepositoryOnK8STest test case.
This is an automated email from the ASF dual-hosted git repository.
shenlin pushed a commit to branch feat/runtime-manager
in repository https://gitbox.apache.org/repos/asf/rocketmq-eventbridge.git
The following commit(s) were added to refs/heads/feat/runtime-manager by this
push:
new d7df5cd feat:add WorkerInstanceRepositoryOnK8STest test case.
d7df5cd is described below
commit d7df5cde64e4c3426bf9b51ff97c203627473011
Author: 2011shenlin <2011shen...@gmail.com>
AuthorDate: Wed Jan 17 16:41:14 2024 +0800
feat:add WorkerInstanceRepositoryOnK8STest test case.
---
.../manager/k8s/repository/WorkerInstanceRepositoryOnK8STest.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/adapter/runtime-on-k8s/src/test/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/k8s/repository/WorkerInstanceRepositoryOnK8STest.java
b/adapter/runtime-on-k8s/src/test/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/k8s/repository/WorkerInstanceRepositoryOnK8STest.java
index d96438b..877799e 100644
---
a/adapter/runtime-on-k8s/src/test/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/k8s/repository/WorkerInstanceRepositoryOnK8STest.java
+++
b/adapter/runtime-on-k8s/src/test/java/org/apache/rocketmq/eventbridge/adapter/runtime/manager/k8s/repository/WorkerInstanceRepositoryOnK8STest.java
@@ -20,7 +20,7 @@ public class WorkerInstanceRepositoryOnK8STest {
Map environments = Maps.newHashMap();
environments.put("key1", "value1");
environments.put("key2", "value2");
-workerInstanceRepositoryOnK8S.applyWorkerInstance("worker-4",
"registry.cn-beijing.cr.aliyuncs.com/eventbridge:20231115195431f55971", new
Gson().fromJson("{\"cpu\":1,\"memory\":1}", WorkerResource.class), null);
+workerInstanceRepositoryOnK8S.applyWorkerInstance("worker-4",
"registry.cn-beijing.cr.aliyuncs.com/eventbridge:20231115195431f55971", new
Gson().fromJson("{\"cpu\":1,\"memory\":1}", WorkerResource.class),
environments);
}
@Test
[I] Check message size before message compression [rocketmq]
Aresxue opened a new issue, #7758: URL: https://github.com/apache/rocketmq/issues/7758 ### Before Creating the Bug Report - [X] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [X] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment All ### RocketMQ version All ### JDK Version All ### Describe the Bug The rocketmq client does both message size and message compression, but check message size before message compression.This makes it possible for a 5M message in the default configuration to be smaller than 4M by compression, but now you have to manually adjust the maxMessageSize parameter, which makes the semantics of the client and broker's check message size not match, please confirm this is a problem, I will submit a pr to fix it if necessary. ### Steps to Reproduce At will ### What Did You Expect to See? If the size of a 5M message is smaller than 4M, the message can pass the verification ### What Did You See Instead? A message that is 5M in size but smaller than 4M in size fails to pass the check ### Additional Context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [ISSUE #7478] On startup, if not check crc, not read message body. avoid startup too slow when data is large [rocketmq]
onejimmyboy commented on PR #7479: URL: https://github.com/apache/rocketmq/pull/7479#issuecomment-1895367207 如果不对消息内容文件的话有什么影响那？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] 这个库支持5.1.4版本的rocketmq么？ [rocketmq-client-cpp]
wedvefv closed issue #463: 这个库支持5.1.4版本的rocketmq么？ URL: https://github.com/apache/rocketmq-client-cpp/issues/463 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [ISSUE #7671] filter inner producer client when report to metrics [rocketmq]
joeCarf commented on code in PR #7673:
URL: https://github.com/apache/rocketmq/pull/7673#discussion_r144424
##
broker/src/main/java/org/apache/rocketmq/broker/metrics/BrokerMetricsManager.java:
##
@@ -459,9 +462,14 @@ private void initConnectionMetrics() {
.ofLongs()
.buildWithCallback(measurement -> {
Map metricsMap = new HashMap<>();
-brokerController.getProducerManager()
-.getGroupChannelTable()
-.values()
+ConcurrentHashMap> groupChannelTable =
brokerController.getProducerManager().getGroupChannelTable();
+ConcurrentHashMap> clone = new ConcurrentHashMap<>(groupChannelTable);
+clone.forEach((group, map) -> {
+if (MixAll.CLIENT_INNER_PRODUCER_GROUP.equals(group)) {
+groupChannelTable.remove(group);
Review Comment:
looks that you removed the keys in `groupChannelTable` but used clone map at
last
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] chore(golang): change log level of heartbeat message to debug [rocketmq-clients]
yangtuooc opened a new pull request, #670: URL: https://github.com/apache/rocketmq-clients/pull/670 ### Which Issue(s) This PR Fixes Fixes #issue_id ### Brief Description ### How Did You Test This Change? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] chore(golang): change log level of heartbeat message to debug [rocketmq-clients]
yangtuooc closed pull request #670: chore(golang): change log level of heartbeat message to debug URL: https://github.com/apache/rocketmq-clients/pull/670 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Does the method onChannelDestroy called in function scanNotActiveBroker unnecessary [rocketmq]
github-actions[bot] commented on issue #5894: URL: https://github.com/apache/rocketmq/issues/5894#issuecomment-1897522677 This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [ISSUE #5836]fix the method queryAssignment request twice [rocketmq]
github-actions[bot] closed pull request #5850: [ISSUE #5836]fix the method queryAssignment request twice URL: https://github.com/apache/rocketmq/pull/5850 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [ISSUE #5836]fix the method queryAssignment request twice [rocketmq]
github-actions[bot] commented on PR #5850: URL: https://github.com/apache/rocketmq/pull/5850#issuecomment-1897522715 This PR was closed because it has been inactive for 3 days since being marked as stale. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] chore(deps): bump golang.org/x/crypto from 0.0.0-20220722155217-630584e8d5aa to 0.17.0 in /golang [rocketmq-clients]
github-actions[bot] commented on PR #652: URL: https://github.com/apache/rocketmq-clients/pull/652#issuecomment-1897541865 This PR is stale because it has been open for 30 days with no activity. It will be closed in 3 days if no further activity occurs. If you wish not to mark it as stale, please leave a comment in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] timerSkipUnknownError can't find set method [rocketmq]
zyhui98 opened a new issue, #7760: URL: https://github.com/apache/rocketmq/issues/7760 ### Before Creating the Bug Report - [X] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [X] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment ignore ### RocketMQ version 5.14 ### JDK Version ignore ### Describe the Bug MessageStoreConfig can't set timerSkipUnknownError property. so I can't set timerSkipUnknownError false to true ### Steps to Reproduce normal start ### What Did You Expect to See? timerSkipUnknownError is true ### What Did You See Instead? timerSkipUnknownError is true ### Additional Context broker.conf can't modify timerSkipUnknownError -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Update 01quickstart.md [rocketmq-site]
wedvefv opened a new pull request, #614: URL: https://github.com/apache/rocketmq-site/pull/614 通过grpc客户端demo发送数据到rocketmq5.x需要配置grpc端口，注意client端代码要使用grpc的地址和端口作为access_point参数 This pull request aims to address the need for configuring gRPC ports when sending data to RocketMQ using the gRPC client demo. The PR highlights that clients must use the gRPC server's address and port as the access_point parameter in their code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Is rocketmq-client-cpp 2.0.0 support rocketmq5.x? [rocketmq-client-cpp]
wedvefv commented on issue #460: URL: https://github.com/apache/rocketmq-client-cpp/issues/460#issuecomment-1897850679 If rocker5.x is set up using proxy mode, clients can use the gRPC protocol for requests. However, in this case, the server needs to be configured with a gRPC port in **conf/rmq-proxy.json**, and the **access_point** should be written as [nameserverIp]:[GrpcPort], rather than nameserverIp:9876. If a non-gRPC client is used, I have tested that rocketmq5.1.4 can send and receive messages normally using rocketmq-client-cpp-2.2.0 (which uses non-gRPC protocol communication by default). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
yuz10 commented on code in PR #7599:
URL: https://github.com/apache/rocketmq/pull/7599#discussion_r1456981076
##
store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java:
##
@@ -343,14 +427,57 @@ private void recover(long maxPhyOffsetOfConsumeQueue)
throws RocksDBException {
log.info("Will set the initial commitlog offset={} for dledger",
dividedCommitlogOffset);
}
+private boolean isMmapFileMatchedRecover(final MmapFile mmapFile) {
+ByteBuffer byteBuffer = mmapFile.sliceByteBuffer();
+
+int magicCode = byteBuffer.getInt(DLedgerEntry.BODY_OFFSET +
MessageDecoder.MESSAGE_MAGIC_CODE_POSITION);
+if (magicCode != MESSAGE_MAGIC_CODE) {
+return false;
+}
+
+int storeTimestampPosition;
+int sysFlag = byteBuffer.getInt(MessageDecoder.SYSFLAG_POSITION);
Review Comment:
DLedgerEntry.BODY_OFFSET + MessageDecoder.SYSFLAG_POSITIO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GH] (rocketmq): Workflow run "Build and Run Tests by Bazel" failed!
The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git has failed. Run started by GitHub user bxfjb (triggered by yuz10). Head commit for run: d0402b0280bf2e7337c751e7be2ff5aba4c9e61b / zhaoyuhan fix correct storeTimestampPosition when bornhost is v6 Report URL: https://github.com/apache/rocketmq/actions/runs/7550736881 With regards, GitHub Actions via GitBox
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
bxfjb commented on code in PR #7599:
URL: https://github.com/apache/rocketmq/pull/7599#discussion_r1456989735
##
store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java:
##
@@ -343,14 +427,57 @@ private void recover(long maxPhyOffsetOfConsumeQueue)
throws RocksDBException {
log.info("Will set the initial commitlog offset={} for dledger",
dividedCommitlogOffset);
}
+private boolean isMmapFileMatchedRecover(final MmapFile mmapFile) {
+ByteBuffer byteBuffer = mmapFile.sliceByteBuffer();
+
+int magicCode = byteBuffer.getInt(DLedgerEntry.BODY_OFFSET +
MessageDecoder.MESSAGE_MAGIC_CODE_POSITION);
+if (magicCode != MESSAGE_MAGIC_CODE) {
+return false;
+}
+
+int storeTimestampPosition;
+int sysFlag = byteBuffer.getInt(MessageDecoder.SYSFLAG_POSITION);
Review Comment:
fixed and thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GH] (rocketmq): Workflow run "Build and Run Tests by Bazel" failed!
The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git has failed. Run started by GitHub user bxfjb (triggered by yuz10). Head commit for run: 72522adf08e95ef92f07a3254b943bdec311b670 / zhaoyuhan fix correct SYSFLAG offset Report URL: https://github.com/apache/rocketmq/actions/runs/7566375815 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Coverage" failed!
The GitHub Actions job "Coverage" on rocketmq.git has failed. Run started by GitHub user bxfjb (triggered by yuz10). Head commit for run: d0402b0280bf2e7337c751e7be2ff5aba4c9e61b / zhaoyuhan fix correct storeTimestampPosition when bornhost is v6 Report URL: https://github.com/apache/rocketmq/actions/runs/7550736884 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "E2E test for pull request" failed!
The GitHub Actions job "E2E test for pull request" on rocketmq.git has failed. Run started by GitHub user bxfjb (triggered by bxfjb). Head commit for run: 7a36d4d736ae8d6d92658e3bdb18f1cd5c0afdb0 / 道君 [ISSUE #7757] Use `CompositeByteBuf` to prevent memory copy. (#7694) * Use CompositeByteBuf to prevent mem_copy. * Fix code * Add tests * Remove useless UTs * Remove unused imports. - Co-authored-by: RongtongJin Report URL: https://github.com/apache/rocketmq/actions/runs/7566327040 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "E2E test for pull request" failed!
The GitHub Actions job "E2E test for pull request" on rocketmq.git has failed. Run started by GitHub user bxfjb (triggered by bxfjb). Head commit for run: 7a36d4d736ae8d6d92658e3bdb18f1cd5c0afdb0 / 道君 [ISSUE #7757] Use `CompositeByteBuf` to prevent memory copy. (#7694) * Use CompositeByteBuf to prevent mem_copy. * Fix code * Add tests * Remove useless UTs * Remove unused imports. - Co-authored-by: RongtongJin Report URL: https://github.com/apache/rocketmq/actions/runs/7566468967 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Coverage" is working again!
The GitHub Actions job "Coverage" on rocketmq.git has succeeded. Run started by GitHub user bxfjb (triggered by yuz10). Head commit for run: 72522adf08e95ef92f07a3254b943bdec311b670 / zhaoyuhan fix correct SYSFLAG offset Report URL: https://github.com/apache/rocketmq/actions/runs/7566375806 With regards, GitHub Actions via GitBox
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
codecov-commenter commented on PR #7599: URL: https://github.com/apache/rocketmq/pull/7599#issuecomment-1897937916 ## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: `27 lines` in your changes are missing coverage. Please review. > Comparison is base [(`65faea2`)](https://app.codecov.io/gh/apache/rocketmq/commit/65faea22fd54fd9875f2ca9d3088b4dc46d31cce?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 43.19% compared to head [(`72522ad`)](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 43.18%. > Report is 38 commits behind head on develop. | [Files](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...pache/rocketmq/store/dledger/DLedgerCommitLog.java](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL2RsZWRnZXIvRExlZGdlckNvbW1pdExvZy5qYXZh) | 67.46% | [16 Missing and 11 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop#7599 +/- ## = - Coverage 43.19% 43.18% -0.02% - Complexity 9782 9828 +46 = Files 1162 1167 +5 Lines 8437884690 +312 Branches 1095510993 +38 = + Hits 3645136572 +121 - Misses 4339743570 +173 - Partials4530 4548 +18 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/rocketmq/pull/7599?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(rocketmq) branch develop updated: [ISSUE #7355] fix dledger recover abnormally may lost consume queue of tail (#7599)
This is an automated email from the ASF dual-hosted git repository.
yuzhou pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
The following commit(s) were added to refs/heads/develop by this push:
new 646e2a4942 [ISSUE #7355] fix dledger recover abnormally may lost
consume queue of tail (#7599)
646e2a4942 is described below
commit 646e2a4942c62eb36b1601a41ebc0828e8580804
Author: bxfjb <48467309+bx...@users.noreply.github.com>
AuthorDate: Thu Jan 18 15:43:01 2024 +0800
[ISSUE #7355] fix dledger recover abnormally may lost consume queue of tail
(#7599)
* fix dledger recover abnormally may lost consume queue of tail
* fix correct storeTimestampPosition when bornhost is v6
* fix correct SYSFLAG offset
-
Co-authored-by: 赵宇晗
---
.../rocketmq/store/dledger/DLedgerCommitLog.java | 137 -
.../store/dledger/DLedgerCommitlogTest.java| 40 ++
2 files changed, 172 insertions(+), 5 deletions(-)
diff --git
a/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java
b/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java
index 70371d83b8..27a18abc9d 100644
---
a/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java
+++
b/store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java
@@ -290,9 +290,9 @@ public class DLedgerCommitLog extends CommitLog {
return false;
}
-private void recover(long maxPhyOffsetOfConsumeQueue) throws
RocksDBException {
+private void dledgerRecoverNormally(long maxPhyOffsetOfConsumeQueue)
throws RocksDBException {
dLedgerFileStore.load();
-if (dLedgerFileList.getMappedFiles().size() > 0) {
+if (!dLedgerFileList.getMappedFiles().isEmpty()) {
dLedgerFileStore.recover();
dividedCommitlogOffset =
dLedgerFileList.getFirstMappedFile().getFileFromOffset();
MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile();
@@ -309,9 +309,93 @@ public class DLedgerCommitLog extends CommitLog {
}
//Indicate that, it is the first time to load mixed commitlog, need to
recover the old commitlog
isInrecoveringOldCommitlog = true;
-//No need the abnormal recover
super.recoverNormally(maxPhyOffsetOfConsumeQueue);
isInrecoveringOldCommitlog = false;
+
+setRecoverPosition();
+
+}
+
+private void dledgerRecoverAbnormally(long maxPhyOffsetOfConsumeQueue)
throws RocksDBException {
+boolean checkCRCOnRecover =
this.defaultMessageStore.getMessageStoreConfig().isCheckCRCOnRecover();
+dLedgerFileStore.load();
+if (!dLedgerFileList.getMappedFiles().isEmpty()) {
+dLedgerFileStore.recover();
+dividedCommitlogOffset =
dLedgerFileList.getFirstMappedFile().getFileFromOffset();
+MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile();
+if (mappedFile != null) {
+disableDeleteDledger();
+}
+List mmapFiles = dLedgerFileList.getMappedFiles();
+int index = mmapFiles.size() - 1;
+MmapFile mmapFile = null;
+for (; index >= 0; index--) {
+mmapFile = mmapFiles.get(index);
+if (isMmapFileMatchedRecover(mmapFile)) {
+log.info("dledger recover from this mappFile " +
mmapFile.getFileName());
+break;
+}
+}
+
+if (index < 0) {
+index = 0;
+mmapFile = mmapFiles.get(index);
+}
+
+ByteBuffer byteBuffer = mmapFile.sliceByteBuffer();
+long processOffset = mmapFile.getFileFromOffset();
+long mmapFileOffset = 0;
+while (true) {
+DispatchRequest dispatchRequest =
this.checkMessageAndReturnSize(byteBuffer, checkCRCOnRecover, true);
+int size = dispatchRequest.getMsgSize();
+
+if (dispatchRequest.isSuccess()) {
+if (size > 0) {
+mmapFileOffset += size;
+if
(this.defaultMessageStore.getMessageStoreConfig().isDuplicationEnable()) {
+if (dispatchRequest.getCommitLogOffset() <
this.defaultMessageStore.getConfirmOffset()) {
+
this.defaultMessageStore.doDispatch(dispatchRequest);
+}
+} else {
+
this.defaultMessageStore.doDispatch(dispatchRequest);
+}
+} else if (size == 0) {
+index++;
+if (index >= mmapFiles.size()) {
+log.info("dledger recover physics file over, last
mapped file " + mmapFile.getFileName());
+brea
Re: [PR] fix dledger recover abnormally may lost consume queue of tail [rocketmq]
yuz10 merged PR #7599: URL: https://github.com/apache/rocketmq/pull/7599 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] ConsumeQueue building exception caused by host downtime [rocketmq]
yuz10 closed issue #7355: [Bug] ConsumeQueue building exception caused by host downtime URL: https://github.com/apache/rocketmq/issues/7355 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GH] (rocketmq): Workflow run "Build and Run Tests by Bazel" failed!
The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git has failed. Run started by GitHub user yuz10 (triggered by yuz10). Head commit for run: 646e2a4942c62eb36b1601a41ebc0828e8580804 / bxfjb <48467309+bx...@users.noreply.github.com> [ISSUE #7355] fix dledger recover abnormally may lost consume queue of tail (#7599) * fix dledger recover abnormally may lost consume queue of tail * fix correct storeTimestampPosition when bornhost is v6 * fix correct SYSFLAG offset - Co-authored-by: 赵宇晗 Report URL: https://github.com/apache/rocketmq/actions/runs/7566821411 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "PUSH-CI" failed!
The GitHub Actions job "PUSH-CI" on rocketmq.git has failed. Run started by GitHub user yuz10 (triggered by yuz10). Head commit for run: 646e2a4942c62eb36b1601a41ebc0828e8580804 / bxfjb <48467309+bx...@users.noreply.github.com> [ISSUE #7355] fix dledger recover abnormally may lost consume queue of tail (#7599) * fix dledger recover abnormally may lost consume queue of tail * fix correct storeTimestampPosition when bornhost is v6 * fix correct SYSFLAG offset - Co-authored-by: 赵宇晗 Report URL: https://github.com/apache/rocketmq/actions/runs/7566821414 With regards, GitHub Actions via GitBox
