chenBright commented on code in PR #3750:
URL: https://github.com/apache/brpc/pull/3750#discussion_r4623857914
##########
src/brpc/rdma/rdma_endpoint.cpp:
##########
@@ -2038,9 +2060,13 @@ void RdmaEndpoint::PollingModeRelease(bthread_tag_t tag)
{
auto& pollers = group.pollers;
auto& running = group.running;
running.store(false, std::memory_order_relaxed);
- for (int i = 0; i < FLAGS_rdma_poller_num; ++i) {
- bthread_join(pollers[i].tid, nullptr);
+ for (auto& poller : pollers) {
+ if (poller.tid != INVALID_BTHREAD) {
+ bthread_join(poller.tid, nullptr);
+ poller.tid = INVALID_BTHREAD;
+ }
}
+ group.initialized.store(false, std::memory_order_release);
Review Comment:
If A pauses just after setting `initialized` to `true`, B can skip all
invalid `tid` values here and clear `initialized`. When A resumes, it will
still start the entire group of pollers, allowing the next initialization to
acquire the group again and overwrite the active threads' `tid` values. Please
use the same per-tag lifecycle protection for release, initialization, and
rollback, and add a case that triggers release before the first poller is
created.
##########
src/brpc/rdma/rdma_endpoint.cpp:
##########
@@ -2012,20 +2019,35 @@ int RdmaEndpoint::PollingModeInitialize(bthread_tag_t
tag,
return nullptr;
};
- for (int i = 0; i < FLAGS_rdma_poller_num; ++i) {
- auto args = new FnArgs{&pollers[i], &running};
+ for (size_t i = 0; i < pollers.size(); ++i) {
+ std::unique_ptr<FnArgs> args(new FnArgs{&pollers[i], &running});
auto attr = FLAGS_rdma_disable_bthread ? BTHREAD_ATTR_PTHREAD
: BTHREAD_ATTR_NORMAL;
attr.tag = tag;
bthread_attr_set_name(&attr, "RdmaPolling");
pollers[i].callback = callback;
pollers[i].init_fn = init_fn;
pollers[i].release_fn = release_fn;
- auto rc = bthread_start_background(&pollers[i].tid, &attr, fn, args);
+#if defined(UNIT_TEST)
+ const int rc = g_start_rdma_poller_for_test(
+#else
+ const int rc = bthread_start_background(
+#endif
+ &pollers[i].tid, &attr, fn, args.get());
if (rc != 0) {
- LOG(ERROR) << "Fail to start rdma polling bthread";
+ running.store(false, std::memory_order_relaxed);
+ for (size_t j = 0; j < i; ++j) {
+ bthread_join(pollers[j].tid, nullptr);
+ pollers[j].tid = INVALID_BTHREAD;
Review Comment:
If the `init_fn` callbacks of two pollers need to wait for each other to
become ready, and the second poller fails to be created after the first enters
its callback, the `join` here will wait forever for the first poller to exit.
`running=false` cannot interrupt the callback because the poller reads it only
after `init_fn` returns. One option is to have new threads wait for the result
of the entire batch creation, then execute the callbacks on success or exit
directly on failure.
##########
src/brpc/rdma/rdma_endpoint.cpp:
##########
@@ -1965,9 +1971,10 @@ int RdmaEndpoint::PollingModeInitialize(bthread_tag_t
tag,
auto& pollers = group.pollers;
auto& running = group.running;
bool expected = false;
- if (!running.compare_exchange_strong(expected, true)) {
+ if (!group.initialized.compare_exchange_strong(expected, true)) {
return 0;
Review Comment:
When A fails to create the second poller and waits for the first one to
exit, `initialized` is still `true`. At that point, B with the same tag will
return `0` here, but A will subsequently clear the entire group. B's `Channel`
can pass initialization, yet no poller will handle its registered CQ. Please
return `0` only for groups that have started successfully; during creation or
rollback, wait for the result or return a retryable error. The current
assertion that `initialize_during_rollback` returns `0` also needs to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]