github-actions[bot] commented on code in PR #69421:
URL: https://github.com/apache/doris/pull/69421#discussion_r4639054453


##########
be/src/io/fs/multi_table_pipe.cpp:
##########
@@ -246,6 +246,10 @@ Status MultiTablePipe::exec_plans(ExecEnv* exec_env,
             return Status::Aborted("Missing vital param: table_name");
         }
 
+        // TODO: Balance this reservation when setup fails before a callback 
exists

Review Comment:
   [P1] Balance the in-flight reservation on plan setup failure
   
   The increment immediately before exec_plan_fragment is not paired when setup 
returns an error before the callback is installed. The caller then decrements 
only the initial consume reservation, leaving _inflight_cnt nonzero; 
_handle_consumer_finished() is never reached and load_status_future.get() can 
wait forever for a multi-table routine load. Use a scope guard or equivalent 
failure path to decrement this plan and remove/cancel its child context before 
returning.



##########
be/src/load/group_commit/group_commit_mgr.cpp:
##########
@@ -505,13 +505,13 @@ Status GroupCommitTable::_create_group_commit_load(
         } else {
             LOG(WARNING) << "_exec_env->cluster_info not set backend_id";
         }
-        TNetworkAddress master_addr = 
_exec_env->cluster_info()->master_fe_addr;
-        st = ThriftRpcHelper::rpc<FrontendServiceClient>(
-                master_addr.hostname, master_addr.port,
+        st = ThriftRpcHelper::rpc_fe_with_master_refresh(

Review Comment:
   [P1] Keep group-commit completion on the elected master
   
   Only the plan request is routed through rpc_fe_with_master_refresh; 
_finish_group_commit_load still sends commit and rollback to the snapshotted 
master_addr with the old one-shot helper. A master election after the plan 
succeeds can therefore reject or disconnect these completion RPCs without 
refreshing the address, leaving the admitted transaction unresolved. Route both 
completion phases through the refresh helper with the stable transaction id.



##########
be/src/util/thrift_rpc_helper.cpp:
##########
@@ -23,14 +23,17 @@
 #include <thrift/protocol/TBinaryProtocol.h>
 #include <thrift/transport/TTransportException.h>
 // IWYU pragma: no_include <bits/chrono.h>
+#include <algorithm>
 #include <chrono> // IWYU pragma: keep
 #include <sstream>
 #include <thread>
 
 #include "common/status.h"
+#include "runtime/cluster_info.h"
 #include "runtime/exec_env.h" // IWYU pragma: keep

Review Comment:
   [P1] Enable master-refresh retries when BE shutdown starts
   
    rpc_fe_with_master_refresh selects the long retry/DNS-refresh path from 
cluster_info()->in_graceful_shutdown(), but the production shutdown path only 
sets k_doris_exit and never calls set_in_graceful_shutdown(true). The flag 
therefore stays false for every real SIGTERM, so a transport failure while the 
FE is being stopped returns from the legacy RPC immediately and the load can 
fail during the exact election window this retry budget is intended to cover. 
Set the flag as part of the shutdown transition (or derive this mode from 
k_doris_exit).



##########
be/src/util/thrift_rpc_helper.cpp:
##########
@@ -23,14 +23,17 @@
 #include <thrift/protocol/TBinaryProtocol.h>
 #include <thrift/transport/TTransportException.h>
 // IWYU pragma: no_include <bits/chrono.h>
+#include <algorithm>
 #include <chrono> // IWYU pragma: keep
 #include <sstream>
 #include <thread>
 
 #include "common/status.h"
+#include "runtime/cluster_info.h"
 #include "runtime/exec_env.h" // IWYU pragma: keep
 #include "util/client_cache.h"
 #include "util/debug_points.h"
+#include "util/dns_cache.h"
 #include "util/network_util.h"
 

Review Comment:
   [P1] Retry healthy NOT_MASTER responses during FE election
   
   When the old master is still reachable it can return NOT_MASTER during an 
election. Because this branch only falls through for FE_SHUTTING_DOWN, the 
helper returns transport OK with the NOT_MASTER result and the Stream Load 
caller aborts; _retry_fe_rpc's address refresh is never used. Route NOT_MASTER 
through the same bounded retry loop so a heartbeat with the new master can 
recover the request.



##########
fe/fe-core/src/main/java/org/apache/doris/DorisFE.java:
##########
@@ -76,6 +79,8 @@
 import java.time.LocalDate;
 import java.util.List;
 import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicBoolean;

Review Comment:
   [P1] Fence HTTP and ordinary Thrift admission before declaring quiescence
   
   This fence stops only the MySQL listener and the two explicit counters. HTTP 
and the general FE Thrift listener stay open until after 
waitForGracefulShutdown, while their planning/stream-load handlers are not 
counted. A request accepted after this point can run with no active-request 
counter, transaction-RPC counter, or Coordinator, so the loop can exit and the 
subsequent listener shutdown can cut off its response or load. Stop/fence those 
listeners or account for every accepted handler before the final zero check.



##########
fe/fe-core/src/main/java/org/apache/doris/qe/QueryAdmissionController.java:
##########
@@ -0,0 +1,56 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.base.Preconditions;
+
+/** Admission and accounting for client requests during FE graceful shutdown. 
*/
+public class QueryAdmissionController {
+    public static final QueryAdmissionController INSTANCE = new 
QueryAdmissionController();
+
+    private boolean accepting = true;
+    private int activeRequests;
+
+    // Admission and the shutdown transition share a lock. A request either 
contributes
+    // to the drain count before shutdown, or is refused before it starts 
planning.
+    public synchronized boolean tryAcquire() {
+        if (!accepting) {
+            return false;
+        }
+        activeRequests++;
+        return true;
+    }
+
+    public synchronized void release() {
+        Preconditions.checkState(activeRequests > 0, "no admitted request to 
release");
+        activeRequests--;
+    }
+
+    // Result retrieval and session cleanup must remain possible after 
admission closes.

Review Comment:
   [P1] Fence Flight continuations with the final drain snapshot
   
   acquireContinuation() still increments after stopAccepting(), and the drain 
loop only performs a non-atomic final read of activeRequests before breaking. A 
DoGet/CloseSession can arrive in that gap, perform teardown or stream a ticket, 
and then be cut off when qeService.stop() closes Flight. Coordinate 
continuation admission with the final fence (or close/fence Flight before the 
zero check) so no continuation can begin after quiescence is declared.



##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -1458,6 +1458,17 @@ private void checkPassword(String user, String passwd, 
String clientIp)
 
     @Override
     public TLoadTxnBeginResult loadTxnBegin(TLoadTxnBeginRequest request) 
throws TException {
+        try (TxnRpcAdmissionController.Guard guard = 
TxnRpcAdmissionController.INSTANCE.tryEnter()) {

Review Comment:
   [P1] Admit multi-table planning into the shutdown fence
   
   streamLoadMultiTablePut is the endpoint used by 
MultiTablePipe::request_and_exec_plans, but only streamLoadPut and the 
transaction commit/rollback methods use TxnRpcAdmissionController. Once 
stopAcceptingIfIdle() observes zero guarded RPCs, this uncounted planner can 
still start, update routine-load state, and lose its response when the Thrift 
server stops. Wrap it in the same guard and return FE_SHUTTING_DOWN before any 
plan work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to