github-actions[bot] commented on code in PR #69423:
URL: https://github.com/apache/doris/pull/69423#discussion_r4639182831


##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -2166,6 +2210,17 @@ private boolean commitTxnImpl(TCommitTxnRequest request) 
throws UserException {
 
     @Override
     public TLoadTxnRollbackResult loadTxnRollback(TLoadTxnRollbackRequest 
request) throws TException {
+        try (TxnRpcAdmissionController.Guard guard = 
TxnRpcAdmissionController.INSTANCE.tryEnter()) {
+            if (guard == null) {
+                TStatus status = new TStatus(TStatusCode.FE_SHUTTING_DOWN);
+                status.addToErrorMsgs("FE is shutting down; retry this RPC on 
the current master");
+                return new TLoadTxnRollbackResult().setStatus(status);
+            }
+            return loadTxnRollbackWithAdmission(request);
+        }
+    }
+
+    private TLoadTxnRollbackResult 
loadTxnRollbackWithAdmission(TLoadTxnRollbackRequest request) throws TException 
{
         String clientAddr = getClientAddrAsString();
         if (LOG.isDebugEnabled()) {
             LOG.debug("receive txn rollback request: {}, backend: {}",

Review Comment:
   [P1] Guard multi-table Stream Load planning during FE drain: 
streamLoadMultiTablePut() still calls generatePlan() without 
TxnRpcAdmissionController.tryEnter(), while the six adjacent transaction RPCs 
now use that fence. A request can start after stopAcceptingIfIdle() and remain 
uncounted, then be interrupted by thriftServerStarter.stop(), aborting 
routine-load planning. Wrap this method with the same admission guard and 
return FE_SHUTTING_DOWN when closed.



##########
be/src/io/fs/multi_table_pipe.cpp:
##########
@@ -246,6 +246,10 @@ Status MultiTablePipe::exec_plans(ExecEnv* exec_env,
             return Status::Aborted("Missing vital param: table_name");
         }
 
+        // TODO: Balance this reservation when setup fails before a callback 
exists

Review Comment:
   [P1] Balance the multi-table in-flight reservation on setup errors: 
exec_plans() increments _inflight_cnt before exec_plan_fragment(), but 
RETURN_IF_ERROR exits when FragmentMgr fails before creating a completion 
callback, so the callback decrement never runs and _handle_consumer_finished() 
never fulfills load_status_promise. A routine-load multi-table request can wait 
indefinitely after a query-context setup failure. Decrement/cancel this 
reservation (and propagate the first error) before returning.



##########
fe/fe-core/src/main/java/org/apache/doris/DorisFE.java:
##########
@@ -741,19 +749,86 @@ public static boolean isServerReady() {
     }
 
     private static void gracefulShutdown() {
-        // wait for all queries to finish
+        // Health is already 503, but keep serving during LB / DNS convergence.
         try {
-            long now = System.currentTimeMillis();
-            List<Coordinator> allCoordinators = 
QeProcessorImpl.INSTANCE.getAllCoordinators();
-            while (!allCoordinators.isEmpty() && System.currentTimeMillis() - 
now < 300 * 1000L) {
-                Thread.sleep(1000);
-                allCoordinators = 
QeProcessorImpl.INSTANCE.getAllCoordinators();
-                LOG.info("waiting {} queries to finish before shutdown", 
allCoordinators.size());
-            }
-        } catch (Throwable t) {
-            LOG.error("", t);
+            LOG.info("graceful shutdown: serving during drain for {} seconds", 
Config.graceful_shutdown_drain_seconds);
+            TimeUnit.SECONDS.sleep(Math.max(0, 
Config.graceful_shutdown_drain_seconds));
+        } catch (InterruptedException e) {
+            Thread.currentThread().interrupt();
+            LOG.warn("graceful shutdown drain interrupted", e);
         }
 
+        // Close admission before the listener: an accepted connection can 
race its closure.
+        QueryAdmissionController.INSTANCE.stopAccepting();
+        if (qeService != null) {
+            qeService.stopAcceptNewConnections();
+        }
+        try {

Review Comment:
   [P1] Count HTTP Stream Load work in the FE drain: waitForGracefulShutdown() 
has no HTTP admission counter, while LoadAction's upload/planning handlers 
remain active until httpServer.shutdown() runs after the wait. An accepted 
upload can therefore be invisible to the zero check and be cut off with its 
transaction/body response when shutdown closes Jetty. Add an HTTP 
admission/active-request fence covering body and response lifetime, or stop 
HTTP admission before draining and wait for existing handlers.



##########
fe/fe-core/src/main/java/org/apache/doris/DorisFE.java:
##########
@@ -741,19 +749,86 @@ public static boolean isServerReady() {
     }
 
     private static void gracefulShutdown() {
-        // wait for all queries to finish
+        // Health is already 503, but keep serving during LB / DNS convergence.
         try {
-            long now = System.currentTimeMillis();
-            List<Coordinator> allCoordinators = 
QeProcessorImpl.INSTANCE.getAllCoordinators();
-            while (!allCoordinators.isEmpty() && System.currentTimeMillis() - 
now < 300 * 1000L) {
-                Thread.sleep(1000);
-                allCoordinators = 
QeProcessorImpl.INSTANCE.getAllCoordinators();
-                LOG.info("waiting {} queries to finish before shutdown", 
allCoordinators.size());
-            }
-        } catch (Throwable t) {
-            LOG.error("", t);
+            LOG.info("graceful shutdown: serving during drain for {} seconds", 
Config.graceful_shutdown_drain_seconds);
+            TimeUnit.SECONDS.sleep(Math.max(0, 
Config.graceful_shutdown_drain_seconds));
+        } catch (InterruptedException e) {
+            Thread.currentThread().interrupt();
+            LOG.warn("graceful shutdown drain interrupted", e);
         }
 
+        // Close admission before the listener: an accepted connection can 
race its closure.
+        QueryAdmissionController.INSTANCE.stopAccepting();
+        if (qeService != null) {
+            qeService.stopAcceptNewConnections();
+        }
+        try {
+            long deadline = System.nanoTime()
+                    + TimeUnit.SECONDS.toNanos(Math.max(0, 
Config.grace_shutdown_wait_seconds));
+            waitForGracefulShutdown(ExecuteEnv.getInstance().getScheduler(), 
deadline);
+        } catch (InterruptedException e) {
+            Thread.currentThread().interrupt();
+            LOG.warn("graceful shutdown wait interrupted", e);
+        } catch (Throwable t) {
+            LOG.error("graceful shutdown failed while draining requests", t);
+        } finally {
+            TxnRpcAdmissionController.INSTANCE.stopAccepting();
+        }
         LOG.info("graceful shutdown finished");
     }
+
+    static void waitForGracefulShutdown(ConnectScheduler scheduler, long 
deadline) throws InterruptedException {
+        // XNIO invokes close listeners synchronously, including transaction 
rollback RPCs.
+        // Keep those waits on a daemon worker so even an idle session cannot 
stall the deadline.
+        ExecutorService idleCloser = 
ThreadPoolManager.newDaemonFixedThreadPool(
+                1, 1, "graceful-idle-mysql-closer", false);
+        try {
+            Future<?> idleCleanup = null;
+            boolean cleanupStartedIdle = false;
+            while (true) {
+                if (idleCleanup == null) {
+                    cleanupStartedIdle = 
QueryAdmissionController.INSTANCE.getActiveRequests() == 0;
+                    idleCleanup = idleCloser.submit(() -> 
scheduler.closeIdleMysqlConnections(deadline));
+                }
+                int activeRequests = 
QueryAdmissionController.INSTANCE.getActiveRequests();
+                int activeTxnRpcs = 
TxnRpcAdmissionController.INSTANCE.getActiveRequests();
+                List<Coordinator> coordinators = 
QeProcessorImpl.INSTANCE.getAllCoordinators();
+                boolean pendingFlightResults = 
scheduler.hasPendingFlightResults();
+                boolean pendingIdleCleanup = !idleCleanup.isDone();

Review Comment:
   [P2] Fence Flight continuations before declaring FE drained: the final zero 
check samples QueryAdmissionController and then breaks, but 
runContinuation()/closeSession() still call acquireContinuation() after 
stopAccepting and can race into the gap. The Flight listener is stopped 
immediately afterward, so a late DoGet or close-session cleanup can be cut off. 
Atomically close continuation admission with the final check and wait for 
holders before stopping Flight.



##########
fe/fe-core/src/main/java/org/apache/doris/qe/QeService.java:
##########
@@ -66,6 +66,16 @@ public void start() throws Exception {
         LOG.info("QE service start.");
     }
 
+    public void stopAcceptNewConnections() {

Review Comment:
   [P2] Fence new Flight session creation during drain: 
stopAcceptNewConnections() stops only mysqlStarter, while Flight authentication 
still opens/registers ConnectContext sessions without QueryAdmissionController. 
Clients can create idle bearer sessions after command admission closes, consume 
FE/Flight connection quotas, then have commands rejected; idle cleanup excludes 
ARROW_FLIGHT_SQL. Gate initial Flight authentication or stop only new-session 
auth while retaining already-issued continuations.



##########
be/src/runtime/exec_env.cpp:
##########
@@ -205,20 +208,38 @@ std::map<TNetworkAddress, FrontendInfo> 
ExecEnv::get_running_frontends() {
     return res;
 }
 
+void ExecEnv::wait_for_shutdown_heartbeat() {
+    int32_t elapsed = 0;
+    while (!k_shutdown_heartbeat_sent && elapsed < 
config::grace_shutdown_heartbeat_wait_seconds) {
+        sleep(1);
+        ++elapsed;
+    }
+    LOG(INFO) << "shutdown heartbeat prepared: " << 
k_shutdown_heartbeat_sent.load() << ", waited "
+              << elapsed << " seconds";
+    // The response may still be in transit. This buffer also gives followers 
time
+    // to replay shutdown state. It does not prove that every FE has applied 
it.
+    sleep(std::max(0, config::grace_shutdown_heartbeat_propagation_seconds));
+}
+
 void ExecEnv::wait_for_all_tasks_done() {
-    // For graceful shutdown, need to wait for all running queries to stop
+    // The cloud load receiver and coordinator must finish. For example,
+    // an HTTP upload or transaction commit can outlive all of its pipeline 
fragments.
     int32_t wait_seconds_passed = 0;
     while (true) {
         int num_queries = _fragment_mgr->running_query_num();
-        if (num_queries < 1) {
+        size_t num_channels = _load_channel_mgr->num_active_channels();

Review Comment:
   [P1] Fence new HTTP loads before the BE shutdown census: 
wait_for_all_tasks_done() breaks on a zero sample, but the HTTP starter remains 
live and Stream Load handlers do not check k_doris_exit or any admission gate. 
A request arriving just after that sample creates a StreamLoadContext and can 
be terminated by the subsequent BE exit/HTTP stop, losing an accepted load. 
Close HTTP admission atomically before the final census (and count in-flight 
body/response work), or recheck/fence context creation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to