Doris-Breakwater commented on issue #70721: URL: https://github.com/apache/doris/issues/70721#issuecomment-6975119819
Breakwater-GitHub-Analysis-Slot: slot_a165d07036bf The reported controller-discovery defect is supported by the source. I would triage this as a high-impact BE memory-detection bug affecting cgroup v2 deployments where workload-group CPU initialization moves the BE below the cgroup exposing its memory interfaces. The kernel memory limit remains enforced; Doris loses visibility of it. The issue currently has no labels. **Verified from source** I checked the local `4.1.4` tag (`ad35a140c7fd`) and the cited master commit `13cff7ff484596f29aa1381750e906db3c8bb704`. Their `cgroup_memory_ctl.cpp`, `cgroup_util.cpp`, and `mem_info.cpp` are identical; the relevant CPU initialization behavior is also unchanged. - [`CgroupCpuCtl::init_cgroup_v2_query_path_public_file`](https://github.com/apache/doris/blob/13cff7ff484596f29aa1381750e906db3c8bb704/be/src/agent/cgroup_cpu_ctl.cpp#L129-L155) enables `+cpu` and writes `getpid()` into `query/cgroup.procs`. It does not propagate the memory controller. Whether `memory` is absent depends on the existing hierarchy; enabling CPU control alone does not prove its absence. - [`cgroupsv2_memory_controller_enabled` and `get_cgroups_path`](https://github.com/apache/doris/blob/13cff7ff484596f29aa1381750e906db3c8bb704/be/src/common/cgroup_memory_ctl.cpp#L34-L171) check only the process's current `cgroup.controllers`. If it lacks `memory`, the v2 path is rejected before the existing ancestor lookup runs. With no v1 memory membership, reader creation fails. - [`CGroupUtil::get_cgroupsv2_path`](https://github.com/apache/doris/blob/13cff7ff484596f29aa1381750e906db3c8bb704/be/src/util/cgroup_util.cpp#L197-L219) already searches ancestors, including the visible mount root. The premature controller gate prevents that behavior in the reported layout. - [`MemInfo::refresh_proc_meminfo`](https://github.com/apache/doris/blob/13cff7ff484596f29aa1381750e906db3c8bb704/be/src/util/mem_info.cpp#L97-L206) explicitly resets the cgroup limit to `INT64_MAX` after lookup failure and schedules a retry after approximately 300 seconds. Without a successful cgroup refresh, both physical-memory sizing and available-memory reporting fall back to `/proc/meminfo`. Percentage `mem_limit` is recalculated from that physical memory, and the soft limit is derived from it. This explains the supplied host-sized thresholds. A successful limit read alone is insufficient: usage refresh must succeed too. The [kernel cgroup v2 documentation](https://docs.kernel.org/admin-guide/cgroup-v2.html#threads) confirms that threaded subtrees support only threaded controllers, excluding `memory`; their domain resource consumption belongs to the threaded domain. Thus, absence of `memory` at the CPU leaf does not establish absence of ancestor memory enforcement. **Suggested fix and validation** Please open the proposed PR. Resolving an ancestor directory with the memory controller and the required memory interfaces, then using that directory consistently for limit and usage, addresses the demonstrated discovery failure. Keep the search bounded by the visible cgroup mount, including private-namespace membership `0::/`, and preserve read-error diagnostics and v1 fallback. One scope caveat: the nearest directory with memory interfaces is not necessarily the tightest enforcing ancestor. A selected child can have `memory.max=max` or a larger limit while a higher ancestor is capped. The current reader also stops at the nearest interface, so this is an existing limitation, not evidence against the proposed fix. Test/document that case separately; simply mixing an ancestor limit with a child's usage would misrepresent shared ancestor pressure. Requested regression coverage: - CPU-only descendants, both threaded and domain layouts, resolving the ancestor's limit **and** usage. - Existing leaf memory-controller behavior, private namespace root `0::/`, and mount-boundary termination. - Unlimited `memory.max`, absent/unreadable interfaces, and v1 behavior. - An integration check after CPU initialization and subsequent memory refresh: with an 8 GiB container and default `mem_limit=90%`, `soft_mem_limit_frac=0.9`, expect approximately 8 GiB physical memory, 7.2 GiB process limit, and 6.48 GiB soft limit when the selected memory cgroup is the capped container cgroup. **Evidence still needed for the observed OOM** The code path is verified; I have not run a BE/container reproduction or the proposed patch's tests. Please attach: - Kernel/container-runtime versions, cgroup namespace mode, `/proc/<BE_PID>/cgroup`, and relevant cgroup2 mount entries from the BE's mount namespace. - From the BE-visible leaf through the container ancestor: `cgroup.type`, `cgroup.controllers`, `cgroup.subtree_control`, `memory.max`, `memory.current`, and `memory.stat`, noting missing files. - Effective `doris_cgroup_cpu_path`, `enable_use_cgroup_memory_info`, `mem_limit`, and `soft_mem_limit_frac`; timestamped CPU initialization, `Refresh cgroup memory limit failed`, and memory-summary logs. - Kernel OOM victim records and changes in `memory.events` (`max`, `oom`, `oom_kill`) at the enforcing cgroup, to distinguish this sizing defect from other causes of termination. Pending a fix, an absolute `mem_limit` below the container's enforced limit, with headroom for other charged memory, can restore lower process thresholds after a BE restart. The checked code supports absolute units; this mitigation does not restore cgroup usage visibility or guarantee protection from kernel OOM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
