Re: [systemd-devel] Question: systemd-resolved.service delayed almost 90s start after systemd-sysctl.service
Hi, Paul Thank you for your reply. This is a low-probability, sporadic problem. Currently, there is no `journalctl -b` information with `debug` on the command line. The unit information of the two services is as follows: 1. systemctl cat systemd-resolved # /usr/lib/systemd/system/systemd-resolved.service # SPDX-License-Identifier: LGPL-2.1-or-later # # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. [Unit] Description=Network Name Resolution Documentation=man:systemd-resolved.service(8) Documentation=man:org.freedesktop.resolve1(5) Documentation= https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers Documentation= https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients DefaultDependencies=no After=systemd-sysctl.service systemd-sysusers.service Before=sysinit.target network.target nss-lookup.target shutdown.target initrd-switch-root.target Conflicts=shutdown.target initrd-switch-root.target Wants=nss-lookup.target [Service] AmbientCapabilities=CAP_SETPCAP CAP_NET_RAW CAP_NET_BIND_SERVICE BusName=org.freedesktop.resolve1 CapabilityBoundingSet=CAP_SETPCAP CAP_NET_RAW CAP_NET_BIND_SERVICE ExecStart=!!/usr/lib/systemd/systemd-resolved LockPersonality=yes MemoryDenyWriteExecute=yes NoNewPrivileges=yes PrivateDevices=yes PrivateTmp=yes ProtectClock=yes ProtectControlGroups=yes ProtectHome=yes ProtectKernelLogs=yes ProtectKernelModules=yes ProtectKernelTunables=yes ProtectSystem=strict Restart=always RestartSec=0 RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6 RestrictNamespaces=yes RestrictRealtime=yes RestrictSUIDSGID=yes RuntimeDirectory=systemd/resolve RuntimeDirectoryPreserve=yes SystemCallArchitectures=native SystemCallErrorNumber=EPERM SystemCallFilter=@system-service Type=notify User=systemd-resolve ImportCredential=network.dns ImportCredential=network.search_domains WatchdogSec=3min [Install] WantedBy=sysinit.target Alias=dbus-org.freedesktop.resolve1.service 2. systemctl cat systemd-sysctl.service # /usr/lib/systemd/system/systemd-sysctl.service # SPDX-License-Identifier: LGPL-2.1-or-later # # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. [Unit] Description=Apply Kernel Variables Documentation=man:systemd-sysctl.service(8) man:sysctl.d(5) DefaultDependencies=no Conflicts=shutdown.target After=systemd-modules-load.service Before=sysinit.target shutdown.target ConditionPathIsReadWrite=/proc/sys/net/ [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/lib/systemd/systemd-sysctl TimeoutSec=90s ImportCredential=sysctl.* Best regards, Alien Kong
[systemd-devel] Question: systemd-binfmt.service delayed almost 90s start
https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems ConditionPathExists=/proc/sys/fs/binfmt_misc/ ConditionPathIsReadWrite=/proc/sys/ DefaultDependencies=no Before=sysinit.target Conflicts=shutdown.target Before=shutdown.target [Automount] Where=/proc/sys/fs/binfmt_misc 2. /usr/lib/systemd/system/proc-sys-fs-binfmt_misc.mount [Unit] Description=Arbitrary Executable File Formats File System Documentation=https://docs.kernel.org/admin-guide/binfmt-misc.html Documentation= https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems DefaultDependencies=no [Mount] What=binfmt_misc Where=/proc/sys/fs/binfmt_misc Type=binfmt_misc Options=nosuid,nodev,noexec [Install] WantedBy=sysinit.target 3. /usr/lib/systemd/system/systemd-binfmt.service [Unit] Description=Set Up Additional Binary Formats Documentation=man:systemd-binfmt.service(8) man:binfmt.d(5) Documentation=https://docs.kernel.org/admin-guide/binfmt-misc.html Documentation= https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems DefaultDependencies=no Conflicts=shutdown.target After=proc-sys-fs-binfmt_misc.automount After=proc-sys-fs-binfmt_misc.mount After=local-fs.target Before=sysinit.target shutdown.target ConditionPathIsMountPoint=/proc/sys/fs/binfmt_misc ConditionDirectoryNotEmpty=|/lib/binfmt.d ConditionDirectoryNotEmpty=|/usr/lib/binfmt.d ConditionDirectoryNotEmpty=|/usr/local/lib/binfmt.d ConditionDirectoryNotEmpty=|/etc/binfmt.d ConditionDirectoryNotEmpty=|/run/binfmt.d [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/lib/systemd/systemd-binfmt ExecStop=/usr/lib/systemd/systemd-binfmt --unregister TimeoutSec=90s Any insight would be appreciated.Thanks~ Best regards, Alien Kong
Re: [systemd-devel] Question: systemd-resolved.service delayed almost 90s start after systemd-sysctl.service
-tmpfiles-setup.service [0m. [2025-05-09 10:38:33.165964] [ [0;32m OK [0m] Found device [0;1;39mdev-ttyUTC0.device [0m. [2025-05-09 10:38:33.229922] Mounting [0;1;39mrun-rpc_pipefs.mount [0m... [2025-05-09 10:38:33.246783] Starting [0;1;39mrpcbind.service [0m... [2025-05-09 10:38:33.247943] Starting [0;1;39msystemd-sysctl.service [0m... [2025-05-09 10:38:33.249473] Starting [0;1;39msystemd-update-utmp.service [0m... [2025-05-09 10:38:33.262237] [ [0;32m OK [0m] Mounted [0;1;39mrun-rpc_pipefs.mount [0m. [2025-05-09 10:38:33.263251] [ [0;32m OK [0m] Found device [0;1;39mdev-tegra_hv_pm_ctl.device [0m. [2025-05-09 10:38:33.265105] [ [0;32m OK [0m] Started [0;1;39mrpcbind.service [0m. [2025-05-09 10:38:33.278968] [ [0;32m OK [0m] Reached target [0;1;39mrpc_pipefs.target [0m. [2025-05-09 10:38:33.280378] [ [0;32m OK [0m] Reached target [0;1;39mrpcbind.target [0m. [2025-05-09 10:38:33.293863] [ [0;32m OK [0m] Reached target [0;1;39mnfs-client.target [0m. [2025-05-09 10:38:33.294915] [ [0;32m OK [0m] Reached target [0;1;39mremote-fs-pre.target [0m. [2025-05-09 10:38:33.296171] [ [0;32m OK [0m] Reached target [0;1;39mremote-fs.target [0m. [2025-05-09 10:38:33.310669] [ [0;32m OK [0m] Finished [0;1;39msystemd-update-utmp.service [0m. [2025-05-09 10:38:33.325848] [ [0;32m OK [0m] Finished [0;1;39msystemd-sysctl.service [0m. // Here is the issue. [2025-05-09 10:40:02.693116] Starting [0;1;39msystemd-resolved.service [0m... [2025-05-09 10:40:03.093112] [ [0;32m OK [0m] Started [0;1;39msystemd-resolved.service [0m. [2025-05-09 10:40:03.094834] [ [0;32m OK [0m] Reached target [0;1;39mnss-lookup.target [0m. [2025-05-09 10:40:03.110386] [ [0;32m OK [0m] Reached target [0;1;39msysinit.target [0m. [2025-05-09 10:40:03.111699] [ [0;32m OK [0m] Started [0;1;39mdpkg-db-backup.timer [0m. [2025-05-09 10:40:03.125272] [ [0;32m OK [0m] Started [0;1;39me2scrub_all.timer [0m. [2025-05-09 10:40:03.126731] [ [0;32m OK [0m] Started [0;1;39mfstrim.timer [0m. [2025-05-09 10:40:03.127872] [ [0;32m OK [0m] Started [0;1;39mlogrotate.timer [0m. [2025-05-09 10:40:03.141129] [ [0;32m OK [0m] Started [0;1;39mmotd-news.timer [0m. [2025-05-09 10:40:03.142378] [ [0;32m OK [0m] Started [0;1;39msystemd-tmpfiles-clean.timer [0m. [2025-05-09 10:40:03.144127] [ [0;32m OK [0m] Started [0;1;39mxm_logrotate.timer [0m. [2025-05-09 10:40:03.157836] [ [0;32m OK [0m] Reached target [0;1;39mtimers.target [0m. [2025-05-09 10:40:03.159283] [ [0;32m OK [0m] Listening on [0;1;39mdbus.socket [0m. [2025-05-09 10:40:03.160640] [ [0;32m OK [0m] Listening on [0;1;39mssh.socket [0m. [2025-05-09 10:40:03.173272] [ [0;32m OK [0m] Reached target [0;1;39msockets.target [0m. [2025-05-09 10:40:03.174836] [ [0;32m OK [0m] Reached target [0;1;39mbasic.target [0m. [2025-05-09 10:40:03.176052] Starting [0;1;39mdbus.service [0m... [2025-05-09 10:40:03.189270] Starting [0;1;39mdisable_power_features_linux.service [0m... [2025-05-09 10:40:03.190659] [ [0;32m OK [0m] Started [0;1;39mdmesg.service [0m. [2025-05-09 10:40:03.191302] Starting [0;1;39me2scrub_reap.service [0m... Best regards, Alien Kong Alien Kong 于2025年5月10日周六 22:55写道： > Hi, Paul > > Thank you for your reply. > > This is a low-probability, sporadic problem. Currently, there is no > `journalctl -b` information with `debug` on the command line. > > The unit information of the two services is as follows: > 1. systemctl cat systemd-resolved > > # /usr/lib/systemd/system/systemd-resolved.service > # SPDX-License-Identifier: LGPL-2.1-or-later > # > # This file is part of systemd. > # > # systemd is free software; you can redistribute it and/or modify it > # under the terms of the GNU Lesser General Public License as published by > # the Free Software Foundation; either version 2.1 of the License, or > # (at your option) any later version. > > [Unit] > Description=Network Name Resolution > Documentation=man:systemd-resolved.service(8) > Documentation=man:org.freedesktop.resolve1(5) > Documentation= > https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers > Documentation= > https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients > > DefaultDependencies=no > After=systemd-sysctl.service systemd-sysusers.service > Before=sysinit.target network.target nss-lookup.target shutdown.target > initrd-switch-root.target > Conflicts=shutdown.target initrd-switch-root.target > Wants=nss-lookup.target > > [Service] > AmbientCapabilities=CAP_SETPCAP CAP_NET_RAW CAP_NET_BIND_SERVICE > BusName=org.freedesktop.resolve1 > CapabilityBoundingSet=CAP_SETPCAP CAP_NET_RAW CAP_NET_BIND_SERVICE > ExecStart=!!/usr/lib/systemd/systemd-resolved > LockPersonality=yes > MemoryDenyWriteExecute=yes > NoNewPrivileg
[systemd-devel] Question: systemd-resolved.service delayed almost 90s start after systemd-sysctl.service
Hello, I'm seeing an unexpected 90s delay between systemd-sysctl.service and systemd-resolved.service in our boot sequence (systemd 255.4-1ubuntu8.4, custom embedded board). systemd-analyze critical-chain shows: multi-user.target @1min 54.340s └─user_medium_priority.service @1min 32.653s +21.686s └─basic.target @1min 32.617s └─sockets.target @1min 32.615s └─ssh.socket @1min 32.614s └─sysinit.target @1min 32.580s └─systemd-resolved.service @1min 32.185s +394ms └─systemd-sysctl.service @2.726s +89ms └─systemd-modules-load.service @1.932s +577ms └─systemd-journald.socket @1.911s └─-.mount @599ms └─-.slice @598ms We have verified no explicit dependency holding this back. >From the information of systemd-analyze blame, the startup time of each service is normal. 21.686s user_medium_priority.service 2.537s nv_duv3.service 1.996s dev-vblkdev30.device 1.993s dev-vblkdev0.device 1.818s dev-vblkdev2.device 1.766s dev-vblkdev4.device 1.762s dev-vblkdev5.device 1.670s dev-vblkdev3.device 794ms dev-vblkdev6.device 792ms dev-vblkdev15.device 785ms e2scrub_reap.service 730ms systemd-logind.service 711ms dev-vblkdev13.device 703ms dev-vblkdev8.device 667ms dev-vblkdev14.device 657ms dev-vblkdev7.device 635ms dev-vblkdev9.device 577ms systemd-modules-load.service 550ms systemd-networkd.service 394ms systemd-resolved.service 367ms ap-com-daemon.service 286ms systemd-udev-trigger.service 253ms user@1000.service 248ms systemd-tmpfiles-setup.service 227ms ssh.service 124ms systemd-tmpfiles-clean.service 119ms systemd-tmpfiles-setup-dev.service 116ms systemd-user-sessions.service 114ms systemd-tmpfiles-setup-dev-early.service ... Any ideas for further investigation of this situation? Thanks~ Any insight would be appreciated. Best regards, Alien Kong
Re: [systemd-devel] Question: systemd-resolved.service delayed almost 90s start after systemd-sysctl.service
Hello, > I found that systemd entered the D state. From the kernel stack, it was blocked at __skb_wait_for_more_packets. Is there any way to determine where the problem occurred? Thank you~ *# cat /proc/1/stack* > [<0>] __skb_wait_for_more_packets+0x134/0x1a0 > [<0>] __unix_dgram_recvmsg+0xdc/0x380 > [<0>] unix_seqpacket_recvmsg+0x60/0xa0 > [<0>] sys_recvmsg+0x1f8/0x2b0 > [<0>] ___sys_recvmsg+0x98/0x100 > [<0>] __sys_recvmsg+0x88/0xf0 > [<0>] __arm64_sys_recvmsg+0x2c/0x40 > [<0>] invoke_syscall+0x50/0x140 > [<0>] el0_svc_common.constprop.0+0x68/0x140 > [<0>] do_el0_svc+0x2c/0x90 > [<0>] el0_svc+0x30/0xb0 > [<0>] el0t_64_sync_handler+0x124/0x130 > [<0>] el0t_64_sync+0x190/0x194 *# cat /proc/1/status* > Name:systemd > Umask: > State:D (disk sleep) > Tgid:1 > Ngid:0 > Pid:1 > PPid:0 > TracerPid:0 > Uid:0000 > Gid:0000 > FDSize:256 > Groups: > NStgid:1 > NSpid:1 > NSpgid:1 > NSsid:1 > VmPeak: 22332 kB > VmSize: 22296 kB > VmLck: 0 kB > VmPin: 0 kB > VmHWM: 12840 kB > VmRSS: 12836 kB > RssAnon:3932 kB > RssFile:8904 kB > RssShmem: 0 kB > VmData:3340 kB > VmStk: 132 kB > VmExe: 84 kB > VmLib: 15676 kB > VmPTE: 84 kB > VmSwap: 0 kB > HugetlbPages: 0 kB > CoreDumping:0 > THP_enabled:0 > Threads:1 > SigQ:2/239905 > SigPnd: > ShdPnd:0001 > SigBlk:7fefc1fe28014a03 > SigIgn:1000 > SigCgt:04ec > CapInh: > CapPrm:01ff > CapEff:01ff > CapBnd:01ff > CapAmb: > NoNewPrivs:0 > Seccomp:0 > Seccomp_filters:0 > Speculation_Store_Bypass:thread vulnerable > SpeculationIndirectBranch:unknown > Cpus_allowed:fff > Cpus_allowed_list:0-11 > Mems_allowed:0001 > Mems_allowed_list:0 > voluntary_ctxt_switches:2594 > nonvoluntary_ctxt_switches:576 Best regards, Alien Kong
[systemd-devel] Questions about executing systemctl isolate and systemctl start in sequence
Hello, I'd like to ask if systemctl isolate and systemctl start can be executed together. *Background* A.target is the system's default.target. A.target is associated with A.service, which executes A.sh. That is, systemd first executes A.sh after booting. A.sh switches to multi-user.target via systemctl isolate multi-user.target in the final stage. The following statement immediately executes systemctl start X.service. X.service has a default dependency on sysinit.target. *A.sh* > > > > *...systemctl --no-block isolate multi-user.targetsystemctl start > X.service* *Question* Does this syntax potentially cause issues, such as systemd server timeouts or job conflicts? Thanks. Best regards, Alien Kong
