Public bug reported: BugLink: https://bugs.launchpad.net/bugs/2162904
On some laptops with a dGPU and an iGPU, the DMCUB's state transition between sleep and wake states on the laptop's built-in panel will (at irregular but frequent intervals, a few times per day in my experience) cause the amdgpu driver to stop working correctly. While some users have reported similar bugs that occur even when no external display is connected, I have personally only experienced this error when I have an external display connected. Given the inconsistency, my suspicion is that this is a race condition that is substantially exacerbated when multiple displays are attached (potentially moreso when this involves interactions between the amdgpu and NVIDIA modules, as in my laptop's case). The issue typically occurs when many applications are open. In my case, this is usually when I have a game or various dev tools active on the external monitor, and Discord (mostly visibly idle) on the built-in panel. Some of us in the community have also noticed correlation with Chromium-based apps being open, but I don't believe causation has been proven for that yet. [ Impact ] When the issue occurs, the user's built-in panel will freeze completely. If connected to an external monitor, that external monitor continues to work, as do many other parts of the system. However, they'll quickly start to notice that various parts of the system that interact with the gpu will fail to work correctly. Trying to drag windows from the frozen monitor to the external monitor will continue to partially work, but eventually will start breaking (ex: artifacts will start appearing, gnome menus for alt+tab will start freezing in place, shutdown menu will be empty, attempting to shutdown will cause unexpected hangs and possibly panics). Background audio and related things usually still work at this point, but many odd things will be failing (ex: won't be able to switch to virtual TTY, seemingly can't even use sysrq+reisub). At this stage, it is only a matter of time before the system becomes completely unusable, at which point the user will need to hard poweroff and boot again. The following dmesg output appears around the same time as when the failure occurs: [24076.056392] amdgpu 0000:07:00.0: [drm] *ERROR* dc_dmub_srv_log_diagnostic_data: DMCUB error - collecting diagnostic data [24076.288041] amdgpu 0000:07:00.0: [drm] *ERROR* dc_dmub_srv_log_diagnostic_data: DMCUB error - collecting diagnostic data [24076.518883] amdgpu 0000:07:00.0: [drm] *ERROR* dc_dmub_srv_log_diagnostic_data: DMCUB error - collecting diagnostic data [24086.631908] amdgpu 0000:07:00.0: [drm] *ERROR* [CRTC:364:crtc-0] flip_done timed out From the data I have gathered, this is the only output that I have observed to be consistent across all instances of this failure mode on my machine. Various upstream discussions exist detailing this and related issues on varying hardware/kernels/distros, where this also tends to be consistent. Triaged as "High" given how disruptive this is for impacted users. [ Fix ] As far as I can tell, the underlying issue here is a race condition that occurs between some combination of the DMCUB, amdgpu kernel module, device firmware, and possibly also nvidia module on impacted systems. I have not been able to find a concrete patch series that fixes the race condition itself, and it seems that work is still ongoing upstream to improve amdgpu module recovery when this happens. Given the intrusiveness of this issue, my proposal is to simply disable the panel self-refresh feature whenever an external display is connected. Various community findings [0] had previously suggested doing this via the `amdgpu.dcdebugmask=0x10` kernel option. I have had this enabled for 3+ weeks, which has eliminated the crashes for me. My proposed patch has the same impact, but only when an external display is connected rather than unconditionally (so we don't lose the power saving benefit of PSR in the scenario where this bug does not occur nearly as much). I will forward this upstream as well, but I propose applying this as UBUNTU: SAUCE since upstream will likely prefer to keep working to fix the underlying issue (which, if successful, should allow us to drop my patch.) [ Where problems could occur ] Since disabling panel self-refresh means that the built-in panel will always refresh at its standard refresh rate, users may experience decreased battery life if not connected to power. HOWEVER, given that we only disable it *when an external display is connected*, I expect that the vast majority of laptop users will not be meaningfully impacted. With the exception of people using mobile external monitors, I would expect most users who have an external display attached to typically also be connected to power. Additionally, while I have never reproduced this bug when only using my built-in panel (with no external monitor attached), related reports across other distros indicate the underlying bug might still be possible with only the built-in monitor - so this patch may not resolve the bug for 100% of users (but I am confident that it will drastically reduce if not eliminate incidence for configs like mine) [ Test Plan ] Install the patched kernel from -proposed (once it lands there) and do the following: 1) sudo watch -n 0.5 cat /sys/kernel/debug/dri/0000:07:00.0/eDP-2/psr_state # (switch eDP-2 to your built in panel identifier). You should see it switch between "0" and nonzero values when no external display is connected (indicating that PSR is still active and switching to/from sleep state regularly). After connecting an external display, confirm that only 0 is shown even when builtin panel isn't drawing (indicating that PSR has been disabled when the second display is connected). 2) Use your machine normally for 1-2 weeks and confirm that the bug does not reproduce. (It typically reproduces within 1 day, so 1-2 weeks should be a comfortable amount of time here.) [ Additional Information ] Upstream bug: https://gitlab.freedesktop.org/drm/amd/-/work_items/5394 System info: Ubuntu 26.04 Linux 7.0.0-27-generic Wayland GNOME 50 Hardware info: ASUSTeK COMPUTER INC. ROG Zephyrus G15 GA503RM_GA503RM FW version: GA503RM.318 AMD Ryzen™ 9 6900HS with Radeon™ Graphics × 16 iGPU: AMD Radeon™ 680M dGPU: NVIDIA GeForce RTX™ 3060 Laptop GPU 16.0 GiB RAM [0] ex: https://askubuntu.com/questions/1566809/ubuntu-26-04-lts- probably-unstable-driver-amdgpu ** Affects: linux (Ubuntu) Importance: High Assignee: Mitchell Augustin (mitchellaugustin) Status: In Progress ** Affects: linux (Ubuntu Resolute) Importance: High Assignee: Mitchell Augustin (mitchellaugustin) Status: In Progress ** Affects: linux (Ubuntu Stonking) Importance: High Assignee: Mitchell Augustin (mitchellaugustin) Status: In Progress ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Mitchell Augustin (mitchellaugustin) ** Also affects: linux (Ubuntu Resolute) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Stonking) Importance: Undecided Assignee: Mitchell Augustin (mitchellaugustin) Status: New ** Changed in: linux (Ubuntu Resolute) Assignee: (unassigned) => Mitchell Augustin (mitchellaugustin) ** Changed in: linux (Ubuntu Resolute) Status: New => In Progress ** Changed in: linux (Ubuntu Stonking) Status: New => In Progress ** Changed in: linux (Ubuntu Resolute) Importance: Undecided => High ** Changed in: linux (Ubuntu Stonking) Importance: Undecided => High ** Description changed: On some laptops with a dGPU and an iGPU, the DMCUB's state transition between sleep and wake states on the laptop's built-in panel will (at irregular but frequent intervals, a few times per day in my experience) cause the amdgpu driver to stop working correctly. While some users have reported similar bugs that occur even when no external display is connected, I have personally only experienced this error when I have an external display connected. Given the inconsistency, my suspicion is that this is a race condition that is at least substantially exacerbated when multiple displays are attached (potentially moreso when this involves interactions between the amdgpu and NVIDIA modules, as in my laptop's case). The issue typically occurs when many applications are open. In my case, this is usually when I have a game or various dev tools active on the external monitor, and Discord (mostly visibly idle) on the built-in panel. Some of us in the community have also noticed correlation with Chromium-based apps being open. [ Impact ] - When the issue occurs, the user's built-in panel will freeze completely. If connected to an external monitor, that external monitor continues to work, as do many other parts of the system. However, they'll quickly start to notice that various parts of the system that interact with the gpu will fail to work correctly. + When the issue occurs, the user's built-in panel will freeze completely. If connected to an external monitor, that external monitor continues to work, as do many other parts of the system. However, they'll quickly start to notice that various parts of the system that interact with the gpu will fail to work correctly. Trying to drag windows from the frozen monitor to the external monitor will continue to partially work, but eventually will start breaking (ex: artifacts will start appearing, gnome menus for alt+tab will start freezing in place, shutdown menu will be empty, attempting to shutdown will cause unexpected hangs and possibly panics). Background audio and related things usually still work at this point, but many odd things will be failing (ex: won't be able to switch to virtual TTY, seemingly can't even use sysrq+reisub). At this stage, the user will need to hard poweroff and boot again. The following dmesg output appears around the same time as when the failure occurs: [24076.056392] amdgpu 0000:07:00.0: [drm] *ERROR* dc_dmub_srv_log_diagnostic_data: DMCUB error - collecting diagnostic data [24076.288041] amdgpu 0000:07:00.0: [drm] *ERROR* dc_dmub_srv_log_diagnostic_data: DMCUB error - collecting diagnostic data [24076.518883] amdgpu 0000:07:00.0: [drm] *ERROR* dc_dmub_srv_log_diagnostic_data: DMCUB error - collecting diagnostic data [24086.631908] amdgpu 0000:07:00.0: [drm] *ERROR* [CRTC:364:crtc-0] flip_done timed out From the data I have gathered, this is the only output that I have observed to be consistent across all instances of this failure mode on my machine. Various upstream discussions exist detailing this and related issues on varying hardware/kernels/distros, where this also tends to be consistent. + Triaged as "High" given how disruptive this is for impacted users. + [ Fix ] As far as I can tell, the underlying issue here is a race condition that occurs between some combination of the DMCUB, amdgpu kernel module, device firmware, and possibly also nvidia module on impacted systems. I have not been able to find a concrete patch series that fixes the race condition itself, and it seems that work is still ongoing upstream to improve amdgpu module recovery when this happens. Given the intrusiveness of this issue, my proposal is to simply disable the panel self-refresh feature whenever an external display is connected. + Various community findings [0] had previously suggested doing this via + the `amdgpu.dcdebugmask=0x10` kernel option. I have had this enabled for + 3+ weeks, which has eliminated the crashes for me. My proposed patch has + the same impact, but only when an external display is connected rather + than unconditionally (so we don't lose the power saving benefit of PSR + in the scenario where this bug does not occur nearly as much). [ Where problems could occur ] Since disabling panel self-refresh means that the built-in panel will always refresh at its standard refresh rate, users may experience decreased battery life if not connected to power. HOWEVER, given that we only disable it *when an external display is connected*, I expect that the vast majority of laptop users will not be meaningfully impacted. With the exception of people using mobile external monitors, I would expect most users who have an external display attached to typically also be connected to power. Additionally, while I have never reproduced this bug when only using my built-in panel (with no external monitor attached), related reports across other distros indicate the underlying bug might still be possible with only the built-in monitor - so this patch may not resolve the bug for 100% of users (but I am confident that it will drastically reduce if not eliminate incidence for configs like mine) [ Test Plan ] Install the patched kernel from -proposed and do the following: 1) sudo watch -n 0.5 cat /sys/kernel/debug/dri/0000:07:00.0/eDP-2/psr_state # (switch eDP-2 to your built in panel identifier). You should see it switch between "0" and nonzero values when no external display is connected (indicating that PSR is still active and switching to/from sleep state regularly). After connecting an external display, confirm that only 0 is shown even when builtin panel isn't drawing (indicating that PSR has been disabled when the second display is connected). 2) Use your machine normally for 1-2 weeks and confirm that the bug does not reproduce. (It typically reproduces within 1 day, so 1-2 weeks should be a comfortable amount of time here.) [ Additional Information ] Upstream bug: https://gitlab.freedesktop.org/drm/amd/-/work_items/5394 System info: Ubuntu 26.04 Linux 7.0.0-27-generic Wayland GNOME 50 Hardware info: ASUSTeK COMPUTER INC. ROG Zephyrus G15 GA503RM_GA503RM FW version: GA503RM.318 AMD Ryzen™ 9 6900HS with Radeon™ Graphics × 16 iGPU: AMD Radeon™ 680M dGPU: NVIDIA GeForce RTX™ 3060 Laptop GPU 16.0 GiB RAM + + [0] ex: https://askubuntu.com/questions/1566809/ubuntu-26-04-lts- + probably-unstable-driver-amdgpu -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2162904 Title: amdgpu panel self-refresh on dual-gpu laptops causes partial system freeze To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2162904/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
