** Tags added: ubuntu-certified
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1861359
Title:
swap storms kills interactive use
Status in linux package in Ubuntu:
Fix Released
Status
I was reminded of this bug earlier today -- Andrea, Sultan, thanks so
much for fixing my issues. I've been happily running along for months
now. :) Thanks!
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launc
This bug was fixed in the package linux - 5.4.0-26.30
---
linux (5.4.0-26.30) focal; urgency=medium
* focal/linux: 5.4.0-26.30 -proposed tracker (LP: #1873882)
* Packaging resync (LP: #1786013)
- update dkms package versions
* swap storms kills interactive use (LP: #186135
** Description changed:
[Impact]
- High watermark boosting can cause large swap activity under certain
- memory intensive workloads, making the system very unresponsive (screen
- does not refresh, keyboard not responding, etc.).
-
- This large swap activity seems to be prevented disabling hi
Seth, thanks for the update!
JFYI, I've just upladed also a v3 kernel (5.4.0-24.28+lp1861359v3) that
I'm currently testing on my laptop with positive result. This change is
even smaller than the previous one (v2), because we simply disable the
direct swap out in the i915 shrinker (I915_SHRINK_WRIT
Andrea, I've been running the v1 kernel for a day or so now:
[0.00] Linux version 5.4.0-24-generic (arighi@sita) (gcc version
9.3.0 (Ubuntu 9.3.0-10ubuntu1)) #28+lp1861359v1 SMP Wed Apr 15 14:49:33
UTC 2020 (Ubuntu 5.4.0-24.28+lp1861359v1-generic 5.4.30)
$ uptime
02:21:45 up 1 day, 9 min
This entry:
* swap storms kills interactive use (LP: #1861359)
- SAUCE: mm/page_alloc.c: disable memory reclaim watermark boosting by
default
closed this bug, but per latest comments, that isn't sufficient to
address the issue. Putting back to Confirmed.
** Changed in: linux (Ubuntu Focal
This bug was fixed in the package linux - 5.4.0-24.28
---
linux (5.4.0-24.28) focal; urgency=medium
* focal/linux: 5.4.0-24.28 -proposed tracker (LP: #1871939)
* getitimer returns it_value=0 erroneously (LP: #1349028)
- [Config] CONTEXT_TRACKING_FORCE policy should be unset
I've uploaded another test kernel (5.4.0-24.28+lp1861359v2):
https://kernel.ubuntu.com/~arighi/LP-1861359/
In this one, instead of completely disabling the i915 shrinker, I'm only
preventing to swap out the i915 caches when the system is short on
memory.
I'm testing this new one on my laptop righ
Hi Seth, sorry for my late response.
I did more tests this morning on my laptop tracing the callers of
__alloc_pages_nodemask() and I noticed that pretty much all the time it
is called by the i915 shrinker. So I tried to disable it and I have to
say that on my laptop (at least) the system is alway
I should point out that the period bursts of writes every five seconds
in my vmstat 1 output is due to zfs's flushing mechanism; by default it
flushes dirty pages every five seconds.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in
Thanks Andrea, I don't think that helped. I'll attach a file with vmstat
1 output and funclatency output, along with a few notes on the testing.
Thanks
** Attachment added: "after-limiting-dirty-bytes"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1861359/+attachment/5349071/+files/aft
TL;DR @seth-arnold, as a test can you try to set the following options?
$ echo $((32 * 1024 * 1024)) | sudo tee /proc/sys/vm/dirty_bytes
$ echo $((32 * 1024 * 1024)) | sudo tee /proc/sys/vm/dirty_background_bytes
Repeat the test and see if the system is still unresponsive.
Details below.
Th
Stefan, while recent kernels seem happier than previous kernels (I think
-14 era was terrible), I don't think this problem is fixed yet:
sarnold@millbarge:/tmp$ uname -a
Linux millbarge 5.4.0-21-generic #25-Ubuntu SMP Sat Mar 28 13:10:28 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
sarnold@millbarge:/
Reporter hasn't confirmed that it's corrected yet... "Fix committed"
seems premature.
** Changed in: linux (Ubuntu Focal)
Status: Fix Committed => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://b
** Also affects: linux (Ubuntu Focal)
Importance: High
Assignee: Andrea Righi (arighi)
Status: Confirmed
** Changed in: linux (Ubuntu Focal)
Status: Confirmed => Fix Committed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscr
Sultan put together a kernel with some debugging for me:
[101616.889859] __alloc_pages_nodemask: stall of 3683ms for order-0, mask:
0x100dca
[101616.889863] Call Trace:
[101616.889880] __alloc_pages_nodemask+0x34f/0x3b0
[101616.889887] alloc_pages_vma+0x7f/0x200
[101616.889893] do_anonymous_pa
As a note: https://platform.leolabs.space/visualizations/leo is not a
valid reproducer for this bug, since the lags it causes are from
overloading the GPU, not from stressing memory.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in
Sultan, thanks for the advice.
I set this watermark boost factor to zero as you suggested, and then
decided to try a stupid simple benchmark of my storage -- my swap is a
zfs dataset on nvme. zfs means it'll go slower than raw nvme block
access:
$ dd if=ubuntu-18.04.4-desktop-amd64.iso of=foo bs=
FYI, this bug has nothing to do with the use of swap. It just happens
that the slow writeback incurred by using a swap device backed by non-
volatile memory makes kswapd's bouts of page thrashing last longer,
enough to the point where there's a visual freeze.
--
You received this bug notification
** Description changed:
+ [Impact]
+
+ High watermark boosting can cause large swap activity under certain
+ memory intensive workloads, making the system very unresponsive (screen
+ does not refresh, keyboard not responding, etc.).
+
+ This large swap activity seems to be prevented disabling hi
This problem is caused by an upstream memory management feature called
watermark boosting. Normally, when a memory allocation fails and falls
back to the page allocator, the page allocator will wake up kswapd to
free up pages in order to make the memory allocation succeed. kswapd
tries to free memo
I'm adding the champagne tag to this bug to bring it to a potential
wider audience; I think we may need to take more drastic steps like
disabling swap on upgrades, not offering swap in our installers, etc.,
to try to have a better experience.
Thanks
** Tags added: champagne
--
You received this
BTW, this is still happening in:
Linux millbarge 5.4.0-20-generic #24-Ubuntu SMP Mon Mar 23 20:55:46 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
I've seen it both with firefox in trello, firefox in launchpad (typing
this comment) and doing two sequential wgets of
http://releases.ubuntu.com/18.04/ubun
Using 5.4.0-17.21-generic, my laptop has 16G of ram. If I launch 3 vms
(xenial desktop (768M), bionic desktop (1.5G) and focal desktop (2.6G))
then load this page: https://people.canonical.com/~ubuntu-
security/oval/com.ubuntu.xenial.cve.oval.xml, at some point while the
page is loading, the deskto
There is an interesting (to me, anyway) change of behaviour with the -17
kernel: while earlier kernels would appear to be locked solid for 30-60
seconds before the screen could update, -17 allows screen updates every
six seconds or so.
I have an always-running mosh session to a remote host running
@seth-arnold ok I'll do this tests also on my side and see if I can
reproduce the problem. If you find a specific web page that can trigger
the problem easily let me know. Thanks!
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ub
Andrea, unfortunately this updated kernel hasn't fixed the problem:
01:02:48 up 21:21, 9 users, load average: 1.45, 0.98, 0.58
Linux millbarge 5.4.0-17-generic #21-Ubuntu SMP Fri Feb 28 16:18:44 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
I was able to reproduce the swap growth and hangs with Fi
Andrea, this new kernel looks promising.
Linux millbarge 5.4.0-17-generic #21-Ubuntu SMP Fri Feb 28 16:18:44 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
Unfortunately I didn't check the reproducer before rebooting: the error
message I get with it now suggests that it might not work any more.
However
The depmod error messages have been fixed in initramfs-tools (see
https://bugs.launchpad.net/bugs/1863261). It doesn't actually prevent
the kernel from booting, so you can safely reboot.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linu
On Sat, Feb 29, 2020 at 10:06:42AM -, Andrea Righi wrote:
> Seth, can you try to see if you can reproduce the problem with the
> latest unstable kernel (5.4.0-17.21)? https://launchpad.net/~canonical-
> kernel-team/+archive/ubuntu/unstable
>
> I can't reproduce the problem with it. I have not
Seth, can you try to see if you can reproduce the problem with the
latest unstable kernel (5.4.0-17.21)? https://launchpad.net/~canonical-
kernel-team/+archive/ubuntu/unstable
I can't reproduce the problem with it. I have not verified yet, but I
suspect it might be related to this commit:
d92ff
Many thanks for the reproducer Seth! I've been able to reproduce the
swapping issue on my laptop! Now I can investigate more on my side. I'll
keep you posted!
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.la
This web page may be a good reproducer candidate:
https://platform.leolabs.space/visualizations/conjunction?type=conjunction&reportId=2004981040
Loading it in firefox would make my computer unresponsive for over a
minute. (Be careful with firefox reloading it when re-opening firefox.)
Loading it
I guess we can't use ftrace and secure boot at the same time then...
would it be possible to disable secure boot / kernel lockdown on your
side and run a test using that kprobe-perf command?
If it's not possible or too complicated we'll find an alternative way,
maybe I can create a custom kernel a
$ uname -a
Linux millbarge 5.4.0-12-generic #15-Ubuntu SMP Tue Jan 21 15:12:29 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
$ cat /proc/sys/kernel/ftrace_enabled
1
$ sudo kprobe-perf -s 'p:shrink_node'
[sudo] password for sarnold:
ERROR: func shrink_node not in
/sys/kernel/debug/tracing/available_fil
Ah! You're right, that's the reason! When the kernel is locked down
ftrace is explicitly disabled. To confirm that, you should have 0 in
/proc/sys/kernel/ftrace_enabled.
Can you try to set it back to 1 and see if kprobe-perf works after that?
Otherwise I'll figure out an alternative way to trace d
$ uname -a
Linux millbarge 5.4.0-12-generic #15-Ubuntu SMP Tue Jan 21 15:12:29 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
$ grep FTRACE /boot/config-`uname -r`
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_STM_SOURCE_FTRACE=m
# CONFIG_PSTORE_FTRACE is not set
CONFIG_HAVE_DYNAMIC_F
Weird that kprobe-perf isn't working... I've just tried it on a fresh
new installed 20.04 instance and:
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 5.4.0-12-generic #15-Ubuntu SMP Tue Jan 21 15:12:29 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ubuntu:~$ sudo ls -l /sys/kernel/debug/tracing/availabl
> What do you have in /proc/sys/vm/swappiness? Could you try to set that
> to 0 (kernel prefers to drop file-backed pages instead of swapping out
> anonymous pages) and see if the swap out activity is still happening?
Unfortunately, this did not solve the problem.
Setting swappiness to 0 and re-e
On Mon, Feb 10, 2020 at 07:53:20AM -, Andrea Righi wrote:
> OK, so we know that it's not related to the memory cgroup subsystem.
But this is a good instinct. It does seem to happen when eg firefox or git
is in heavy memory use, not the system as a whole.
> Another reason of such unexpected sw
OK, so we know that it's not related to the memory cgroup subsystem.
Another reason of such unexpected swapping activity could be due to
memory compaction code that is triggering some direct memory reclaim and
forcing to swap out pages.
What do you have in /proc/sys/vm/swappiness? Could you try t
$ systemctl show '*.slice' | grep -e '^Slice' -e '^ControlGroup' -e
"^DefaultMem" -e "^Memory"
Slice=system.slice
ControlGroup=/system.slice/system-systemd\x2dfsck.slice
MemoryCurrent=[not set]
MemoryAccounting=yes
DefaultMemoryLow=0
DefaultMemoryMin=0
MemoryMin=0
MemoryLow=0
MemoryHigh=infinity
M
This kernel command line parameter didn't appear to help:
Moments after loading a new URL into firefox:
1 00 69917617804 369982000 0 0
2289 863 2 1 98 0 0
0 00 70286017804 369978000 0 2288
4
Hello Seth, thanks for reporting the problem. I was wondering if this
could be related to the memory cgroup controller.
As a simple test could you try to reboot the system adding
cgroup_disable=memory to the kernel boot parameters?
In this way if the problem goes away at least we know it's relate
** Changed in: linux (Ubuntu)
Assignee: Colin Ian King (colin-king) => Andrea Righi (arighi)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1861359
Title:
swap storms kills interact
I had a vmstat 1 running; the entire time the system was swapping out, X
was unusable.
Thanks
** Attachment added: "vmstat 1 output"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1861359/+attachment/5324495/+files/vmstat-1
--
You received this bug notification because you are a membe
** Changed in: linux (Ubuntu)
Assignee: (unassigned) => Colin Ian King (colin-king)
** Changed in: linux (Ubuntu)
Importance: Undecided => High
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpa
I forgot to mention, I also have nvme.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1861359
Title:
swap storms kills interactive use
Status in linux package in Ubuntu:
Confirmed
Bu
FYI, I decided to do this:
$ sudo swapoff -a && sudo swapon -a
$ free -h
totalusedfree shared buff/cache available
Mem: 15Gi 5.9Gi 4.8Gi 2.0Gi 4.8Gi 7.2Gi
Swap: 15Gi 348Mi15Gi
Even though I am no
Seth and I talked about this and I marked this as affects me. If it
helps, I saw this on eoan and focal doesn't make a difference (which
might suggest the change is between disco and eoan).
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to l
** Changed in: linux (Ubuntu)
Status: Incomplete => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1861359
Title:
swap storms kills interactive use
Status in linux pack
apport information
** Description changed:
Hello, several times since upgrading to focal from 19.04 I've found my
computer entirely unresponsive for periods of twenty or thirty seconds.
No mouse movement, no keyboard input, the screen output does not change.
My computer was using swap
apport information
** Tags added: apport-collected
** Description changed:
Hello, several times since upgrading to focal from 19.04 I've found my
computer entirely unresponsive for periods of twenty or thirty seconds.
No mouse movement, no keyboard input, the screen output does not change.
** Package changed: linux-signed-5.4 (Ubuntu) => linux (Ubuntu)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1861359
Title:
swap storms kills interactive use
Status in linux package i
55 matches
Mail list logo
