This bug is missing log files that will aid in diagnosing the problem.
From a terminal window please run:

apport-collect 2098730

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable
to run this command, please add a comment stating that and change the
bug status to 'Confirmed'.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2098730

Title:
  Kernel 6.8.0 memory leak

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  We discovered a kernel memory leak in the Ubuntu 24.04 distribution
  using the 6.8.0 series kernel while testing on our cluster.

  After performing a fresh installation of Ubuntu Server 24.04 from the
  official ISO: `https://ubuntu.com/download/server/thank-
  you?version=24.04.1&architecture=amd64&lts=true`, with the built-in
  kernel `linux-image-6.8.0-51-generic`, we observed that Slab consumes
  all available memory due to the kernel threads `kmalloc-rnd-(*)-2k`
  and `skbuff_head_cache` when the audit subsystem is enabled. The value
  of `SUnreclaim` in `/proc/meminfo` nearly reaches the system's total
  memory.

  We validated this bug on both amd64 and aarch64 architectures,
  affecting bare metal and virtual machines alike across all platforms.
  Here’s the stack information obtained using the eBPF tool `memleak`:
  "https://github.com/iovisor/bcc/blob/master/tools/memleak.py"; to
  detect the kernel memory leak:

    195362816 bytes in 5962 allocations from stack
            0xffffffff96652314      __alloc_pages+0x264 [kernel]
            0xffffffff96652314      __alloc_pages+0x264 [kernel]
            0xffffffff9665b0a8      allocate_slab+0xa8 [kernel]
            0xffffffff9665b3b8      new_slab+0x38 [kernel]
            0xffffffff9665e3d5      ___slab_alloc+0x435 [kernel]
            0xffffffff9665f62b      __kmalloc_node_track_caller+0x18b [kernel]
            0xffffffff970d8317      kmalloc_reserve+0x67 [kernel]
            0xffffffff970db4aa      __alloc_skb+0x8a [kernel]
            0xffffffff96457f98      audit_log_start+0x198 [kernel]
            0xffffffff96462103      audit_log_exit+0x433 [kernel]
            0xffffffff96462dbe      __audit_syscall_exit+0xee [kernel]
            0xffffffff963f111b      syscall_exit_work+0x12b [kernel]
            0xffffffff963f1189      syscall_exit_to_user_mode_prepare+0x39 
[kernel]
            0xffffffff97430ae1      syscall_exit_to_user_mode+0x11 [kernel]
            0xffffffff97428fec      do_syscall_64+0x8c [kernel]
            0xffffffff97600130      entry_SYSCALL_64_after_hwframe+0x78 [kernel]

  To reproduce this memory leak issue, follow these steps:
  1. Install and start auditd.
  2. Add audit rules in `/etc/audit/rules.d/audit.rules`:
    ```
    -D
    -b 8192
    -f 1
    -r 100
    -a always,exit -F arch=b64 -S openat -S truncate -S ftruncate -F 
exit=-EACCES -F auid>=1000 -F auid!=4294967295 -k access
    ```
  3. Reboot or run `augenrules --load`, then execute `auditctl -l` to verify 
that the audit rule above has been loaded.
  4. Run command `while :; do cat /proc/1/environ; done` as a normal user (uid 
>= 1000) to triger the audit events.
  5. Monitor kernel memory allocation by running either: 
    ```
    watch -d -n 1 'cat /proc/meminfo | grep -i SUnreclaim'
    ```
    or simply use 
    ```
    slabtop -s c
    ```

  We tested several kernels within the 6.8.0 series, and this bug was
  present in all of them, including when installing HWE Kernel 6.8.0 in
  Ubuntu 22.04 via `apt install linux-generic-hwe-22.04`. However, after
  installing mainline kernels v6.8.1 or higher from
  https://kernel.ubunt.com/mainline/, this bug disappears, indicating it
  may have been fixed on upstream.

  Therefore, it is essential to update your stable repository's kernel
  and refresh your ISO accordingly.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2098730/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to