On Sun, Mar 15, 2026 at 09:24:39PM -0500, Bjorn Andersson wrote: > On Wed, Mar 11, 2026 at 01:45:44AM +0530, Mukesh Ojha wrote: > > First of all, I want to thank Eugene for his excellent work on this > > series. What began as the Qualcomm Minidump driver from me has now > > evolved into meminspect. He also presented meminspect a few months ago > > at Linux Plumbers 2025. > > > > Video of the recording is available here for anyone interested: > > https://www.youtube.com/watch?v=aDZv4-kOLSc > > > > Introduction: > > > > meminspect is a mechanism which allows the kernel to mark specific > > memory areas for memory dumping or specific inspection, statistics, > > usage. Once regions are marked, meminspect keeps an internal list with > > the regions in a dedicated table. Further, these regions can be > > accessed using specific API by any interested driver. Regions being > > marked beforehand, when the system is up and running, there is no need > > nor dependency on a panic handler, or a working kernel that can dump the > > debug information. meminspect can be primarily used for debugging. The > > approach is feasible to work when pstore, kdump, or another mechanism do > > not. Pstore relies on persistent storage, a dedicated RAM area or > > flash, which has the disadvantage of having the memory reserved all the > > time, or another specific non volatile memory. Some devices cannot keep > > the RAM contents on reboot so ramoops does not work. Some devices do not > > allow kexec to run another kernel to debug the crashed one. For such > > devices, that have another mechanism to help debugging, like firmware, > > kmemdump is a viable solution. > > > > meminspect can create a core image, similar with /proc/vmcore, with only > > the registered regions included. This can be loaded into crash tool/gdb > > and analyzed. This happens if CRASH_DUMP=y. To have this working, > > specific information from the kernel is registered, and this is done at > > meminspect init time, no need for the meminspect users to do anything. > > > > This version of the meminspect patch series includes two drivers that > > make use of it: one is the Qualcomm Minidump, and the other one is the > > Debug Kinfo backend for Android devices, reworked from this source here: > > https://android.googlesource.com/kernel/common/+/refs/heads/android-mainline/drivers/android/debug_kinfo.c > > written originally by Jone Chou <[email protected]> > > > > *** History, motivation and available online resources *** > > > > The patch series is based on both minidump and kmemdump previous > > implementations. > > > > After the three RFC kmemdump versions, considering the ML discussions, it > > was decided to > > move this into kernel/ directory and rework it into naming it meminspect, > > as Thomas Gleixner > > suggested. > > > > Initial version of kmemdump and discussion is available here: > > https://lore.kernel.org/lkml/[email protected]/ > > > > Kmemdump has been presented and discussed at Linaro Connect 2025, > > including motivation, scope, usability and feasability. > > Video of the recording is available here for anyone interested: > > https://www.youtube.com/watch?v=r4gII7MX9zQ&list=PLKZSArYQptsODycGiE0XZdVovzAwYNwtK&index=14 > > > > Linaro blog on kmemdump can be found here: > > https://www.linaro.org/blog/introduction-to-kmemdump/ > > > > Linaro blog on kmemdump step by stem using minidump backend is available > > here: > > https://www.linaro.org/blog/kmemdump-step-by-step-on-qualcomm-automotive-platform/ > > > > The implementation is based on the initial Pstore/directly mapped zones > > published as an RFC here: > > https://lore.kernel.org/all/[email protected]/ > > > > The back-end implementation for qcom_minidump is based on the minidump > > patch series and driver written by Mukesh Ojha, thanks: > > https://lore.kernel.org/lkml/[email protected]/ > > > > The RFC v2 version with .section creation and macro annotation kmemdump > > is available here: > > https://lore.kernel.org/all/[email protected]/ > > > > The RFC v3 version with making everything static, which was pretty much > > rejected due to > > all reasons discussed on the public ML: > > https://lore.kernel.org/all/[email protected]/ > > > > *** How to use meminspect with minidump backend on Qualcomm platform guide > > *** > > > > Prerequisites: > > Crash tool compiled with target=ARM64 and minor changes required for > > usual crash mode (minimal mode works without the patch) **A patch can be > > applied from here https://p.calebs.dev/1687bc ** This patch will be > > eventually sent in a reworked way to crash tool. > > > > That patch was written 8 months ago, what's the timeline for landing > this? > > It's not feasible to have every users rebuild crash from source and > maintain this copy in order to use the tool.
Right, Let me see what I can do to make it acceptable.. > > > Target kernel must be built with : CONFIG_DEBUG_INFO_REDUCED=n ; this > > will have vmlinux include all the debugging information needed for crash > > tool. > > > > Also, the kernel requires these as well: CONFIG_MEMINSPECT, > > CONFIG_CRASH_DUMP and the driver CONFIG_QCOM_MINIDUMP > > > > Kernel arguments: Kernel firmware must be set to mode 'mini' by kernel > > module parameter like this : qcom_scm.download_mode=mini > > > > After the kernel boots, and minidump module is loaded, everything is > > ready for a possible crash. > > > > Once the crash happens, the firmware will kick in and you will see on > > the console the message saying Sahara init, etc, that the firmware is > > waiting in download mode. (this is subject to firmware supporting this > > mode, I am using sa8775p-ride board) > > > > Example of log on the console: > > " > > [...] > > B - 1096414 - usb: init start > > B - 1100287 - usb: qusb_dci_platform , 0x19 > > B - 1105686 - usb: usb3phy: PRIM success: lane_A , 0x60 > > B - 1107455 - usb: usb2phy: PRIM success , 0x4 > > B - 1112670 - usb: dci, chgr_type_det_err > > B - 1117154 - usb: ID:0x260, value: 0x4 > > B - 1121942 - usb: ID:0x108, value: 0x1d90 > > B - 1124992 - usb: timer_start , 0x4c4b40 > > B - 1129140 - usb: vbus_det_pm_unavail > > B - 1133136 - usb: ID:0x252, value: 0x4 > > B - 1148874 - usb: SUPER , 0x900e > > B - 1275510 - usb: SUPER , 0x900e > > B - 1388970 - usb: ID:0x20d, value: 0x0 > > B - 1411113 - usb: ENUM success > > B - 1411113 - Sahara Init > > B - 1414285 - Sahara Open > > " > > This doesn't add any specific value, it's just "Device entered ramdump > mode". Sure, will remove it. > > > > > Once the board is in download mode, you can use the qdl tool (I > > personally use edl , have not tried qdl yet) > > Is this your or Eugen's comment? Why haven't you tested qdl yet? It was Eugene's comment, but I get it, I can try QDL and check, if it works. > > >, to get all the regions as > > separate files. The tool from the host computer will list the regions > > in the order they were downloaded. > > > > Once you have all the files simply use `cat` to put them all together, > > in the order of the indexes. For my kernel config and setup, here is my > > cat command : (you can use a script or something, I haven't done that so > > far): > > So these need to be sorted in numerical order, by that number at the end > of the file name? > > Do you manually punch these in? How do we make this user friendly? Yes, manually.. but I think we can do better. We could make this more user‑friendly by using the section header and string table in the md_KELF binary both of which existed in the earlier implementation. Then, we can write an upstream‑friendly script that reads this KELF metadata file, checks whether a binary with the registered name is present, and stitches everything together to form a complete ELF that the crash tool can consume. Let me know if you have any suggestion.. > > Regards, > Bjorn > > > > > `cat md_KELF1.BIN md_Kvmcorein2.BIN md_Kconfig3.BIN \ > > md_Ktotalram4.BIN md_Kcpu_poss5.BIN md_Kcpu_pres6.BIN \ > > md_Kcpu_onli7.BIN md_Kcpu_acti8.BIN md_Kmem_sect9.BIN \ > > md_Kjiffies10.BIN md_Klinux_ba11.BIN md_Knr_threa12.BIN \ > > md_Knr_irqs13.BIN md_Ktainted_14.BIN md_Ktaint_fl15.BIN \ > > md_Knode_sta16.BIN md_K__per_cp17.BIN md_Knr_swapf18.BIN \ > > md_Kinit_uts19.BIN md_Kprintk_r20.BIN md_Kprintk_r21.BIN \ > > md_Kprb22.BIN md_Kprb_desc23.BIN md_Kprb_info24.BIN \ > > md_Kprb_data25.BIN md_Khigh_mem26.BIN md_Kinit_mm27.BIN \ > > md_Kunknown29.BIN md_Kunknown30.BIN md_Kunknown31.BIN \ > > md_Kunknown32.BIN md_Kunknown33.BIN md_Kunknown34.BIN \ > > md_Kunknown35.BIN md_Kunknown37.BIN \ > > md_Kunknown38.BIN md_Kunknown39.BIN md_Kunknown40.BIN \ > > md_Kunknown41.BIN md_Kunknown42.BIN md_Kunknown43.BIN \ > > md_Kunknown44.BIN md_Kunknown45.BIN md_Kunknown46.BIN \ > > md_Kunknown47.BIN md_Kunknown48.BIN md_Kunknown49.BIN \ > > md_Kunknown50.BIN md_Kunknown51.BIN md_Kunknown52.BIN \ > > md_Kunknown53.BIN md_Kunknown54.BIN > ./minidump_image` > > > > Once you have the resulted file, use `crash` tool to load it, like this: > > `./crash --no_modules --no_panic --no_kmem_cache --zero_excluded vmlinux > > minidump_image` > > > > There is also a --minimal mode for ./crash that would work without any > > patch applied > > to crash tool, but you can't inspect symbols, etc. > > > > Once you load crash you will see something like this : > > KERNEL: minidump/20260310-235110/vmlinux [TAINTED] > > DUMPFILE: ./minidump/20260310-235110/minidump_image > > CPUS: 8 [OFFLINE: 7] > > DATE: Thu Jan 1 05:30:00 +0530 1970 > > UPTIME: 00:00:27 > > TASKS: 0 > > NODENAME: qemuarm64 > > RELEASE: 7.0.0-rc3-next-20260309-00028-g528b3c656121 > > VERSION: #5 SMP PREEMPT Tue Mar 10 18:18:41 UTC 2026 > > MACHINE: aarch64 (unknown Mhz) > > MEMORY: 0 > > PANIC: "Kernel panic - not syncing: sysrq triggered crash" > > > > crash> log > > [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x514f0014] > > [ 0.000000] Linux version 7.0.0-rc3-next-20260309-00028-g528b3c656121 > > (@21e3bca4168f) (aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04) > > 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #5 SMP PREEMPT Tue Mar 10 > > 18:18:41 UTC 2026 > > > > *** Debug Kinfo backend driver *** > > I need help with the testing of this driver, Anyone who actually wants > > to test this, feel free to reply to the patch. we have also written a > > simple DT binding for the driver. > > > > Thanks in advance for the review, and apologies if I missed addressing any > > comment. > > > > -Mukesh > > > > Changes in v2: > > https://lore.kernel.org/lkml/[email protected]/ > > - Fixed doc warnings > > - Fixed kernel-test robot warnings. > > - Took Mike suggestion to remove mark inspect flag for dynamic memory. > > - Added R-b for printk patch. > > - Modified some commit messages for clarity. > > - corrected binding change for debug-kinfo as per Rob suggestion. > > > > Changelog for meminspect v1: > > - rename to meminspect > > - start on top of v2 actually, with the section and all. > > - remove the backend thing, change the API to access the table > > - move everything to kernel/ > > - add dependency to CRASH_DUMP instead of a separate knob > > - move the minidump driver to soc/qcom > > - integrate the meminspect better into memblock by using a new memblock flag > > - minor fixes : use dev_err_probe everywhere, rearrange variable > > declarations, > > remove some useless code, etc. > > > > Changelog for RFC v3: > > - V2 available here : > > https://lore.kernel.org/all/[email protected]/ > > - Removed the .section as requested by David Hildenbrand. > > - Moved all kmemdump registration(when possible) to vmcoreinfo. > > - Because of this, some of the variables that I was registering had to be > > non-static > > so I had to modify this as per David Hildenbrand suggestion. > > - Fixed minor things in the Kinfo driver: one field was broken, fixed some > > compiler warnings, fixed the copyright and remove some useless includes. > > - Moved the whole kmemdump from drivers/debug into mm/ and Kconfigs into > > mm/Kconfig.debug > > and it's now available in kernel hacking, as per Randy Dunlap review > > - Reworked some of the Documentation as per review from Jon Corbet > > > > Changelog for RFC v2: > > - V1 available here: > > https://lore.kernel.org/lkml/[email protected]/ > > - Reworked the whole minidump implementation based on suggestions from > > Thomas Gleixner. > > This means new API, macros, new way to store the regions inside kmemdump > > (ditched the IDR, moved to static allocation, have a static default > > backend, etc) > > - Reworked qcom_minidump driver based on review from Bjorn Andersson > > - Reworked printk log buffer registration based on review from Petr Mladek > > > > I appologize if I missed any review comments. > > Patches are sent on top on next-20260309 tag > > > > --- > > Eugen Hristev (21): > > kernel: Introduce meminspect > > init/version: Annotate static information into meminspect > > mm/percpu: Annotate static information into meminspect > > cpu: Annotate static information into meminspect > > genirq/irqdesc: Annotate static information into meminspect > > timers: Annotate static information into meminspect > > kernel/fork: Annotate static information into meminspect > > mm/page_alloc: Annotate static information into meminspect > > mm/show_mem: Annotate static information into meminspect > > mm/swapfile: Annotate static information into meminspect > > kernel/vmcore_info: Register dynamic information into meminspect > > kernel/configs: Register dynamic information into meminspect > > mm/init-mm: Annotate static information into meminspect > > panic: Annotate static information into meminspect > > kallsyms: Annotate static information into meminspect > > mm/mm_init: Annotate static information into meminspect > > sched/core: Annotate runqueues into meminspect > > remoteproc: qcom: Move minidump data structures into its own header > > soc: qcom: Add minidump backend driver > > soc: qcom: smem: Add minidump platform device > > meminspect: Add debug kinfo compatible driver > > > > Mukesh Ojha (4): > > mm/numa: Register node data information into meminspect > > mm/sparse: Register information into meminspect > > printk: Register information into meminspect > > dt-bindings: reserved-memory: Add Google Kinfo Pixel reserved memory > > > > Documentation/dev-tools/index.rst | 1 + > > Documentation/dev-tools/meminspect.rst | 144 +++++++ > > .../bindings/reserved-memory/google,kinfo.yaml | 46 ++ > > MAINTAINERS | 14 + > > drivers/of/platform.c | 1 + > > drivers/remoteproc/qcom_common.c | 56 +-- > > drivers/soc/qcom/Kconfig | 13 + > > drivers/soc/qcom/Makefile | 1 + > > drivers/soc/qcom/minidump.c | 272 ++++++++++++ > > drivers/soc/qcom/smem.c | 10 + > > include/asm-generic/vmlinux.lds.h | 13 + > > include/linux/meminspect.h | 263 ++++++++++++ > > include/linux/soc/qcom/minidump.h | 72 ++++ > > init/Kconfig | 1 + > > init/version-timestamp.c | 3 + > > init/version.c | 3 + > > kernel/Makefile | 1 + > > kernel/configs.c | 6 + > > kernel/cpu.c | 5 + > > kernel/fork.c | 3 + > > kernel/irq/irqdesc.c | 2 + > > kernel/kallsyms.c | 9 + > > kernel/meminspect/Kconfig | 30 ++ > > kernel/meminspect/Makefile | 4 + > > kernel/meminspect/kinfo.c | 284 +++++++++++++ > > kernel/meminspect/meminspect.c | 471 > > +++++++++++++++++++++ > > kernel/panic.c | 4 + > > kernel/printk/printk.c | 11 + > > kernel/sched/core.c | 2 + > > kernel/time/timer.c | 2 + > > kernel/vmcore_info.c | 4 + > > mm/init-mm.c | 11 + > > mm/mm_init.c | 2 + > > mm/numa.c | 2 + > > mm/page_alloc.c | 2 + > > mm/percpu.c | 2 + > > mm/show_mem.c | 2 + > > mm/sparse.c | 6 + > > mm/swapfile.c | 2 + > > 39 files changed, 1725 insertions(+), 55 deletions(-) > > --- > > base-commit: 343f51842f4ed7143872f3aa116a214a5619a4b9 > > change-id: 20260311-minidump-v2-eed8da647ce5 > > > > Best regards, > > -- > > -Mukesh Ojha > > -- -Mukesh Ojha
