RE: Possible memory leak in the kernel (contigmalloc)
Hi, I was debugging the issue by viewing the free ques "sysctl vm.phys_free" and also using "show page" in ddb. The inactive memory is never being released back into the free que. I thought that when inactive memory reaches a certain threshold that the kernel will start reclaiming and move it to the free list? In my program this is not happening, the program uses free memory (contigmalloc), and then it is put into the inactive que (contiigfree) when the program frees it. This inactive memory is never released by the kernel, and the inactive que grows until all the memory is in this que. I have attached a xml sheet that shows the memory usage in the system. Ciunas. -Original Message- From: Konstantin Belousov [mailto:kostik...@gmail.com] Sent: Friday, October 26, 2018 9:13 PM To: Bennett, Ciunas Cc: freebsd-stable@freebsd.org Subject: Re: Possible memory leak in the kernel (contigmalloc) On Wed, Oct 24, 2018 at 04:27:52PM +, Bennett, Ciunas wrote: > Hello, > > I have encountered an issue with a kernel application that I have > written, the issue might be caused by a memory leak in the kernel. > The application allocates and deallocates contiguous memory using > contigmalloc() and contigfree(). The application will fail after a > period of time because there is not enough free contiguous memory > left. There could be an issue with the freeing of memory when using > the contigfree() function. > It is unlikely that there is an issue with a leak, but I would be not surprised if your allocation/free pattern would cause fragmentation on free lists that results in contigmalloc(9) failures after. Look at the vmstat -z/vmstat -m output to see uma and malloc stats. More interesting for your case can be the output from sysctl vm.phys_free which provides information about the free queues and order of free pages on them. -- Intel Research and Development Ireland Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Possible memory leak in the kernel (contigmalloc)
On Tue, Oct 30, 2018 at 09:46:59AM +, Bennett, Ciunas wrote: > Hi, > > I was debugging the issue by viewing the free ques "sysctl > vm.phys_free" and also using "show page" in ddb. The inactive memory > is never being released back into the free que. I thought that when > inactive memory reaches a certain threshold that the kernel will start > reclaiming and move it to the free list? In my program this is not > happening, the program uses free memory (contigmalloc), and then it > is put into the inactive que (contiigfree) when the program frees it. Contigfree() does not release memory into inactive queue. By definition, inactive pages have valid content, which is not possible for the pages freed by contigfree(). contigfree()->kmem_free()->kmem_unback()->vm_page_free(). > This inactive memory is never released by the kernel, and the inactive > que grows until all the memory is in this que. I have attached a xml > sheet that shows the memory usage in the system. Inactive memory is not freed, it makes no sense as far as there is valid content, which is either not recoverable (anon memory or dirty file pages) or high-cost to restore (clean file pages, need to re-read from disk). Inactive is processed by pagedaemon when system has memory deficit, and either inactive pages are written to swap, or written to their file backing storage. I suspect that you have other activities on your system going on, which cause creation of the inactive memory and unrecoverable fragmentation. Note that contigmalloc() tries to do defragmentation to satisfy requests, but this is not always possible. > Ciunas. > > -Original Message- > From: Konstantin Belousov [mailto:kostik...@gmail.com] > Sent: Friday, October 26, 2018 9:13 PM > To: Bennett, Ciunas > Cc: freebsd-stable@freebsd.org > Subject: Re: Possible memory leak in the kernel (contigmalloc) > > On Wed, Oct 24, 2018 at 04:27:52PM +, Bennett, Ciunas wrote: > > Hello, > > > > I have encountered an issue with a kernel application that I have > > written, the issue might be caused by a memory leak in the kernel. > > The application allocates and deallocates contiguous memory using > > contigmalloc() and contigfree(). The application will fail after a > > period of time because there is not enough free contiguous memory > > left. There could be an issue with the freeing of memory when using > > the contigfree() function. > > > > It is unlikely that there is an issue with a leak, but I would be not > surprised if your allocation/free pattern would cause fragmentation on free > lists that results in contigmalloc(9) failures after. > > Look at the vmstat -z/vmstat -m output to see uma and malloc stats. > More interesting for your case can be the output from > sysctl vm.phys_free > which provides information about the free queues and order of free pages on > them. > -- > Intel Research and Development Ireland Limited > Registered in Ireland > Registered Office: Collinstown Industrial Park, Leixlip, County Kildare > Registered Number: 308263 > > > This e-mail and any attachments may contain confidential material for the sole > use of the intended recipient(s). Any review or distribution by others is > strictly prohibited. If you are not the intended recipient, please contact the > sender and delete all copies. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
RE: Possible memory leak in the kernel (contigmalloc)
Hi, The only other activity that is running is the csh script that is inserting and removing the kernel module. I am not using the VM for any other purpose but to run this test. In my tests the correlation between memory allocation and moving to inactive list can be seen. I don't think any other process is creating the inactive memory. Thanks. -Original Message- From: Konstantin Belousov [mailto:kostik...@gmail.com] Sent: Tuesday, October 30, 2018 10:12 AM To: Bennett, Ciunas Cc: freebsd-stable@freebsd.org Subject: Re: Possible memory leak in the kernel (contigmalloc) On Tue, Oct 30, 2018 at 09:46:59AM +, Bennett, Ciunas wrote: > Hi, > > I was debugging the issue by viewing the free ques "sysctl > vm.phys_free" and also using "show page" in ddb. The inactive memory > is never being released back into the free que. I thought that when > inactive memory reaches a certain threshold that the kernel will start > reclaiming and move it to the free list? In my program this is not > happening, the program uses free memory (contigmalloc), and then it is > put into the inactive que (contiigfree) when the program frees it. Contigfree() does not release memory into inactive queue. By definition, inactive pages have valid content, which is not possible for the pages freed by contigfree(). contigfree()->kmem_free()->kmem_unback()->vm_page_free(). > This inactive memory is never released by the kernel, and the inactive > que grows until all the memory is in this que. I have attached a xml > sheet that shows the memory usage in the system. Inactive memory is not freed, it makes no sense as far as there is valid content, which is either not recoverable (anon memory or dirty file pages) or high-cost to restore (clean file pages, need to re-read from disk). Inactive is processed by pagedaemon when system has memory deficit, and either inactive pages are written to swap, or written to their file backing storage. I suspect that you have other activities on your system going on, which cause creation of the inactive memory and unrecoverable fragmentation. Note that contigmalloc() tries to do defragmentation to satisfy requests, but this is not always possible. > Ciunas. > > -Original Message- > From: Konstantin Belousov [mailto:kostik...@gmail.com] > Sent: Friday, October 26, 2018 9:13 PM > To: Bennett, Ciunas > Cc: freebsd-stable@freebsd.org > Subject: Re: Possible memory leak in the kernel (contigmalloc) > > On Wed, Oct 24, 2018 at 04:27:52PM +, Bennett, Ciunas wrote: > > Hello, > > > > I have encountered an issue with a kernel application that I have > > written, the issue might be caused by a memory leak in the kernel. > > The application allocates and deallocates contiguous memory using > > contigmalloc() and contigfree(). The application will fail after a > > period of time because there is not enough free contiguous memory > > left. There could be an issue with the freeing of memory when using > > the contigfree() function. > > > > It is unlikely that there is an issue with a leak, but I would be not > surprised if your allocation/free pattern would cause fragmentation on free > lists that results in contigmalloc(9) failures after. > > Look at the vmstat -z/vmstat -m output to see uma and malloc stats. > More interesting for your case can be the output from > sysctl vm.phys_free > which provides information about the free queues and order of free pages on > them. > -- > Intel Research and Development Ireland Limited Registered in Ireland > Registered Office: Collinstown Industrial Park, Leixlip, County > Kildare Registered Number: 308263 > > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. -- Intel Research and Development Ireland Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Possible memory leak in the kernel (contigmalloc)
On Tue, Oct 30, 2018 at 11:15:47AM +, Bennett, Ciunas wrote: > Hi, > The only other activity that is running is the csh script that is inserting > and removing the kernel module. > I am not using the VM for any other purpose but to run this test. > In my tests the correlation between memory allocation and moving to inactive > list can be seen. > I don't think any other process is creating the inactive memory. But it is created. More, since anon private memory is freed (not deactivated) on the process exit, something is accumulating the memory. The other possibility is that the memory is the caching pages from vnodes, but for this buffers must be created and then reclaimed, which would suggest even more activity on the system. > Thanks. > > -Original Message- > From: Konstantin Belousov [mailto:kostik...@gmail.com] > Sent: Tuesday, October 30, 2018 10:12 AM > To: Bennett, Ciunas > Cc: freebsd-stable@freebsd.org > Subject: Re: Possible memory leak in the kernel (contigmalloc) > > On Tue, Oct 30, 2018 at 09:46:59AM +, Bennett, Ciunas wrote: > > Hi, > > > > I was debugging the issue by viewing the free ques "sysctl > > vm.phys_free" and also using "show page" in ddb. The inactive memory > > is never being released back into the free que. I thought that when > > inactive memory reaches a certain threshold that the kernel will start > > reclaiming and move it to the free list? In my program this is not > > happening, the program uses free memory (contigmalloc), and then it is > > put into the inactive que (contiigfree) when the program frees it. > Contigfree() does not release memory into inactive queue. By definition, > inactive pages have valid content, which is not possible for the pages freed > by contigfree(). > contigfree()->kmem_free()->kmem_unback()->vm_page_free(). > > > > This inactive memory is never released by the kernel, and the inactive > > que grows until all the memory is in this que. I have attached a xml > > sheet that shows the memory usage in the system. > Inactive memory is not freed, it makes no sense as far as there is valid > content, which is either not recoverable (anon memory or dirty file > pages) or high-cost to restore (clean file pages, need to re-read from disk). > Inactive is processed by pagedaemon when system has memory deficit, and > either inactive pages are written to swap, or written to their file backing > storage. > > I suspect that you have other activities on your system going on, which cause > creation of the inactive memory and unrecoverable fragmentation. > Note that contigmalloc() tries to do defragmentation to satisfy requests, but > this is not always possible. > > > > Ciunas. > > > > -Original Message- > > From: Konstantin Belousov [mailto:kostik...@gmail.com] > > Sent: Friday, October 26, 2018 9:13 PM > > To: Bennett, Ciunas > > Cc: freebsd-stable@freebsd.org > > Subject: Re: Possible memory leak in the kernel (contigmalloc) > > > > On Wed, Oct 24, 2018 at 04:27:52PM +, Bennett, Ciunas wrote: > > > Hello, > > > > > > I have encountered an issue with a kernel application that I have > > > written, the issue might be caused by a memory leak in the kernel. > > > The application allocates and deallocates contiguous memory using > > > contigmalloc() and contigfree(). The application will fail after a > > > period of time because there is not enough free contiguous memory > > > left. There could be an issue with the freeing of memory when using > > > the contigfree() function. > > > > > > > It is unlikely that there is an issue with a leak, but I would be not > > surprised if your allocation/free pattern would cause fragmentation on free > > lists that results in contigmalloc(9) failures after. > > > > Look at the vmstat -z/vmstat -m output to see uma and malloc stats. > > More interesting for your case can be the output from > > sysctl vm.phys_free > > which provides information about the free queues and order of free pages on > > them. > > -- > > Intel Research and Development Ireland Limited Registered in Ireland > > Registered Office: Collinstown Industrial Park, Leixlip, County > > Kildare Registered Number: 308263 > > > > > > This e-mail and any attachments may contain confidential material for > > the sole use of the intended recipient(s). Any review or distribution > > by others is strictly prohibited. If you are not the intended > > recipient, please contact the sender and delete all copies. > > > -- > Intel Research and Development Ireland Limited > Registered in Ireland > Registered Office: Collinstown Industrial Park, Leixlip, County Kildare > Registered Number: 308263 > > > This e-mail and any attachments may contain confidential material for the sole > use of the intended recipient(s). Any review or distribution by others is > strictly prohibited.
How to boot from GPT partition without "bootme" attribute?
I have disk with GPT scheme and three partitions: p1 - freebsd-boot p2 - freebsd-ufs p3 - freebsd-ufs pmbr is installed on this disk, and gptboot is installed on p1. Both p2 and p3 contains valid FreeBSD installation, with /boot/loader, kernel, and everything. I have attribute "bootme" set on p3, but not on p2. What should I do to boot from p2? I've tried to interrupt gptboot and override its choice: 0:ad(0p3)/boot/loader with 0:ad(0p2)/boot/loader After that loader, loaded from p2, loads kernel from p3 and boots system from p3! If I have MBR, I could override "active" slice in boot0 MBR loader interactively. Is it analogous feature for GPT? -- // Lev Serebryakov signature.asc Description: OpenPGP digital signature
Re: How to boot from GPT partition without "bootme" attribute?
Lev Serebryakov lev at FreeBSD.org wrote on Tue Oct 30 18:37:14 UTC 2018 : > I have disk with GPT scheme and three partitions: > > p1 - freebsd-boot > p2 - freebsd-ufs > p3 - freebsd-ufs > > pmbr is installed on this disk, and gptboot is installed on p1. Both p2 > and p3 contains valid FreeBSD installation, with /boot/loader, kernel, > and everything. > > I have attribute "bootme" set on p3, but not on p2. > > What should I do to boot from p2? > > I've tried to interrupt gptboot and override its choice: > > 0:ad(0p3)/boot/loader > > with > > 0:ad(0p2)/boot/loader > > After that loader, loaded from p2, loads kernel from p3 and boots > system from p3! Are the kernel's on p2 and p3 distinct in an identifiable way? Can you be sure it was not a mix of the p2 kernel and p3 world that booted? I ask because . . . One way to control what world is booted is to adjust the /etc/fstab where the /boot/kernel/kernel is loaded from, having that /etc/fstab to point to a different / area. I do this on small, single board computers to get the kernel from a microsd card but world from a USB storage media device. (I tend to use some form of labeling style reference to avoid device numbering dependencies.) The /etc/fstab where world is from has / agreeing and directs swap partition bindings and such that are appropriate to the specific world. (I've frequently had a world on the microsd card that the initial /etc/fstab can be edited to point to. This gives me a way to boot if there is a problem for the USB media.) I've done such things in gpt and non-gpt contexts. Any chance that that /etc/fstab initially used points to p3's world for / ? There are also things like /boot/loader.conf having something like: vfs.root.mountfrom='ufs:/dev/gpt/MyRoot' to control where things are booted from. > If I have MBR, I could override "active" slice in boot0 MBR loader > interactively. > > Is it analogous feature for GPT? === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: How to boot from GPT partition without "bootme" attribute?
On Tue, 2018-10-30 at 21:37 +0300, Lev Serebryakov wrote: > I have disk with GPT scheme and three partitions: > > p1 - freebsd-boot > p2 - freebsd-ufs > p3 - freebsd-ufs > > pmbr is installed on this disk, and gptboot is installed on p1. Both > p2 > and p3 contains valid FreeBSD installation, with /boot/loader, > kernel, > and everything. > > I have attribute "bootme" set on p3, but not on p2. > > What should I do to boot from p2? > > I've tried to interrupt gptboot and override its choice: > > 0:ad(0p3)/boot/loader > > with > > 0:ad(0p2)/boot/loader > > After that loader, loaded from p2, loads kernel from p3 and boots > system from p3! > > If I have MBR, I could override "active" slice in boot0 MBR loader > interactively. > > Is it analogous feature for GPT? > While loader(8) is loading the kernel, interrupt it to get the console prompt (or ask the menu to give the prompt if you use menus) and do: unload set currdev=disk0p2 boot -- Ian ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: How to boot from GPT partition without "bootme" attribute?
Hello Mark, Wednesday, October 31, 2018, 12:38:07 AM, you wrote: >> After that loader, loaded from p2, loads kernel from p3 and boots >> system from p3! > Are the kernel's on p2 and p3 distinct in an identifiable way? > Can you be sure it was not a mix of the p2 kernel and p3 world > that booted? I ask because . . . Maybe, it is kernel from p2 and userland from p3, but anyway, it is not what I want to achieve. I'll add other kernel IDENT for other partition for check, but I'm sure, that p3 is moutned as /. I have different /etc/fstab on different partitions, of course. If kernel on "bootme" GPT partition will be faulty, it will be disaster, as gptboot could not (or I can not find how to) switch "bootme" attribute of GPT partition. I want to migrate my NanoBSD images from MBR to GPT, but I can not find way to easily boot from "previous" partition when "new" one is messed up. > Any chance that that /etc/fstab initially used points > to p3's world for / ? /etc/fstab on p2 points to p2, and /etc/fstab on p3 points to p3, I'm sure. -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: How to boot from GPT partition without "bootme" attribute?
Hello Ian, Wednesday, October 31, 2018, 1:58:49 AM, you wrote: > While loader(8) is loading the kernel, interrupt it to get the console > prompt (or ask the menu to give the prompt if you use menus) and do: > unload > set currdev=disk0p2 > boot Thnx! Is it possible to pass this information from gptboot(8) to loader(8) somehow? Selecting "active" partition this way looks tedious. IMHO, better solution will be add some interactivity to gptboot(8) akin to super-compact, but mighty boot0(8) -- Best regards, Levmailto:l...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
