Hi Fengguang,
I run tests and attached the results. The line below I guess shows the data-1
page caches.
0x000000080000006c 6584051 25718
__RU_lA___________________P________ referenced,uptodate,lru,active,private
Metin
________________________________
From: Jaegeuk Hanse <jaegeuk.ha...@gmail.com>
To: Fengguang Wu <fengguang...@intel.com>
Cc: metin d <met...@yahoo.com>; Jan Kara <j...@suse.cz>;
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>;
"linux...@kvack.org" <linux...@kvack.org>
Sent: Wednesday, November 21, 2012 11:42 AM
Subject: Re: Problem in Page Cache Replacement
On 11/21/2012 05:02 PM, Fengguang Wu wrote:
> On Wed, Nov 21, 2012 at 04:34:40PM +0800, Jaegeuk Hanse wrote:
>> Cc Fengguang Wu.
>>
>> On 11/21/2012 04:13 PM, metin d wrote:
>>>> Curious. Added linux-mm list to CC to catch more attention. If you run
>>>> echo 1 >/proc/sys/vm/drop_caches does it evict data-1 pages from memory?
>>> I'm guessing it'd evict the entries, but am wondering if we could run any
>>> more diagnostics before trying this.
>>>
>>> We regularly use a setup where we have two databases; one gets used
>>> frequently and the other one about once a month. It seems like the memory
>>> manager keeps unused pages in memory at the expense of frequently used
>>> database's performance.
>>> My understanding was that under memory pressure from heavily
>>> accessed pages, unused pages would eventually get evicted. Is there
>>> anything else we can try on this host to understand why this is
>>> happening?
> We may debug it this way.
>
> 1) run 'fadvise data-2 0 0 dontneed' to drop data-2 cached pages
> (please double check via /proc/vmstat whether it does the expected work)
>
> 2) run 'page-types -r' with root, to view the page status for the
> remaining pages of data-1
>
> The fadvise tool comes from Andrew Morton's ext3-tools. (source code attached)
> Please compile them with options "-Dlinux -I. -D_GNU_SOURCE
> -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE"
>
> page-types can be found in the kernel source tree tools/vm/page-types.c
>
> Sorry that sounds a bit twisted.. I do have a patch to directly dump
> page cache status of a user specified file, however it's not
> upstreamed yet.
Hi Fengguang,
Thanks for you detail steps, I think metin can have a try.
flags page-count MB symbolic-flags long-symbolic-flags
0x0000000000000000 607699 2373
___________________________________
0x0000000100000000 343227 1340
_______________________r___________ reserved
But I have some questions of the print of page-type:
Is 2373MB here mean total memory in used include page cache? I don't
think so.
Which kind of pages will be marked reserved?
Which line of long-symbolic-flags is for page cache?
Regards,
Jaegeuk
>
> Thanks,
> Fengguang
>
>>> On Tue 20-11-12 09:42:42, metin d wrote:
>>>> I have two PostgreSQL databases named data-1 and data-2 that sit on the
>>>> same machine. Both databases keep 40 GB of data, and the total memory
>>>> available on the machine is 68GB.
>>>>
>>>> I started data-1 and data-2, and ran several queries to go over all their
>>>> data. Then, I shut down data-1 and kept issuing queries against data-2.
>>>> For some reason, the OS still holds on to large parts of data-1's pages
>>>> in its page cache, and reserves about 35 GB of RAM to data-2's files. As
>>>> a result, my queries on data-2 keep hitting disk.
>>>>
>>>> I'm checking page cache usage with fincore. When I run a table scan query
>>>> against data-2, I see that data-2's pages get evicted and put back into
>>>> the cache in a round-robin manner. Nothing happens to data-1's pages,
>>>> although they haven't been touched for days.
>>>>
>>>> Does anybody know why data-1's pages aren't evicted from the page cache?
>>>> I'm open to all kind of suggestions you think it might relate to problem.
>>> Curious. Added linux-mm list to CC to catch more attention. If you run
>>> echo 1 >/proc/sys/vm/drop_caches
>>> does it evict data-1 pages from memory?
>>>
>>>> This is an EC2 m2.4xlarge instance on Amazon with 68 GB of RAM and no
>>>> swap space. The kernel version is:
>>>>
>>>> $ uname -r
>>>> 3.2.28-45.62.amzn1.x86_64
>>>> Edit:
>>>>
>>>> and it seems that I use one NUMA instance, if you think that it can a
>>>> problem.
>>>>
>>>> $ numactl --hardware
>>>> available: 1 nodes (0)
>>>> node 0 cpus: 0 1 2 3 4 5 6 7
>>>> node 0 size: 70007 MB
>>>> node 0 free: 360 MB
>>>> node distances:
>>>> node 0
>>>> 0: 10
flags page-count MB symbolic-flags
long-symbolic-flags
0x0000000000000000 5508317 21516
___________________________________
0x0000000100000000 335993 1312
_______________________r___________ reserved
0x0000002100000000 35634 139
_______________________r____O______ reserved,owner_private
0x0000000000010000 45069 176
________________T__________________ compound_tail
0x0000002000000000 1516 5
____________________________O______ owner_private
0x0000000800000004 1 0
__R_______________________P________ referenced,private
0x0000000000008000 10 0
_______________H___________________ compound_head
0x0000000000000004 1 0
__R________________________________ referenced
0x0000000800000024 166 0
__R__l____________________P________ referenced,lru,private
0x0000000400000028 295 1
___U_l___________________d_________ uptodate,lru,mappedtodisk
0x0001000400000028 3 0
___U_l___________________d_____I___ uptodate,lru,mappedtodisk,readahead
0x0000000000000028 1 0
___U_l_____________________________ uptodate,lru
0x000000040000002c 262144 1024
__RU_l___________________d_________ referenced,uptodate,lru,mappedtodisk
0x000000080000002c 5 0
__RU_l____________________P________ referenced,uptodate,lru,private
0x000000000000403c 185 0
__RUDl________b____________________
referenced,uptodate,dirty,lru,swapbacked
0x0000000800000060 163 0
_____lA___________________P________ lru,active,private
0x0000000800000064 36739 143
__R__lA___________________P________ referenced,lru,active,private
0x0000000400000068 527810 2061
___U_lA__________________d_________ uptodate,lru,active,mappedtodisk
0x0000000800000068 576 2
___U_lA___________________P________ uptodate,lru,active,private
0x0000000c00000068 116 0
___U_lA__________________dP________
uptodate,lru,active,mappedtodisk,private
0x000000080000006c 6584051 25718
__RU_lA___________________P________
referenced,uptodate,lru,active,private
0x000000040000006c 1302211 5086
__RU_lA__________________d_________
referenced,uptodate,lru,active,mappedtodisk
0x0000000c0000006c 431 1
__RU_lA__________________dP________
referenced,uptodate,lru,active,mappedtodisk,private
0x000000000000006c 128 0
__RU_lA____________________________ referenced,uptodate,lru,active
0x0000000800000074 2 0
__R_DlA___________________P________ referenced,dirty,lru,active,private
0x0000000000004078 56 0
___UDlA_______b____________________ uptodate,dirty,lru,active,swapbacked
0x000000000000407c 122 0
__RUDlA_______b____________________
referenced,uptodate,dirty,lru,active,swapbacked
0x000000080000007c 1 0
__RUDlA___________________P________
referenced,uptodate,dirty,lru,active,private
0x0000000000008080 14495 56
_______S_______H___________________ slab,compound_head
0x0000000000000080 250498 978
_______S___________________________ slab
0x0000000000000400 2990908 11683
__________B________________________ buddy
0x0000000000000800 16 0
___________M_______________________ mmap
0x0000000100000804 1 0
__R________M___________r___________ referenced,mmap,reserved
0x000000060004082c 391 1
__RU_l_____M______u_____md_________
referenced,uptodate,lru,mmap,unevictable,mlocked,mappedtodisk
0x0000000a0004082c 321 1
__RU_l_____M______u_____m_P________
referenced,uptodate,lru,mmap,unevictable,mlocked,private
0x0000000000004838 8450 33
___UDl_____M__b____________________ uptodate,dirty,lru,mmap,swapbacked
0x000000000000483c 2045 7
__RUDl_____M__b____________________
referenced,uptodate,dirty,lru,mmap,swapbacked
0x0000000800000868 19 0
___U_lA____M______________P________ uptodate,lru,active,mmap,private
0x0000000400000868 5 0
___U_lA____M_____________d_________ uptodate,lru,active,mmap,mappedtodisk
0x000000040000086c 1891 7
__RU_lA____M_____________d_________
referenced,uptodate,lru,active,mmap,mappedtodisk
0x000000080000086c 126 0
__RU_lA____M______________P________
referenced,uptodate,lru,active,mmap,private
0x0000000000004878 85 0
___UDlA____M__b____________________
uptodate,dirty,lru,active,mmap,swapbacked
0x000000000000487c 2263 8
__RUDlA____M__b____________________
referenced,uptodate,dirty,lru,active,mmap,swapbacked
0x0000000000005008 13 0
___U________a_b____________________ uptodate,anonymous,swapbacked
0x0000000000005808 16 0
___U_______Ma_b____________________ uptodate,mmap,anonymous,swapbacked
0x0000000200045828 8 0
___U_l_____Ma_b___u_____m__________
uptodate,lru,mmap,anonymous,swapbacked,unevictable,mlocked
0x000000020004582c 651 2
__RU_l_____Ma_b___u_____m__________
referenced,uptodate,lru,mmap,anonymous,swapbacked,unevictable,mlocked
0x0000000000005868 8058 31
___U_lA____Ma_b____________________
uptodate,lru,active,mmap,anonymous,swapbacked
0x000000000000586c 42 0
__RU_lA____Ma_b____________________
referenced,uptodate,lru,active,mmap,anonymous,swapbacked
total 17922048 70008
flags page-count MB symbolic-flags
long-symbolic-flags
0x0000000000000000 121628 475
___________________________________
0x0000000100000000 335993 1312
_______________________r___________ reserved
0x0000002100000000 35634 139
_______________________r____O______ reserved,owner_private
0x0000000000010000 45429 177
________________T__________________ compound_tail
0x0000002000000000 1389 5
____________________________O______ owner_private
0x0000000400000001 6 0
L________________________d_________ locked,mappedtodisk
0x0000000000008000 10 0
_______________H___________________ compound_head
0x0000000000000004 1 0
__R________________________________ referenced
0x0000000400000021 64 0
L____l___________________d_________ locked,lru,mappedtodisk
0x0001000400000021 1 0
L____l___________________d_____I___ locked,lru,mappedtodisk,readahead
0x0000000800000024 171 0
__R__l____________________P________ referenced,lru,private
0x0000000400000028 4093 15
___U_l___________________d_________ uptodate,lru,mappedtodisk
0x0001000400000028 59 0
___U_l___________________d_____I___ uptodate,lru,mappedtodisk,readahead
0x0000000000000028 1 0
___U_l_____________________________ uptodate,lru
0x000000040000002c 8598032 33586
__RU_l___________________d_________ referenced,uptodate,lru,mappedtodisk
0x000000080000002c 10 0
__RU_l____________________P________ referenced,uptodate,lru,private
0x000000000000403c 185 0
__RUDl________b____________________
referenced,uptodate,dirty,lru,swapbacked
0x0000000800000060 163 0
_____lA___________________P________ lru,active,private
0x0000000800000064 36741 143
__R__lA___________________P________ referenced,lru,active,private
0x0000000400000068 527834 2061
___U_lA__________________d_________ uptodate,lru,active,mappedtodisk
0x0000000800000068 695 2
___U_lA___________________P________ uptodate,lru,active,private
0x0000000c00000068 116 0
___U_lA__________________dP________
uptodate,lru,active,mappedtodisk,private
0x000000080000006c 6584066 25719
__RU_lA___________________P________
referenced,uptodate,lru,active,private
0x000000040000006c 1325273 5176
__RU_lA__________________d_________
referenced,uptodate,lru,active,mappedtodisk
0x0000000c0000006c 431 1
__RU_lA__________________dP________
referenced,uptodate,lru,active,mappedtodisk,private
0x000000000000006c 128 0
__RU_lA____________________________ referenced,uptodate,lru,active
0x0000000000004078 56 0
___UDlA_______b____________________ uptodate,dirty,lru,active,swapbacked
0x000000000000407c 122 0
__RUDlA_______b____________________
referenced,uptodate,dirty,lru,active,swapbacked
0x000000080000007c 1 0
__RUDlA___________________P________
referenced,uptodate,dirty,lru,active,private
0x0000000000008080 14571 56
_______S_______H___________________ slab,compound_head
0x0000000000000080 250546 978
_______S___________________________ slab
0x0000000000000400 14701 57
__________B________________________ buddy
0x0000000000000800 16 0
___________M_______________________ mmap
0x0000000100000804 1 0
__R________M___________r___________ referenced,mmap,reserved
0x000000060004082c 391 1
__RU_l_____M______u_____md_________
referenced,uptodate,lru,mmap,unevictable,mlocked,mappedtodisk
0x0000000a0004082c 321 1
__RU_l_____M______u_____m_P________
referenced,uptodate,lru,mmap,unevictable,mlocked,private
0x0000000000004838 8385 32
___UDl_____M__b____________________ uptodate,dirty,lru,mmap,swapbacked
0x000000000000483c 2045 7
__RUDl_____M__b____________________
referenced,uptodate,dirty,lru,mmap,swapbacked
0x0000000800000868 19 0
___U_lA____M______________P________ uptodate,lru,active,mmap,private
0x0000000400000868 5 0
___U_lA____M_____________d_________ uptodate,lru,active,mmap,mappedtodisk
0x000000040000086c 1891 7
__RU_lA____M_____________d_________
referenced,uptodate,lru,active,mmap,mappedtodisk
0x000000080000086c 126 0
__RU_lA____M______________P________
referenced,uptodate,lru,active,mmap,private
0x0000000000004878 85 0
___UDlA____M__b____________________
uptodate,dirty,lru,active,mmap,swapbacked
0x000000000000487c 2263 8
__RUDlA____M__b____________________
referenced,uptodate,dirty,lru,active,mmap,swapbacked
0x0000000000005008 4 0
___U________a_b____________________ uptodate,anonymous,swapbacked
0x0000000000005808 25 0
___U_______Ma_b____________________ uptodate,mmap,anonymous,swapbacked
0x0000000200045828 8 0
___U_l_____Ma_b___u_____m__________
uptodate,lru,mmap,anonymous,swapbacked,unevictable,mlocked
0x000000020004582c 651 2
__RU_l_____Ma_b___u_____m__________
referenced,uptodate,lru,mmap,anonymous,swapbacked,unevictable,mlocked
0x0000000000005868 7623 29
___U_lA____Ma_b____________________
uptodate,lru,active,mmap,anonymous,swapbacked
0x000000000000586c 39 0
__RU_lA____Ma_b____________________
referenced,uptodate,lru,active,mmap,anonymous,swapbacked
total 17922048 70008
