Finally, my tiny ceph cluster get 3 monitors, newly added mon.b and mon.c
both running on cubieboard2, which is cheap but still with enough cpu
power(dual-core arm A7 cpu, 1.2G) and memory(1G).
But compare to mon.a which running on an amd64 cpu, both mon.b and mon.c
easily consume too much memory, so I want to know whether this is caused by
memory leak. Below is the output of 'ceph tell mon.a heap stats' and 'ceph
tell mon.c heap stats'(mon.c only start 12hr ago, while mon.a already
running for more than 10 days)
mon.atcmalloc heap stats:------------------------------------------------
MALLOC: 5480160 ( 5.2 MiB) Bytes in use by application
MALLOC: + 28065792 ( 26.8 MiB) Bytes in page heap freelist
MALLOC: + 15242312 ( 14.5 MiB) Bytes in central cache freelist
MALLOC: + 10116608 ( 9.6 MiB) Bytes in transfer cache freelist
MALLOC: + 10432216 ( 9.9 MiB) Bytes in thread cache freelists
MALLOC: + 1667224 ( 1.6 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 71004312 ( 67.7 MiB) Actual memory used (physical + swap)
MALLOC: + 57540608 ( 54.9 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 128544920 ( 122.6 MiB) Virtual address space used
MALLOC:
MALLOC: 4655 Spans in use
MALLOC: 34 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via
madvise()).
Bytes released to the
mon.ctcmalloc heap stats:------------------------------------------------
MALLOC: 175861640 ( 167.7 MiB) Bytes in use by application
MALLOC: + 2220032 ( 2.1 MiB) Bytes in page heap freelist
MALLOC: + 1007560 ( 1.0 MiB) Bytes in central cache freelist
MALLOC: + 2871296 ( 2.7 MiB) Bytes in transfer cache freelist
MALLOC: + 4686000 ( 4.5 MiB) Bytes in thread cache freelists
MALLOC: + 2758880 ( 2.6 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 189405408 ( 180.6 MiB) Actual memory used (physical + swap)
MALLOC: + 0 ( 0.0 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 189405408 ( 180.6 MiB) Virtual address space used
MALLOC:
MALLOC: 3445 Spans in use
MALLOC: 14 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via
madvise()).
Bytes released to the
The ceph versin is 0.67.4, compiled with tcmalloc enabled,
gcc(armv7a-hardfloat-linux-gnueabi-gcc) version 4.7.3 and I also try to
dump heap, but I can not find anything useful, below is a recent dump,
output by command "pprof --text /usr/bin/ceph-mon mon.c.profile.0021.heap".
What extra step should I take to make the dump more meaningful?
Using local file /usr/bin/ceph-mon.
Using local file mon.c.profile.0021.heap.
Total: 149.3 MB
146.2 97.9% 97.9% 146.2 97.9% 00000000b6a7ce7c
1.4 0.9% 98.9% 1.4 0.9% std::basic_string::_Rep::_S_create
??:0
1.4 0.9% 99.8% 1.4 0.9% 00000000002dd794
0.1 0.1% 99.9% 0.1 0.1% 00000000b6a81170
0.1 0.1% 99.9% 0.1 0.1% 00000000b6a80894
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e2ac
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a81410
0.0 0.0% 100.0% 0.0 0.0% 0000000000367450
0.0 0.0% 100.0% 0.0 0.0% 00000000001d4474
0.0 0.0% 100.0% 0.0 0.0% 000000000028847c
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e8d8
0.0 0.0% 100.0% 0.0 0.0% 000000000020c80c
0.0 0.0% 100.0% 0.0 0.0% 000000000028bd20
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a63248
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a83478
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a806f0
0.0 0.0% 100.0% 0.0 0.0% 00000000002eb8b8
0.0 0.0% 100.0% 0.0 0.0% 000000000024efb4
0.0 0.0% 100.0% 0.0 0.0% 000000000027e550
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a77104
0.0 0.0% 100.0% 0.0 0.0% _dl_mcount ??:0
0.0 0.0% 100.0% 0.0 0.0% 00000000003673ec
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7a91c
0.0 0.0% 100.0% 0.0 0.0% 0000000000295e44
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7ee38
0.0 0.0% 100.0% 0.0 0.0% 0000000000283948
0.0 0.0% 100.0% 0.0 0.0% 00000000002a53c4
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7665c
0.0 0.0% 100.0% 0.0 0.0% 00000000002c4590
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e88c
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a8456c
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a76ed4
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a842f0
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a72bd0
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a73cf8
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7100c
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7dec4
0.0 0.0% 100.0% 0.0 0.0% 000000000035e6e8
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a78f68
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7de9c
0.0 0.0% 100.0% 0.0 0.0% 0000000000220528
0.0 0.0% 100.0% 0.0 0.0% 000000000035e7c0
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a6b2f8
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a80a04
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a62e7c
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a66f50
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e958
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a6cfb8
0.0 0.0% 100.0% 0.0 0.0% leveldb::DBImpl::MakeRoomForWrite
(inline) ??:0
0.0 0.0% 100.0% 0.0 0.0% 000000000020797c
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a69de0
0.0 0.0% 100.0% 0.0 0.0% 00000000001d0af0
0.0 0.0% 100.0% 0.0 0.0% 00000000001d0ebc
0.0 0.0% 100.0% 0.0 0.0% 00000000002a0cd4
0.0 0.0% 100.0% 0.0 0.0% 000000000036909c
0.0 0.0% 100.0% 0.0 0.0% 000000000040b02c
0.0 0.0% 100.0% 0.0 0.0% 00000000001d0b68
0.0 0.0% 100.0% 0.0 0.0% 0000000000392fa0
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a64404
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a791b4
0.0 0.0% 100.0% 0.0 0.0% 00000000001d9824
0.0 0.0% 100.0% 0.0 0.0% 0000000000213928
0.0 0.0% 100.0% 0.0 0.0% 00000000002a0cb8
0.0 0.0% 100.0% 0.0 0.0% 00000000002a4fcc
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a725ac
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a66308
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a79068
0.0 0.0% 100.0% 0.0 0.0% 00000000000013b2
0.0 0.0% 100.0% 0.0 0.0% 000000000040b000
0.0 0.0% 100.0% 0.1 0.1% 00000000004d839b
0.0 0.0% 100.0% 0.0 0.0% 0000000000f29887
0.0 0.0% 100.0% 0.0 0.0% 0000000000f3eb6b
0.0 0.0% 100.0% 0.0 0.0% 0000000000f6e1cb
0.0 0.0% 100.0% 0.0 0.0% 0000000000f6edab
0.0 0.0% 100.0% 0.0 0.0% 0000000000f873ab
0.0 0.0% 100.0% 0.0 0.0% 0000000000f8a26b
0.0 0.0% 100.0% 0.0 0.0% 0000000000f8b0cb
0.0 0.0% 100.0% 0.0 0.0% 0000000000f92dab
0.0 0.0% 100.0% 0.0 0.0% 0000000000f9c96b
0.0 0.0% 100.0% 0.0 0.0% 0000000000fa24bf
0.0 0.0% 100.0% 0.0 0.0% 0000000000fadd8b
0.0 0.0% 100.0% 0.0 0.0% 0000000000fb06ab
0.0 0.0% 100.0% 0.0 0.0% 0000000000fb0d0b
0.0 0.0% 100.0% 0.0 0.0% 0000000000fb494b
0.0 0.0% 100.0% 0.0 0.0% 0000000000fbad6b
0.0 0.0% 100.0% 0.0 0.0% 0000000000fbb2cb
0.0 0.0% 100.0% 0.0 0.0% 0000000000fbea6b
0.0 0.0% 100.0% 0.0 0.0% 0000000000fed0eb
0.0 0.0% 100.0% 0.0 0.0% 0000000000fed69b
0.0 0.0% 100.0% 0.0 0.0% 000000000129920b
0.0 0.0% 100.0% 0.0 0.0% 00000000014250eb
0.0 0.0% 100.0% 0.0 0.0% 000000000166cfc5
0.0 0.0% 100.0% 0.1 0.1% 000000000166d711
0.0 0.0% 100.0% 0.0 0.0% 0000000003531d2b
0.0 0.0% 100.0% 0.0 0.0% 000000000379adbb
0.0 0.0% 100.0% 0.0 0.0% 0000000004e888fb
0.0 0.0% 100.0% 0.0 0.0% 0000000004e894ab
0.0 0.0% 100.0% 0.0 0.0% 0000000004e8951b
0.0 0.0% 100.0% 0.0 0.0% 00000000060146d3
0.0 0.0% 100.0% 0.0 0.0% 000000000601482f
0.0 0.0% 100.0% 0.0 0.0% 00000000060fcd2b
0.0 0.0% 100.0% 0.0 0.0% 00000000060fd33b
0.0 0.0% 100.0% 0.0 0.0% 00000000060fdfbb
0.0 0.0% 100.0% 0.0 0.0% 000000000a820749
0.0 0.0% 100.0% 0.0 0.0% 000000000bfb1950
0.0 0.0% 100.0% 0.0 0.0% 00000000b6a43f23
0.0 0.0% 100.0% 0.0 0.0% __clone ??:0
0.0 0.0% 100.0% 0.0 0.0% leveldb::DBImpl::MakeRoomForWrite
??:0
0.0 0.0% 100.0% 0.2 0.1% std::num_put::do_put@806e4 ??:0
0.0 0.0% 100.0% 0.4 0.2% std::num_put::do_put@80b44 ??:0
0.0 0.0% 100.0% 0.1 0.1% std::num_put::do_put@80e00 ??:0
PS: there's a cubietruck board(
http://docs.cubieboard.org/products/start#cubietruck_cubieboard3) released
recently, which features a dual-core arm A7 cpu, 2G RAM, 1Gbit eth port,
and a sata 2.0 port, for $89, maybe suitable for cheap dedicate osd server
with single disk.
--
Best regards,
Changyuan
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
