-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Here's the info you requested as well as the bgp summaries and I have attached a png that shows the memory deterioration over the period discussed.
The green area of the graph is free memory and the blue line is free swap. We upgraded the Quagga package the evening of the 26th, and as the chart shows, It ate all the ram and swap. The system killed bgpd early AM on the 30th, releasing the ram. We restarted quagga at that time and as shown in the graph. It started consuming ram again. These are production systems, and I will need to consider reverting to 99.17 soon if we can not get a grip on this. Thanks for all your help! Stuart Sheldon ACT USA - ---------- show mem ------------- border13# show memory Memory statistics for zebra: System allocator statistics: Total heap allocated: 146 MiB Holding block headers: 0 bytes Used small blocks: 0 bytes Used ordinary blocks: 146 MiB Free small blocks: 32 bytes Free ordinary blocks: 327 KiB Ordinary blocks: 2645 Small blocks: 1 Holding blocks: 0 (see system documentation for 'mallinfo' for meaning) - ----------------------------- Temporary memory : 30 String vector : 4354 Vector : 2280 Vector index : 2280 Link List : 83 Link Node : 139 Thread : 12 Thread master : 1 Thread stats : 8 Thread function name : 19 VTY : 2 Interface : 28 Connected : 59 Buffer : 5 Buffer data : 1 Stream : 8 Stream data : 8 Prefix : 77 Hash : 1 Hash Bucket : 8 Hash Index : 1 Route table : 32 Route node : 762372 Prefix List : 3 Prefix List Entry : 13 Prefix List Str : 3 Route map : 7 Route map name : 7 Route map index : 7 Command desc : 2195 Privilege information : 2 Logging : 1 Work queue : 2 Work queue name string : 1 Host config : 4 - ----------------------------- Router Advertisement Prefix : 10 VRF : 1 VRF name : 1 Nexthop : 412746 RIB : 412746 Static IPv4 route : 10 Static IPv6 route : 6 - ----------------------------- Memory statistics for ospfd: System allocator statistics: Total heap allocated: 800 KiB Holding block headers: 0 bytes Used small blocks: 0 bytes Used ordinary blocks: 745 KiB Free small blocks: 48 bytes Free ordinary blocks: 55 KiB Ordinary blocks: 15 Small blocks: 1 Holding blocks: 0 (see system documentation for 'mallinfo' for meaning) - ----------------------------- Temporary memory : 1 String vector : 6261 Vector : 3113 Vector index : 3113 Link List : 178 Link Node : 245 Thread : 16 Thread master : 1 Thread stats : 20 Thread function name : 32 VTY : 2 Interface : 28 Connected : 59 Buffer : 2 Buffer data : 1 Stream : 4 Stream data : 4 Prefix : 77 Hash : 1 Hash Bucket : 20 Hash Index : 1 Route table : 159 Route node : 294 Access List : 1 Access List Str : 1 Access Filter : 4 Prefix List : 1 Prefix List Entry : 5 Prefix List Str : 1 Route map : 7 Route map name : 7 Route map index : 7 Command desc : 3138 Privilege information : 2 Logging : 1 Zclient : 1 Host config : 4 - ----------------------------- OSPF top : 1 OSPF area : 1 OSPF network : 1 OSPF interface : 1 OSPF neighbor : 2 OSPF route : 39 OSPF LSA : 63 OSPF LSA data : 63 OSPF LSDB : 2 OSPF packet : 1 OSPF FIFO queue : 1 OSPF path : 39 OSPF ext. info : 26 OSPF if info : 28 OSPF if params : 28 - ----------------------------- Memory statistics for ospf6d: System allocator statistics: Total heap allocated: 660 KiB Holding block headers: 0 bytes Used small blocks: 0 bytes Used ordinary blocks: 590 KiB Free small blocks: 48 bytes Free ordinary blocks: 70 KiB Ordinary blocks: 114 Small blocks: 1 Holding blocks: 0 (see system documentation for 'mallinfo' for meaning) - ----------------------------- Temporary memory : 1 String vector : 4792 Vector : 2287 Vector index : 2287 Link List : 35 Link Node : 92 Thread : 86 Thread master : 1 Thread stats : 30 Thread function name : 106 VTY : 2 Interface : 28 Connected : 59 Buffer : 2 Buffer data : 1 Stream : 2 Stream data : 2 Prefix : 77 Hash : 1 Hash Bucket : 30 Hash Index : 1 Route table : 25 Route node : 644 Prefix List : 2 Prefix List Entry : 8 Prefix List Str : 2 Route map : 7 Route map name : 7 Route map index : 7 Command desc : 2402 Privilege information : 2 Logging : 1 Zclient : 1 Host config : 2 - ----------------------------- OSPF6 top : 1 OSPF6 area : 1 OSPF6 interface : 1 OSPF6 neighbor : 1 OSPF6 route : 54 OSPF6 message : 2 OSPF6 LSA : 134 OSPF6 LSA database : 15 OSPF6 vertex : 3 OSPF6 ext. info : 22 - ----------------------------- Memory statistics for bgpd: System allocator statistics: Total heap allocated: 1494 MiB Holding block headers: 260 KiB Used small blocks: 0 bytes Used ordinary blocks: 1424 MiB Free small blocks: 32 bytes Free ordinary blocks: 70 MiB Ordinary blocks: 187317 Small blocks: 1 Holding blocks: 1 (see system documentation for 'mallinfo' for meaning) - ----------------------------- Temporary memory : 1 String vector : 14526 Vector : 7222 Vector index : 7222 Link List : 36 Link Node : 97 Thread : 17989 Thread master : 1 Thread stats : 15 Thread function name : 6007 VTY : 2 Interface : 28 Connected : 59 Buffer : 3 Buffer data : 1 Stream : 17 Stream data : 17 Stream FIFO : 6 Prefix : 77 Hash : 55 Hash Bucket : 282248 Hash Index : 55 Access List : 1 Access List Str : 1 Access Filter : 4 Prefix List : 6 Prefix List Entry : 28 Prefix List Str : 6 Route map : 7 Route map name : 7 Route map index : 7 Route map rule : 7 Route map rule str : 7 Route map compiled : 6 Command desc : 7274 Socket union : 10 Privilege information : 2 Logging : 1 Zclient : 2 Work queue : 2 Work queue name string : 2 Host config : 5 - ----------------------------- Nexthop : 6 - ----------------------------- BGP instance : 1 BGP listen socket details : 2 BGP peer : 6 BGP peer hostname : 6 Peer description : 5 BGP attribute : 216434 BGP extra attributes : 12978759 BGP aspath : 65800 BGP aspath seg : 65876 BGP aspath segment data : 65876 BGP aspath str : 65800 - ----------------------------- BGP table : 31 BGP node : 762375 BGP route : 825251 BGP ancillary route info : 412624 BGP connected : 34 BGP static : 6 BGP synchronise : 48 BGP adj in : 825286 BGP adj out : 25 - ----------------------------- BGP AS list : 1 BGP AS filter : 2 BGP AS filter str : 2 - ----------------------------- community : 1 community val : 1 - ----------------------------- community-list handler : 1 - ----------------------------- BGP nexthop : 6 BGP regexp : 2 - --------- show bgp mem ------------ border13# show bgp memory 762394 RIB nodes, using 70 MiB of memory 825273 BGP routes, using 44 MiB of memory 412638 BGP route ancillaries, using 9671 KiB of memory 6 Static routes, using 288 bytes of memory 825308 Adj-In entries, using 25 MiB of memory 24 Adj-Out entries, using 960 bytes of memory 6 Nexthop cache entries, using 144 bytes of memory 216184 BGP attributes, using 12 MiB of memory 13062993 BGP extra attributes, using 1096 MiB of memory 66044 BGP AS-PATH entries, using 1548 KiB of memory 66122 BGP AS-PATH segments, using 1550 KiB of memory 6 peers, using 27 KiB of memory 55 hash tables, using 2200 bytes of memory 282243 hash buckets, using 6615 KiB of memory 2 compiled regexes, using 128 bytes of memory - -------- show (ip|ipv6) bgp summary -------- Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd nnn.nnn.nn.nn 4 ebgp 95361 502 0 0 0 08:18:20 403934 nnn.nnn.nnn.nn 4 ibgp 502 505 0 0 0 08:18:17 1 nnn.nnn.nnn.nn 4 ibgp 503 505 0 0 0 08:18:17 1 nnn.nnn.nnn.nn 4 ibgp 3348339 34476 0 0 0 08:18:18 403935 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd nnn.nnn.nnn.nn 4 ibgp 505 507 0 0 0 08:21:12 2 nnn.nnn.nnn.nn 4 ibgp 506 507 0 0 0 08:21:12 2 nnn.nnn.nnn.nn 4 ibgp 3366472 34578 0 0 0 08:21:13 8698 nnnn:n:n:n::n 4 ebgp 2445075 506 0 0 0 08:21:15 8695 - -------- End ------------------------------- On 04/30/2012 09:09 AM, Christian Hammers wrote: > tags 670940 + upstream > forwarded 670940 https://bugzilla.quagga.net/show_bug.cgi?id=722 > stop > > > Hello > > Thanks for the report, I filed an upstream bug report: > https://bugzilla.quagga.net/show_bug.cgi?id=722 > > I could not add your email to the cc list so if you don't want to wait for > me to relay the answers, please register at their bugzilla and add yourself > to the Cc list. > > In the meantime, please provide the output of the bgpd console commands > show memory > show bgp memory > (and whatever seems usefull as well) > It also could not hurt to install the new quagga-dbg package to > get some more details output if it crashes and collect the output of > "ipcs -a -p PID" and "memstat -w -p PID". If you know "valgrind", you might > try that too. > > If you can post your bgpd.conf (without passwords and other sensitive > information), I could try to reproduce the problem in a chroot environment. > > bye, > > -christian- > > > > > Begin forwarded message: > > Date: Mon, 30 Apr 2012 08:08:53 -0700 > From: Stuart Sheldon <[email protected]> > To: Debian Bug Tracking System <[email protected]> > Subject: Bug#670940: quagga: Quagga bgpd memory leak. Consumes all Ram and > Swap over time. > > > Package: quagga > Version: 0.99.20.1-0+squeeze1 > Severity: important > > After upgrading to latest security release of quagga on our two border > routers, the bgpd daemon began to suck up all the available memory. This took > place at a rate of about 4.5G per 24hr period. This continued until it > consumed all the ram (8G) and all the swap (4G) which caused the kernel to > remove the bgpd process. > > This appeared to occur at the same rate on both routers. Although these > routers both receive full route loads from upstream for both ipv4 and ipv6, > the previous version averaged right around 1G of memory use total for the > routers. > > routers are identical hardware and software. > > Please advise if you require configuration info. > > Stuart Sheldon > ACT USA > > -- System Information: > Debian Release: 6.0.4 > APT prefers stable-updates > APT policy: (500, 'stable-updates'), (500, 'stable') > Architecture: amd64 (x86_64) > > Kernel: Linux 2.6.32-5-amd64 (SMP w/2 CPU cores) > Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) > Shell: /bin/sh linked to /bin/dash > > Versions of packages quagga depends on: > ii adduser 3.112+nmu2 add and remove users and groups > ii debconf [debconf-2.0] 1.5.36.1 Debian configuration management > sy > ii iproute 20100519-3 networking and traffic control > too > ii libc6 2.11.3-3 Embedded GNU C Library: Shared > lib > ii libcap2 1:2.19-3 support for getting/setting > POSIX. > ii libpam0g 1.1.1-6.1+squeeze1 Pluggable Authentication Modules > l > ii libreadline6 6.1-3 GNU readline and history > libraries > ii logrotate 3.7.8-6 Log rotation utility > > quagga recommends no packages. > > Versions of packages quagga suggests: > ii snmpd 5.4.3~dfsg-2 SNMP (Simple Network Management > Pr > > -- Configuration Files: > /etc/quagga/daemons changed: > zebra=yes > bgpd=yes > ospfd=yes > ospf6d=yes > ripd=no > ripngd=no > isisd=no > > > -- debconf information: > * quagga/really_stop: true > > > > - -- Pancho was a bandit boys, his horse was fast as polished steel He wore his gun outside his pants for all the honest world to feel. Pancho met his match you know, on the deserts down in Mexico Nobody heard his dying words, ah but that's the way it goes. -- Willie Nelson and Merle Haggart "Pancho and Lefty" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBCAAGBQJPnsfdAAoJEFKVLITDJSGS8v0QAIqXN5/Nk+O2UT1VmUaz+bJl 5lio4T8fO+4nRsYJALFWTiOVCwDKSbH3sz8kQrKqDk4dXbpaoYHPrtjtthy+Dxrm 6ZfJSNOia0x1t3okWei4mPkU+rHW7bE2S3ye3NB/qE8OvwpPffirrgBOspotSd80 0YgG0nGRRcgsjlyYNSsqZKr6yIcDf+sjq8z6GewkRtvbMd0iwXnbaH5igo8Cc5P9 FNGh7zFJ8DkF3mCz3efg56M+6OFaGd9HZVzaFG8Dy5jVpa7ul1dyNqxZEgbc9DFA ZFGMLeS0OvJ9gp8u/izJEPeUVJHAdYWqEGX3nuay4Fg4MxKjolsG7aOCLLtKv40G u4HZRCfKwQK6AFKokTuw/06w3Spr5mBBqMAe4JyiJwmxKURaxeT4Zvf7p+6icAsK dfhdM8nyBRHnIaCX4Drc1T5PTizAAu2W9lYKvNEiQtifF9DZIzYjvYG2YQz/9lYp ho4tGj2Us588s7H/BJ7cWjf1MpBLcFWi5aOB78CD48I8NY/lxPTa+tgSRQ/z1DWS SM2h4rsapHqAbmcKWHTv52jHXaejn70lIDJ0AfH98r6rb61AVsIZpUmshECuoyXp 98t3U74ecX3enGyFiKZkrU9JihD/U/N4yx0E1y97pOtDGgXwOf1gvbnQBxKci7SZ KtBRM4SseE1hXkZHo1Tt =33KO -----END PGP SIGNATURE-----
<<attachment: QuaggaMemoryLeak.png>>
