Hello All, Amos or Alexander can you help me to compile a working squid 5.6 version on Ubuntu 22.04 without this "memory leak". I would need a patch here. As you can see in the mail history below, between me and Eliezer, I have a memory leak problem with Ubuntu 22.04 and the squid 5.2 from the ubuntu repository.
--- With kind regards Ronny ---------- Forwarded message --------- Von: <ngtech1...@gmail.com> Date: Di., 19. Juli 2022 um 12:13 Uhr Subject: RE: [squid-users] Sqid uses all RAM / killed by OOM To: Ronny Preiss <ronny.pre...@gmail.com> No Problem. ---- Eliezer Croitoru NgTech, Tech Support Mobile: +972-5-28704261 Email: ngtech1...@gmail.com Web: https://ngtech.co.il/ My-Tube: https://tube.ngtech.co.il/ *From:* Ronny Preiss <ronny.pre...@gmail.com> *Sent:* Tuesday, 19 July 2022 12:42 *To:* ngtech1...@gmail.com *Subject:* Re: [squid-users] Sqid uses all RAM / killed by OOM Hi Eliezer, Is it ok for you if I use this mail as reference when writing to Amos or Alex via the squid-user mailing list? --- With kind regards Ronny Am Di., 19. Juli 2022 um 11:34 Uhr schrieb <ngtech1...@gmail.com>: There was a memory leak in 5.x which was fixed later on. Currently there is no patch for 5.6 so that you will be able to run 5.6 on Ubuntu 22.04. It will take time for this patch to get into squid 5.7 or 5.8. My recommendation for now is to use either 20.04 or another Linux version. I am using Oracle Enterprise Linux 8 which is free and is very stable. If you want to resolve the issue on Ubuntu 22.04 you will need to compile and patch squid yourself. Try to post again in the squid-user mailing list and maybe Amos or Alex be able to offer you some guidance. All The Bests, Eliezer ---- Eliezer Croitoru NgTech, Tech Support Mobile: +972-5-28704261 Email: ngtech1...@gmail.com Web: https://ngtech.co.il/ My-Tube: https://tube.ngtech.co.il/ *From:* Ronny Preiss <ronny.pre...@gmail.com> *Sent:* Tuesday, 19 July 2022 10:10 *To:* ngtech1...@gmail.com *Subject:* Re: [squid-users] Sqid uses all RAM / killed by OOM Hey Eliezer, >First you better use 16k FD just to be in the safe side. I have increased the value to 16384 and increased the RAM to 8GB. Furthermore I have uploaded logs again. Enclosed the link: https://1drv.ms/u/s!Aj-l0jAnmUwxrzk9LShqiEAORyEg (OneDrive) >cache deny all >Should do the trick. It looks like the RAM and SWAP are running full despite 'cache deny all'. top - 09:07:23 up 21:08, 3 users, load average: 0.12, 0.13, 0.09 Tasks: 244 total, 1 running, 243 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.6 us, 3.5 sy, 0.0 ni, 91.9 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 s MiB Mem : 7951.2 total, 119.6 free, 7688.0 used, 143.6 buff/cache MiB Swap: 3925.0 total, 649.2 free, 3275.8 used. 53.3 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ 1769 proxy 20 0 13.3g 7.2g 8988 S 22.0 92.9 40:03.05 1770 proxy 20 0 6060 672 672 S 0.0 0.0 0:17.24 1771 proxy 20 0 6052 688 652 S 0.0 0.0 0:00.08 2172 proxy 20 0 6172 3040 2768 S 0.0 0.0 0:01.47 > If you ask me it’s better to use 22.04 and or other versions of linux until a fix will be found >(if possible….) I have a requirement to upgrade from Ubuntu 20.04.4 to Ubuntu 22.04. >Please let me know what is your: >squid -v Squid Cache: Version 5.2 Service Name: squid Ubuntu linux configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' 'BUILDCXXFLAGS=-g -O2 -ffile-prefix-map=/build/squid-DQ03Be/squid-5.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now ' 'BUILDCXX=g++' '--with-build-environment=default' '--enable-build-info=Ubuntu linux' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid' '--mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,SMB_LM' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group' '--enable-security-cert-validators=fake' '--enable-storeid-rewrite-helpers=file' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/run/squid.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' '--with-systemd' '--with-gnutls' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/build/squid-DQ03Be/squid-5.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now ' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/build/squid-DQ03Be/squid-5.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security' root@01-proxyha03:~# Here also again my new adjusted current configuration: root@01-proxyha01:~# grep -v ^# /etc/squid/squid.conf | grep . acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN) acl localnet src 10.172.xxx.0/18 # RFC 1918 local private network (LAN) acl localnet src 100.64.0.0/10 # RFC 6598 shared address space (CGN) acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly plugged) machines acl localnet src 172.16.0.0/12 # RFC 1918 local private network (LAN) acl localnet src 192.168.0.0/16 # RFC 1918 local private network (LAN) acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http http_access deny !Safe_ports http_access allow localnet Safe_ports http_access allow localhost manager http_access deny manager http_access allow localhost http_access deny all include /etc/squid/conf.d/* cache_peer 10.210.xxx.xxx parent 8080 0 no-query default cache deny all cache_dir ufs /var/spool/squid 3000 16 256 coredump_dir /var/spool/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern \/(Packages|Sources)(|\.bz2|\.gz|\.xz)$ 0 0% 0 refresh-ims refresh_pattern \/Release(|\.gpg)$ 0 0% 0 refresh-ims refresh_pattern \/InRelease$ 0 0% 0 refresh-ims refresh_pattern \/(Translation-.*)(|\.bz2|\.gz|\.xz)$ 0 0% 0 refresh-ims refresh_pattern . 0 20% 4320 never_direct allow all max_filedescriptors 16384 dns_nameservers 10.244.xxx.xxx root@01-proxyha01:~# --- With kind regards Ronny Am Mo., 18. Juli 2022 um 09:54 Uhr schrieb <ngtech1...@gmail.com>: Hey Ronny, First you better use 16k FD just to be in the safe side. Seconds I have seen that you need more then 4GB of ram ie at least 8GB. Another thing to try to see if it helps is to disable the cache to verify that this is not what causing the issue. cache deny all Should do the trick. I didn’t manage to build Squid 5.6 on Ubuntu 22.04 yet and will test it later on. If you ask me it’s better to use 22.04 and or other versions of linux until a fix will be found (if possible….) Please let me know what is your: squid -v output so I can try to what are the right configure options for your use case. Eliezer ---- Eliezer Croitoru NgTech, Tech Support Mobile: +972-5-28704261 Email: ngtech1...@gmail.com Web: https://ngtech.co.il/ My-Tube: https://tube.ngtech.co.il/ *From:* Ronny Preiss <ronny.pre...@gmail.com> *Sent:* Monday, 18 July 2022 8:35 *To:* ngtech1...@gmail.com *Subject:* Re: [squid-users] Sqid uses all RAM / killed by OOM Hey Eliezer, I hope I can answer everything so far. > Second, use Squid 5.6 and not 5.2 How can I use Squid 5.6 in Ubuntu 22.04? I've installed squid 5.2 with "apt install squid" > The issue is not necessarily because of the Squid version but other things. On the Ubuntu 22.04 server only squid is installed so far. > First upgrade to 5.6 and then we need the top snapshot. How can I upgrade squid to 5.6? Compile it myself? If yes, which configure options are needed? Do you have any hints I should follow? > I am using it like this: > /usr/local/bin/dump-cache-mgr-to-file.sh 2>&1 | tee “cachemg-data$(date +%Y-%m-%d_%H-%M-%S)” > cat > Try to take couple dumps with 4.1 and 5.6 (if you want 5.2 also) and upload the snapshots to somewhere in a zip file. Download Link: https://1drv.ms/u/s!Aj-l0jAnmUwxrzjzqZwAF5LwTMEw?e=amk6om (OneDrive) >A machine that should handle “max_filedescriptors 40960” ie above 16k should have more RAM ie 8-16 GB as a starter. I watched the value with the squidclient mgr:info for a while and adjusted it to 12288. --- With kind regards Ronny Am Mo., 11. Juli 2022 um 13:31 Uhr schrieb <ngtech1...@gmail.com>: Hey Ronny, First to make the data more readable use a top snapshot to illustrate the memory usage. Second, use Squid 5.6 and not 5.2 The issue is not necessarily because of the Squid version but other things. We should narrow down the issues as any other Squid issue. First upgrade to 5.6 and then we need the top snapshot. After that we need snapshots of cache-manager pages. To dump snapshots of the cache manager pages you can use the next script: https://gist.githubusercontent.com/elico/8790bdc835d8e9ecbc57e72fc31effc0/raw/60d140b0e772fa4f418779bfc27e4804a345ce23/dump-cache-mgr-to-file.sh I am using it like this: /usr/local/bin/dump-cache-mgr-to-file.sh 2>&1 | tee “cachemg-data$(date +%Y-%m-%d_%H-%M-%S)” cat Try to take couple dumps with 4.1 and 5.6 (if you want 5.2 also) and upload the snapshots to somewhere in a zip file. Just take into account that these snapshots may contains confidential data so you are advised to review them and deduct specific parts of them before making them public. Just an advice, A machine that should handle “max_filedescriptors 40960” ie above 16k should have more RAM ie 8-16 GB as a starter. All The Bests, Eliezer ---- Eliezer Croitoru NgTech, Tech Support Mobile: +972-5-28704261 Email: ngtech1...@gmail.com Web: https://ngtech.co.il/ My-Tube: https://tube.ngtech.co.il/ *From:* squid-users <squid-users-boun...@lists.squid-cache.org> *On Behalf Of *Ronny Preiss *Sent:* Monday, 11 July 2022 9:55 *To:* squid-users@lists.squid-cache.org *Subject:* [squid-users] Sqid uses all RAM / killed by OOM Hello all, I have the following problem with squid 5.2 on ubuntu 22.04. Squid consumes all ram and the entire SWAP. When swap and ram are completely full, the OOM killer strikes and terminates the process. We use three internal child proxy servers with keepalived and haproxy as load balancers. From our ISP we use a parent upstream proxy for external internet traffic. As an operating system we have so far Ubuntu 20.04.4 with squid 4.1 in use. This constellation works flawlessly. Now I want to update the Server to Ubuntu 22.04 and squid 5.2. But with Ubuntu 22.04 and squid 5.2 the above mentioned problem with the OOM Killer occurs. The new machine has only the OS and squid installed. Who can help me with a solution? With kind regards Ronny Attached the squid configuration and the VMWare specs. ### VM Specs ### OS: Ubuntu 22.04 Server CPU: 4x (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz) RAM: 4 GB VMWare: ESXi 7.0 U2 ### CONFIG ### acl 10.172.xxx.xxx/18 <http://10.172.128.0/18> src 10.172.xxx.xxx/18 <http://10.172.128.0/18> acl 172.16.xxx.xxx/12 <http://172.16.0.0/12> src 172.16.xxx.xxx/12 <http://172.16.0.0/12> acl 192.168.xxx.xxx/16 <http://192.168.0.0/16> src 192.168.xxx.xxx/16 <http://192.168.0.0/16> acl Safe_ports port 80 acl Safe_ports port 21 acl Safe_ports port 443 acl Safe_ports port 70 acl Safe_ports port 210 acl Safe_ports port 1025-65535 acl Safe_ports port 280 acl Safe_ports port 488 acl Safe_ports port 591 acl Safe_ports port 777 http_access allow 10.172.xxx.xxx/18 <http://10.172.128.0/18> Safe_ports http_access allow 172.16.xxx.xxx/12 <http://172.16.0.0/12> Safe_ports http_access allow 192.168.xxx.xxx/16 <http://192.168.0.0/16> Safe_ports http_access allow localhost manager http_access allow localhost http_access deny manager http_access deny all include /etc/squid/conf.d/* http_port 10.172.xxx.xxx:3128 <http://10.172.128.34:3128> cache_peer 10.210.xxx.xxx parent 8080 0 cache_dir ufs /var/spool/squid 3000 16 256 cache_effective_user proxy cache_effective_group proxy coredump_dir /var/spool/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern \/(Packages|Sources)(|\.bz2|\.gz|\.xz)$ 0 0% 0 refresh-ims refresh_pattern \/Release(|\.gpg)$ 0 0% 0 refresh-ims refresh_pattern \/InRelease$ 0 0% 0 refresh-ims refresh_pattern \/(Translation-.*)(|\.bz2|\.gz|\.xz)$ 0 0% 0 refresh-ims refresh_pattern . 0 20% 4320 never_direct allow all max_filedescriptors 40960 dns_nameservers 10.244.xxx.xxx ### DMESG ### [256929.150801] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/squid.service,task=squid,pid=26390,uid=13 [256929.150822] Out of memory: Killed process 26390 (squid) total-vm:9691764kB, anon-rss:3657748kB, file-rss:2320kB, shmem-rss:0kB, UID:13 pgtables:18932kB oom_score_adj:0 [256929.510641] oom_reaper: reaped process 26390 (squid), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB _______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
