Hello folks, we're running Squid 3.5.28 on four Windows 2016 VM's, each has 4 CPU, 8 GB memory, 10 GB NIC. We implemented Squid to support forwarding Azure Log Analytics data, it's all CONNECT. The Squids are load balanced through a F5. There are less than 1,000 servers sending data to Log Analytics.
All four Squids are regularly crashing, and I don't know how to interpret the errors in cache.log. It crashes if we disable caching too. Any insight is appreciated-I've inherited this responsibility and more a cloud engineer than a network specialist. Thanks in advance! Typical error sequence in cache.log 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:32 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| FD 12, 10.5.11.12 [Stopped, reason:Listener socket closed job1]: (14) Bad address 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 1 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 2 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 3 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 4 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 5 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 6 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 7 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 8 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 9 2019/02/20 09:42:33 kid1| comm_poll: poll failure: (14) Bad address 2019/02/20 09:42:33 kid1| Select loop Error. Retry 10 2019/02/20 09:42:33 kid1| Closing HTTP port 10.5.11.12:3128 FATAL: Event loop exited with failure. Squid Cache (Version 3.5.28): Terminated abnormally. CPU Usage: 12.640 seconds = 4.234 user + 8.406 sys Maximum Resident Size: 5159680 KB Page faults with physical i/o: 20341 Squid restarts, and will often start erroring right away: 2019/02/20 09:42:33 kid1| storeDirWriteCleanLogs: Starting... 2019/02/20 09:42:33 kid1| Finished. Wrote 0 entries. 2019/02/20 09:42:33 kid1| Took 0.00 seconds ( 0.00 entries/sec). 2019/02/20 09:42:36 kid1| Set Current Directory to /cygdrive/e/squid/var/coredump 2019/02/20 09:42:36 kid1| Starting Squid Cache version 3.5.28 for x86_64-unknown-cygwin... 2019/02/20 09:42:36 kid1| Service Name: squid 2019/02/20 09:42:36 kid1| Process ID 2292 2019/02/20 09:42:36 kid1| Process Roles: worker 2019/02/20 09:42:36 kid1| With 3200 file descriptors available 2019/02/20 09:42:36 kid1| Initializing IP Cache... 2019/02/20 09:42:36 kid1| parseEtcHosts: /etc/hosts: (2) No such file or directory 2019/02/20 09:42:36 kid1| DNS Socket created at [::], FD 5 2019/02/20 09:42:36 kid1| DNS Socket created at 0.0.0.0, FD 6 2019/02/20 09:42:36 kid1| Adding nameserver 208.67.220.220 from squid.conf 2019/02/20 09:42:36 kid1| Adding nameserver 208.67.222.222 from squid.conf 2019/02/20 09:42:36 kid1| Logfile: opening log daemon:/var/log/squid/access.log 2019/02/20 09:42:36 kid1| Logfile Daemon: opening log /var/log/squid/access.log 2019/02/20 09:42:36 kid1| WARNING: no_suid: setuid(0): (22) Invalid argument 2019/02/20 09:42:36 kid1| Store logging disabled 2019/02/20 09:42:36 kid1| Swap maxSize 262144 + 262144 KB, estimated 40329 objects 2019/02/20 09:42:36 kid1| Target number of buckets: 2016 2019/02/20 09:42:36 kid1| Using 8192 Store buckets 2019/02/20 09:42:36 kid1| Max Mem size: 262144 KB 2019/02/20 09:42:36 kid1| Max Swap size: 262144 KB 2019/02/20 09:42:36 kid1| Rebuilding storage in /cygdrive/e/squid/cache (clean log) 2019/02/20 09:42:36 kid1| Using Least Load store dir selection 2019/02/20 09:42:36 kid1| Set Current Directory to /cygdrive/e/squid/var/coredump 2019/02/20 09:42:36 kid1| Finished loading MIME types and icons. 2019/02/20 09:42:36 kid1| HTCP Disabled. 2019/02/20 09:42:36 kid1| Squid plugin modules loaded: 0 2019/02/20 09:42:36 kid1| Adaptation support is off. 2019/02/20 09:42:36 kid1| Accepting HTTP Socket connections at local=10.5.11.12:3128 remote=[::] FD 12 flags=9 2019/02/20 09:42:36 kid1| Done reading /cygdrive/e/squid/cache swaplog (0 entries) 2019/02/20 09:42:36 kid1| Store rebuilding is 0.00% complete 2019/02/20 09:42:36 kid1| Finished rebuilding storage from disk. 2019/02/20 09:42:36 kid1| 0 Entries scanned 2019/02/20 09:42:36 kid1| 0 Invalid entries. 2019/02/20 09:42:36 kid1| 0 With invalid flags. 2019/02/20 09:42:36 kid1| 0 Objects loaded. 2019/02/20 09:42:36 kid1| 0 Objects expired. 2019/02/20 09:42:36 kid1| 0 Objects cancelled. 2019/02/20 09:42:36 kid1| 0 Duplicate URLs purged. 2019/02/20 09:42:36 kid1| 0 Swapfile clashes avoided. 2019/02/20 09:42:36 kid1| Took 0.04 seconds ( 0.00 objects/sec). 2019/02/20 09:42:36 kid1| Beginning Validation Procedure 2019/02/20 09:42:36 kid1| Completed Validation Procedure 2019/02/20 09:42:36 kid1| Validated 0 Entries 2019/02/20 09:42:36 kid1| store_swap_size = 0.00 KB 2019/02/20 09:42:37 kid1| storeLateRelease: released 0 objects 2019/02/20 09:42:45 kid1| FD 12, 10.5.11.12 [ job1]: (14) Bad address Squid.conf # Example rule allowing access from your local networks. # Adapt to list your (internal) IP networks from where browsing # should be allowed acl localnet src 10.0.0.0/8 # RFC1918 possible internal network #acl localnet src 172.16.0.0/12 # RFC1918 possible internal network #acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT # Only allow cachemgr access from localhost http_access allow localhost manager http_access deny manager # Deny requests to certain unsafe ports http_access deny !Safe_ports # Deny CONNECT to other than secure SSL ports http_access deny CONNECT !SSL_ports # We strongly recommend the following be uncommented to protect innocent # web applications running on the proxy server who think the only # one who can access services on "localhost" is a local user http_access deny to_localhost # Example rule allowing access from your local networks. # Adapt localnet in the ACL section to list your (internal) IP networks # from where browsing should be allowed http_access allow localnet http_access allow localhost # And finally deny all other access to this proxy http_access deny all # Squid normally listens to port 3128 http_port 10.5.11.12:3128 # Uncomment the line below to enable disk caching - path format is /cygdrive/<full path to cache folder>, i.e. cache_dir aufs /cygdrive/e/squid/cache 256 8 64 #cache deny all # Limit number of days to keep logs logfile_rotate 2 # Coredump directory coredump_dir /cygdrive/e/squid/var/coredump # Add any of your own refresh_pattern entries above these. refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 dns_nameservers 208.67.220.220 208.67.222.222 max_filedescriptors 3200 Typical access.log 1550724138.034 213 10.27.18.220 TCP_TUNNEL/200 4301 CONNECT ac3d8ead-8d6e-423e-8f51-1beadafe281a.ods.opinsights.azure.com:443 - HIER_DIRECT/40.71.12.224 - 1550724153.063 122686 10.27.18.220 TCP_TUNNEL/200 8820 CONNECT eus2-jobruntimedata-prod-su1.azure-automation.net:443 - HIER_DIRECT/104.208.163.218 - 1550724155.287 635036 10.27.18.220 TCP_TUNNEL/200 11107 CONNECT 593a6510-ebfc-4d6b-a8f0-a0411dfee098.ods.opinsights.azure.com:443 - HIER_DIRECT/40.71.12.224 - 1550724155.299 455045 10.27.18.220 TCP_TUNNEL/200 9280 CONNECT 593a6510-ebfc-4d6b-a8f0-a0411dfee098.ods.opinsights.azure.com:443 - HIER_DIRECT/40.71.12.224 - 1550724158.005 187 10.27.18.220 TCP_TUNNEL/200 4301 CONNECT ac3d8ead-8d6e-423e-8f51-1beadafe281a.ods.opinsights.azure.com:443 - HIER_DIRECT/40.71.12.224 - 1550724178.345 505 10.27.18.220 TCP_TUNNEL/200 4301 CONNECT ac3d8ead-8d6e-423e-8f51-1beadafe281a.ods.opinsights.azure.com:443 - HIER_DIRECT/40.71.12.224 - Andrew Van Order CTO | Application Delivery Services | Hosting Services - Monitoring and Configuration Services Deloitte Services LP Tel/Direct: +1 615 882 7836 | Fax: +1 615 750 7836 dvanor...@deloitte.com | www.deloitte.com This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited. v.E.1
_______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
