thanks for your detailed reply, Amos. now I have found out the culprit is large file descriptor number limit. Docker CE for Debian package has set this limit (RLIMIT_NOFILE) in the container to 1048576, thus squid set Squid_MaxFD to 1048576, then allocates 1048576*392 = 392m memory for `fd_table`. in the host, RLIMIT_NOFILE is only 1024.
> So, if you actually do not want the proxy caching anything, then > disabling the cache_mem (set it to 0 as per Alex response) would be the > best choice of action before you go any further. in fact, I have tried this option before but failed. it is very nice to learn this advice, I will add it to my proxy only squid config. > So disabling swap entirely on the server is not a great idea. It just > moves the error and shutdown to happen at peak traffic times when it is > least wanted. but I guess this is common in the "cloud" era, eg: Google Compute Engine. Xie Shi On Tue, Mar 5, 2019 at 4:13 PM Amos Jeffries <squ...@treenet.co.nz> wrote: > > On 4/03/19 9:45 pm, George Xie wrote: > > > On 4/03/19 5:39 pm, George Xie wrote: > > > > hi all: > > > > > > > > Squid version: 3.5.23-5+deb9u1 > > > > Docker version 18.09.3, build 774a1f4 > > > > Linux instance-4 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) > > > > x86_64 GNU/Linux > > > > > > > > I have the following squid config: > > > > > > > > > > > > http_port 127.0.0.1:3128 <http://127.0.0.1:3128> > > > > cache deny all > > > > access_log none > > > > > > > What is it exactly that you think this is doing in regards to Squid > > > memory needs? > > > > > > > > > sorry, I don't get your quest. > > > > I was asking to see what you were thinking was going on with those settings. > > As Alex already pointed out the "cache deny all" does not reduce memory > needs of Squid in any way. It just makes 256MB of that RAM become > pointless allocating. > > So, if you actually do not want the proxy caching anything, then > disabling the cache_mem (set it to 0 as per Alex response) would be the > best choice of action before you go any further. > > Or if you *do* want caching, and were trying to disable it for testing > the memory issue. Then your test was wrong, and produces incorrect > conclusion. Just reducing cache_mem would be best for this case - set it > to a value that should reasonably fit this container and see if the > proxy runs okay. > > > ... > > > > > > > > it appears that squid (or glibc) tries to allocate 392m memory, > > which is > > > > larger than host free memory 370m. > > > > but I guess squid don't need that much memory, I have another > > running > > > > squid instance, which only uses < 200m memory. > > > No doubt it is configured to use less memory. For example by reducing > > > the default memory cache size. > > > > > > > > > that running squid instance has the same config. > > > > Then something odd is going on between the two. They should indeed have > had the same behaviour (either work or same error). > > Whatever the issue is it is being triggered by the large blocks of RAM > allocated by a default Squid. The easiest to modify is the cache_mem. > > > > > > > > the oddest thing is if I run squid on the host (also Debian 9) > > directly, > > > > not in the container, squid could start and run as normal. > > > > > > > Linux typically allows RAM over-allocation. Which works okay so > > long as > > > there is sufficient swap space and there is time between memory > > usage to > > > do the swap in/out process. > > > Amos > > > > > > swap is disabled in the host server, so do in the container. > > > > after all, I wonder why squid would try to claim 392m memory if don't > > need that much. > > > > Squid thinks it does. All client traffic is denied being cached by that > "deny all". BUT ... there are internally generated items which also use > cache. So there is 256MB default RAM cache allocated and only those few > small things being put in it. > > You could set it to '0' or to some small value and the allocation size > should go down accordingly. > > > That said, every bit of client traffic headed towards the proxy uses > memory of volatile amount and at peak times it may need to allocate > large blocks. > > So disabling swap entirely on the server is not a great idea. It just > moves the error and shutdown to happen at peak traffic times when it is > least wanted. > > > Amos > _______________________________________________ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users _______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
