Thanks for the reply
> What OS are you using? How many CPU cores do you want to dedicate to Squid?
> How much memory?
I am using FreeBSD variant 4GB ram 2 CPUs pfSense plus
Starting CPU 1 (1)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
cpu1: <Open Firmware CPU> on cpulist0
e6000sw0: CPU port at 5
CPU 0: ARM Cortex-A53 r0p4 affinity: 0
CPU 1: ARM Cortex-A53 r0p4 affinity: 1
I do not know how to dedicate specific CPU cores to Squid I do not think I can
with pfSense plus.
> What filesystem are you using? For modern filesystems (ext4, btrfs, apfs)
> this parameter is much less meaningful than 10 years ago as they store
> directories as trees instead of lists.
=> 1 250069679 ada0 MBR (119G)
1 532480 1 efi (260M)
532481 131072 2 fat32 (64M)
663553 249406127 3 freebsd [active] (119G)
=> 0 249406127 ada0s3 BSD (119G)
0 16 - free - (8.0K)
16 235528175 1 freebsd-zfs (112G)
235528191 13877248 2 freebsd-swap (6.6G)
249405439 688 - free - (344K)
=> 40 500118112 nda0 GPT (238G)
40 2008 - free - (1.0M)
2048 16777216 1 freebsd-swap (8.0G)
16779264 482344960 2 freebsd-ufs (230G)
499124224 993928 - free - (485M)
ada0 is for the host os
nda0 is my cache is uses freebsd-ufs I use the command mount_msdosfs
/dev/nda0p2 /nvme/LOGS_Octane
The only way to mount the NVMe drive I have found is with mount_msdosfs
maybe this causes a slow down I do not know but I can write and save to the
drive this way. it is on nda0p2 230GB
Geom name: nda0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 500118151
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: nda0p1
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 1048576
Mode: r1w1e2
efimedia: HD(1,GPT,04d31fb2-c0fd-11ef-8536-90ec770dda25,0x800,0x1000000)
rawuuid: 04d31fb2-c0fd-11ef-8536-90ec770dda25
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: swapUSB
length: 8589934592
offset: 1048576
type: freebsd-swap
index: 1
end: 16779263
start: 2048
2. Name: nda0p2
Mediasize: 246960619520 (230G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 8590983168
Mode: r1w1e1
efimedia: HD(2,GPT,d84dfc00-cb1c-11ef-afd9-90ec770dda25,0x1000800,0x1cc00000)
rawuuid: d84dfc00-cb1c-11ef-afd9-90ec770dda25
rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
label: LOG
length: 246960619520
offset: 8590983168
type: freebsd-ufs
index: 2
end: 499124223
start: 16779264
Consumers:
1. Name: nda0
Mediasize: 256060514304 (238G)
Sectorsize: 512
Mode: r2w2e5
>> my memory replacement policy is LRU for memory cache it seems to run better
>> with that, my options for memory replacement policy are HEAP GDSF I assume
>> any HEAP will require more memory, HEAP LFUDF, HEAP LRU and just LRU.
>
> I think so but shouldn't be significantly more
Should I change Memory Replacement Policy from LRU to HEAP LRU? I have tried
every one again there is also the Cache Replacement Policy: Currently set to
HEAP LFUDA Should memory replacement policy and cache replacement policy both
be the same, and or does one cause any performance issues with the other?
Current Config
# This file is automatically generated by pfSense
# Do not edit manually !
http_port 192.168.1.1:3128 ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=20MB cert=/usr/local/etc/squid/serverkey.pem
cafile=/usr/local/share/certs/ca-root-nss.crt capath=/usr/local/share/certs/
cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
tls-dh=prime256v1:/etc/dh-parameters.2048
options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
http_port 127.0.0.1:3128 intercept ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=20MB cert=/usr/local/etc/squid/serverkey.pem
cafile=/usr/local/share/certs/ca-root-nss.crt capath=/usr/local/share/certs/
cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
tls-dh=prime256v1:/etc/dh-parameters.2048
options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
https_port 127.0.0.1:3129 intercept ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=20MB cert=/usr/local/etc/squid/serverkey.pem
cafile=/usr/local/share/certs/ca-root-nss.crt capath=/usr/local/share/certs/
cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
tls-dh=prime256v1:/etc/dh-parameters.2048
options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
icp_port 0
digest_generation off
dns_v4_first on
pid_filename /var/run/squid/squid.pid
cache_effective_user squid
cache_effective_group proxy
error_default_language en
icon_directory /usr/local/etc/squid/icons
visible_hostname Lee_Family.home.arpa
cache_mgr jonathanlee...@gmail.com
access_log /nvme/LOGS_Optane/Squid_Logs/access.log
cache_log /nvme/LOGS_Optane/Squid_Logs/cache.log
cache_store_log none
netdb_filename /nvme/LOGS_Optane/Squid_Logs/netdb.state
pinger_enable on
pinger_program /usr/local/libexec/squid/pinger
sslcrtd_program /usr/local/libexec/squid/security_file_certgen -s
/var/squid/lib/ssl_db -M 4MB -b 2048
tls_outgoing_options cafile=/usr/local/share/certs/ca-root-nss.crt
tls_outgoing_options capath=/usr/local/share/certs/
tls_outgoing_options options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
tls_outgoing_options
cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
sslcrtd_children 10
logfile_rotate 10
debug_options rotate=10
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src 192.168.1.0/27
forwarded_for delete
via off
httpd_suppress_version_string on
uri_whitespace strip
acl block_hours time 00:30-05:00
ssl_bump terminate all block_hours
http_access deny all block_hours
icp_port 0
htcp_port 0
snmp_port 0
icp_access deny all
htcp_access deny all
snmp_access deny all
acl getmethod method GET
acl to_ipv6 dst ipv6
acl from_ipv6 src ipv6
#tls_outgoing_options
cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
tls_outgoing_options
options=NO_SSLv3,NO_TLSv1,NO_TLSv1_1,NO_TICKET,SINGLE_DH_USE,SINGLE_ECDH_USE
#tls_outgoing_options default-ca=on
acl HttpAccess dstdomain '/usr/local/pkg/http.access'
acl windowsupdate dstdomain '/usr/local/pkg/windowsupdate'
#acl rewritedoms dstdomain '/usr/local/pkg/desdom'
#store_id_program /usr/local/libexec/squid/storeid_file_rewrite
/var/squid/storeid/storeid_rewrite.txt
#store_id_children 10 startup=5 idle=1 concurrency=0
#always_direct allow all
#store_id_access deny connect
#store_id_access deny !getmethod
#store_id_access allow rewritedoms
#store_id_access deny all
refresh_all_ims on
reload_into_ims on
max_stale 20 years
minimum_expiry_time 0
#refresh_pattern -i ^http.*squid.internal.* 43200 100% 79900 override-expire
override-lastmod ignore-reload ignore-no-store ignore-must-revalidate
ignore-private ignore-auth
refresh_pattern -i
windowsupdate.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80%
129600 reload-into-ims
refresh_pattern -i
microsoft.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80%
129600 reload-into-ims
refresh_pattern -i
windows.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80%
129600 reload-into-ims
refresh_pattern -i
microsoft.com.akadns.net/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf)
43200 80% 129600 reload-into-ims
refresh_pattern -i
deploy.akamaitechnologies.com/.*.(cab|exe|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf)
43200 80% 129600 reload-into-ims
acl https_login url_regex -i ^https.*(login|Login).*
cache deny https_login
#range_offset_limit 512 MB windowsupdate
range_offset_limit 0 !windowsupdate
quick_abort_min -1 KB
cache_mem 64 MB
maximum_object_size_in_memory 256 KB
memory_replacement_policy lru
cache_replacement_policy heap LFUDA
minimum_object_size 0 KB
maximum_object_size 512 MB
cache_dir aufs /nvme/LOGS_Optane/Squid_Cache 32000 16 256
offline_mode off
cache_swap_low 90
cache_swap_high 95
acl donotcache dstdomain '/var/squid/acl/donotcache.acl'
cache deny donotcache
cache allow all
# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|?) 0 0% 0
refresh_pattern . 0 20% 4320
#Remote proxies
# Setup some default acls
# ACLs all, manager, localhost, and to_localhost are predefined.
acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 8080 3128 3129
1025-65535
acl sslports port 443 563 8080 5223 2197
acl purge method PURGE
acl connect method CONNECT
# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS
# SslBump Peek and Splice
# http://wiki.squid-cache.org/Features/SslPeekAndSplice
# http://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit
# Match against the current step during ssl_bump evaluation [fast]
# Never matches and should not be used outside the ssl_bump context.
#
# At each SslBump step, Squid evaluates ssl_bump directives to find
# the next bumping action (e.g., peek or splice). Valid SslBump step
# values and the corresponding ssl_bump evaluation moments are:
# SslBump1: After getting TCP-level and HTTP CONNECT info.
# SslBump2: After getting TLS Client Hello info.
# SslBump3: After getting TLS Server Hello info.
# These ACLs exist even when 'SSL/MITM Mode' is set to 'Custom' so that
# they can be used there for custom configuration.
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
acl banned_hosts src '/var/squid/acl/banned_hosts.acl'
acl blacklist dstdom_regex -i '/var/squid/acl/blacklist.acl'
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports
# Always allow localhost connections
http_access allow localhost
quick_abort_min 0 KB
quick_abort_max 0 KB
quick_abort_pct 95
request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc
# Reverse Proxy settings
deny_info TCP_RESET allsrc
# Package Integration
url_rewrite_program /usr/local/bin/squidGuard -c
/usr/local/etc/squidGuard/squidGuard.conf
url_rewrite_bypass off
url_rewrite_children 25 startup=12 idle=8 concurrency=0
# Custom options before auth
#host_verify_strict on
# These hosts are banned
http_access deny banned_hosts
# Block access to blacklist domains
http_access deny blacklist
# List of domains allowed to logging in to Google services
request_header_access X-GoogApps-Allowed-Domains deny all
request_header_add X-GoogApps-Allowed-Domains consumer_accounts
# Set YouTube safesearch restriction
acl youtubedst dstdomain -n www.youtube.com m.youtube.com
youtubei.googleapis.com youtube.googleapis.com www.youtube-nocookie.com
request_header_access YouTube-Restrict deny all
request_header_add YouTube-Restrict none youtubedst
# Custom SSL/MITM options before auth
acl wpad urlpath_regex ^/wpad.dat$
acl wpad urlpath_regex ^/proxy.pac$
acl wpad urlpath_regex ^/wpad.da$
deny_info TCP_RESET wpad
#deny_info 200:/etc/squid/wpad.dat wpad
reply_header_access Content-Type deny wpad
http_access deny wpad
http_access deny !safeports
http_access deny CONNECT !sslports
cachemgr_passwd disable offline_toggle reconfigure shutdown
cachemgr_passwd redacted all
eui_lookup on
acl no_miss url_regex -i gateway.facebook.com/ws/realtime?
acl no_miss url_regex -i web-chat-e2ee.facebook.com/ws/chat
acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow CONNECT wuCONNECT localhost
http_access allow CONNECT windowsupdate localnet
http_access allow CONNECT windowsupdate localhost
http_access allow CONNECT HttpAccess localnet
http_access allow CONNECT HttpAccess localhost
http_access deny to_ipv6
http_access deny from_ipv6
acl BrokenButTrustedServers dstdomain '/usr/local/pkg/dstdom.broken'
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all
acl splice_only_ip src 192.168.1.8
acl splice_only_ip src 192.168.1.10
acl splice_only_ip src 192.168.1.11
acl splice_only_ip src 192.168.1.15
acl splice_only_ip src 192.168.1.16
:::: = redacted mac address
acl splice_only_mac arp :::::
acl splice_only_mac arp :::::
acl splice_only_mac arp :::::
acl splice_only_mac arp :::::
acl splice_only_mac arp :::::
acl NoSSLIntercept ssl::server_name_regex -i '/usr/local/pkg/reg.url.nobump'
acl NoBumpDNS dstdomain -n '/usr/local/pkg/dns.nobump'
acl SSL_Intercept_Terminate dstdomain -n '/usr/local/pkg/url.bump'
acl active_use annotate_client active=true
acl bump_only_ip src 192.168.1.3
acl bump_only_ip src 192.168.1.4
acl bump_only_ip src 192.168.1.5
#acl bump_only_ip src 192.168.1.6
acl bump_only_ip src 192.168.1.9
acl bump_only_ip src 192.168.1.13
acl bump_only_mac arp :::::
acl bump_only_mac arp :::::
acl bump_only_mac arp :::::
acl bump_only_mac arp :::::
acl bump_only_mac arp :::::
#acl bump_only_mac arp :::::
coredump_dir /nvme/LOGS_Optane/Squid_Dump
acl splice_group any-of https_login NoBumpDNS NoSSLIntercept
acl splice_only_local_group all-of splice_only_mac splice_only_ip
acl splice_main any-of splice_group splice_only_local_group
acl bump_main all-of bump_only_mac bump_only_ip
ssl_bump peek step1
ssl_bump terminate SSL_Intercept_Terminate
miss_access deny no_miss active_use
ssl_bump splice splice_main active_use
ssl_bump bump bump_main active_use
acl activated note active_use true
ssl_bump terminate !activated
# Setup allowed ACLs
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc
Does delay pool setting cause any issues? They seem to be default values one
pool.
> On Jan 10, 2025, at 00:25, Francesco Chemolli <gkin...@gmail.com> wrote:
>
>
>
> On Fri, Jan 10, 2025 at 7:22 AM Jonathan Lee <jonathanlee...@gmail.com
> <mailto:jonathanlee...@gmail.com>> wrote:
>>
>> After trying every setting inside of Squid I thought I should ask I have 4bg
>> ram and 128GB M.2 SSD onboard disk I am using a NVMe secondary Intel Optane
>> M.2 drive for my cache.
>
> What OS are you using? How many CPU cores do you want to dedicate to Squid?
> How much memory?
>
>> What is a good recommendation for Hard Drive Cache System I use UFS but AUFS
>
> UFS is the slowest option; AUFS or rock are considered the fastest
>
>> inside of the Squid definitive guide says it is way faster like formula one
>> versus UFS the options I have are UFS AUFS DISKD, I have 16 Level 1
>> directories
>
> What filesystem are you using? For modern filesystems (ext4, btrfs, apfs)
> this parameter is much less meaningful than 10 years ago as they store
> directories as trees instead of lists.
>
>> my memory replacement policy is LRU for memory cache it seems to run better
>> with that, my options for memory replacement policy are HEAP GDSF I assume
>> any HEAP will require more memory, HEAP LFUDF, HEAP LRU and just LRU.
>
> I think so but shouldn't be significantly more
>
>> I also have a Cache Replacement policy with the same options I have it set
>> to Heap LFUDA that is the default. Squid Memory cache Size I have set to
>> default 64MB with max object size 256kb for the memory, for disk I have
>> 256GB available I only have it set to 32000MB or 3.2GB for fear of
>> overloading the RAM when it fills up.
>
> 4gb is plenty of memory; what other workloads do you want to run on that
> machine? You can also tune these parameters after checking behaviour in
> practice, no need to fix them once and for all now
>
>> for level 1 directories I can have 4,8,16,32,64,128,256 each layer one
>> containers 256 sub directories so this could hog memory if you did
>> 256*256=65,536 I imagine not ok with only 4GB I have onboard memory I can’t
>> make it any bigger. I use this with SSL intercept it does cache and works
>> well I just want to get rid of the lag on news websites.
>
> number of directories has no impact on memory use. Just be aware that if you
> change it, you need to wipe and rebuild your cache.
>
>> rewrite process children I have it set max 25 with process children startup
>> at 12 and idle at 8
>> SSL certificate deamon children I have it set to start 10
>
> Sure.
>
>> it runs well I have tried many different things as you know from all the
>> emails, I am sorry it is the most fascinating software to me. Code that runs
>> as fast as the internet. Is there any thing I can do to make it go faster?
>> Some website have a lag fox news yahoo only do on the SSL intercept devices
>> the splice devices never have any issues, its lighting fast for them. I
>> thought I should finally ask after 4-5 years of doing changes. I have got it
>> to work as fast as I can on my own, time to ask the community.
>
> I think it boils down mainly to how much memory you're willing to dedicate to
> squid. More memory more performance. Apart from that, they main advice is to
> chang from UFS to just about anything else
>
> --
> Francesco
> _______________________________________________
> squid-users mailing list
> squid-users@lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users
