Hi, thanks so much for the detailed response. I chose to test option 2 from your recommendations as I am new to squid and I do not understand how to set it up as a reverse proxy anyway. I made the change to my squid.conf :
#ssl_bump peek step1 ssl_bump bump step1 ssl_bump bump all This made it work - which is great news. My curl requests now are satisfied by the cache when the pc is offline! I do have 1 followup question which I think is unrelated, let me know if etiquette demands I create a new post for this. When I test using chromium browser, chromium sends OPTION requests - which I think is something to do with CORS. These always cause cache MISS from squid,.. I think because the return code is 204...? 1705669236.776 113 ::1 TCP_MISS/204 680 OPTIONS https://stuff.amazonaws.com/api/v1/stuff/stuff.json - HIER_DIRECT/ 3.135.146.17 application/json I can prevent my chromium instance from making these (pointless?) OPTIONS calls using the following args, but I would rather not have to do this. --disable-web-security --disable-features=IsolateOrigins,site-per-process Any way I can get squid to cache these calls? Thanks again and all the best, Robin On Thu, 18 Jan 2024 at 16:03, Alex Rousskov < rouss...@measurement-factory.com> wrote: > On 2024-01-18 09:53, Robin Carlisle wrote: > > > My expectation/hope is that squid would return the cached object on > > any network failure in between ubuntu-pc and the AWS endpoint - and > > continue to return this cached object forever. Is this something > > squid can do? It would seem that offline_mode should do this? > > Yes and yes. The reason you are getting errors are not related to cache > hits or misses. Those errors happen _before_ Squid gets the requested > resource URL and looks up that resource in Squid cache. > > > ssl_bump peek step1 > > ssl_bump bump all > > To get that URL (in your configuration), Squid must bump the connection. > To bump the connection at step2, Squid must contact the origin server. > When the cable is unplugged, Squid obviously cannot do that: The attempt > to open a Squid-AWS connection fails. > > > .../200 0 CONNECT stuff.amazonaws.com:443 - HIER_DIRECT > > .../503 4087 GET https://stuff.amazonaws.com/api/... - HIER_NONE > > Squid reports bumping errors to the client using HTTP responses. To do > that, Squid remembers the error response, bumps the client connection, > receives GET from the client on that bumped connection, and sends that > error response to the client. This is why you see both CONNECT/200 and > GET/503 access.log records. Note that Squid does not check whether the > received GET request would have been a cache hit in this case -- the > response to that request has been preordained by the earlier bumping > failure. > > > Solution candidates to consider include: > > * Stop bumping: https_port 443 cert=/etc/squid/stuff.pem > > Configure Squid as (a reverse HTTPS proxy for) the AWS service. Use > https_port. No SslBump rules/options! The client would think that it is > sending HTTPS requests directly to the service. Squid will forward > client requests to the service. If this works (and I do not have enough > information to know that this will work in your specific environment), > then you will get a much simpler setup. > > > * Bump at step1, before Squid contacts AWS: ssl_bump bump all > > Bugs notwithstanding, there will be no Squid-AWS connection for cache > hits. The resulting certificate will not be based on AWS service info, > but it looks like your client is ignorant enough to ignore related > certificate problems. > > > HTH, > > Alex. > > > > Hi, Hoping someone can help me with this issue that I have been > > struggling with for days now. I am setting up squid on an ubuntu PC to > > forward HTTPS requests to an API and an s3 bucket under my control on > > amazon AWS. The reason I am setting up the proxy is two-fold... > > > > 1) To reduce costs from AWS. > > 2) To provide content to the client on the ubuntu PC if there is a > > networking issue somewhere in between the ubuntu PC and AWS. > > > > Item 1 is going well so far. Item 2 is not going well. Setup details > ... > > > > *# squid - setup cache folder* > > mkdir -p /var/cache/squid > > chown -R proxy:proxy /var/cache/squid > > > > *# ssl - generate key* > > apt --yes install squid-openssl libnss3-tools > > openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 \ > > -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com > > <http://www.example.com>" \ > > -keyout /etc/squid/stuff.pem -out /etc/squid/stuff.pem > > chown root:proxy /etc/squid/stuff.pem > > chmod 644 /etc/squid/stuff.pem > > > > *# ssl - ssl DB* > > mkdir -p /var/lib/squid > > rm -rf /var/lib/squid/ssl_db > > /usr/lib/squid/security_file_certgen -c -s /var/lib/squid/ssl_db -M 4MB > > chown -R proxy:proxy /var/lib/squid/ssl_db > > > > *# /etc/squid/squid.conf :* > > acl to_aws dstdomain .amazonaws.com <http://amazonaws.com> > > acl from_local src localhost > > http_access allow to_aws > > http_access allow from_local > > cache allow all > > cache_dir ufs /var/cache/squid 1024 16 256 > > offline_mode on > > http_port 3129 ssl-bump cert=/etc/squid/stuff.pem > > generate-host-certificates=on dynamic_cert_mem_cache_size=4MB > > sslcrtd_program /usr/lib/squid/security_file_certgen -s > > /var/lib/squid/ssl_db -M 4MB > > acl step1 at_step SslBump1 > > ssl_bump peek step1 > > ssl_bump bump all > > sslproxy_cert_error deny all > > cache_store_log stdio:/var/log/squid/store.log > > logfile_rotate 0 > > > > *# /usr/bin/proxy-test :* > > #!/bin/bash > > curl --proxy http://localhost:3129 <http://localhost:3129> \ > > --cacert /etc/squid/stuff.pem \ > > -v "https://stuff.amazonaws.com/api/v1/stuff/stuff.json > > <https://stuff.amazonaws.com/api/v1/stuff/stuff.json>" \ > > -H "Authorization: token MYTOKEN" \ > > -H "Content-Type: application/json" \ > > --output "/tmp/stuff.json" > > > > > > > > When network connectivity is GOOD, everything works well and I get cache > > HITS ... > > > > *# /var/log/squid/access.log* > > 1705587538.837 238 127.0.0.1 NONE_NONE/200 0 CONNECT > > stuff.amazonaws.com:443 <http://stuff.amazonaws.com:443> - > > HIER_DIRECT/3.136.246.238 <http://3.136.246.238> - > > 1705587538.838 0 127.0.0.1 TCP_MEM_HIT/200 32818 GET > > https://stuff.amazonaws.com/api/v1/stuff/stuff.json > > <https://stuff.amazonaws.com/api/v1/stuff/stuff.json> - HIER_NONE/- > > application/json > > > > *# extract from /usr/bin/proxy-test output* > > < HTTP/1.1 200 OK > > < Date: Thu, 18 Jan 2024 13:38:01 GMT > > < Content-Type: application/json > > < Content-Length: 32187 > > < x-amzn-RequestId: 8afba80e-6df7-4d5b-a34b-a70bd9b54380 > > < Last-Modified: 2024-01-03T11:23:19.000Z > > < Access-Control-Allow-Origin: * > > < x-amz-apigw-id: RvN1CF2_iYcEokA= > > < Cache-Control: max-age=2147483648,public,stale-if-error > > < ETag: "53896156c4e8e26933188a092c4e40f1" > > < X-Amzn-Trace-Id: Root=1-65a929b9-3bd3285934151c1a2495481a > > < Age: 2578 > > < Warning: 110 squid/5.7 "Response is stale" > > < X-Cache: HIT from ubuntu-pc > > < X-Cache-Lookup: HIT from ubuntu-pc:3129 > > < Via: 1.1 ubuntu-pc (squid/5.7) > > < Connection: keep-alive > > > > > > When network connectivity is BAD, I get errors and a cache MISS. In > > this test case I unplugged the ethernet cable from the back on the > > ubuntu-pc ... > > > > *# /var/log/squid/access.log* > > 1705588717.420 11 127.0.0.1 NONE_NONE/200 0 CONNECT > > stuff.amazonaws.com:443 <http://stuff.amazonaws.com:443> - > > HIER_DIRECT/3.135.162.228 <http://3.135.162.228> - > > 1705588717.420 0 127.0.0.1 NONE_NONE/503 4087 GET > > https://stuff.amazonaws.com/api/v1/stuff/stuff.json > > <https://stuff.amazonaws.com/api/v1/stuff/stuff.json> - HIER_NONE/- > > text/html > > > > *# extract from /usr/bin/proxy-test output* > > < HTTP/1.1 503 Service Unavailable > > < Server: squid/5.7 > > < Mime-Version: 1.0 > > < Date: Thu, 18 Jan 2024 14:38:37 GMT > > < Content-Type: text/html;charset=utf-8 > > < Content-Length: 3692 > > < X-Squid-Error: ERR_CONNECT_FAIL 101 > > < Vary: Accept-Language > > < Content-Language: en > > < X-Cache: MISS from ubuntu-pc > > < X-Cache-Lookup: NONE from ubuntu-pc:3129 > > < Via: 1.1 ubuntu-pc (squid/5.7) > > < Connection: close > > > > I have also seen it error in a different way with a 502 but with the > > same ultimate result. > > > > My expectation/hope is that squid would return the cached object on any > > network failure in between ubuntu-pc and the AWS endpoint - and continue > > to return this cached object forever. Is this something squid can do? > > It would seem that offline_mode should do this? > > > > Hope you can help, > > > > Robin > > > > > > > > _______________________________________________ > > squid-users mailing list > > squid-users@lists.squid-cache.org > > https://lists.squid-cache.org/listinfo/squid-users > >
_______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org https://lists.squid-cache.org/listinfo/squid-users
