Control: tags -1 patch Hi,
Thank you Vlad for the nice hint here! I was searching for such curl option one year ago in Debian stretch but it was missing there as curl there was only 7.52. Sure, the fix is much nicer in Debian buster with new curl nowadays. See attached fix #2 for approx behind HTTPS proxy which is using "HTTP CONNECT" method. Please consider to include it into default approx codebase. P.S. the patch has been taken from GitHub: > https://github.com/raol/approx-clone/commit/b5b2fb03d35bbc49ccf39661e024a5e2136daa00 P.P.S. I have also attached the Dockerfile for the test environment where you can reproduce the issue. It contains stock approx and basic squid proxy. Please see the steps to reproduce at the end of Dockerfile. P.P.P.S If someone cannot wait for the fix here, you can workaround the problem adding to /etc/approx/approx.conf: > $curl_path /usr/bin/curl${IFS}--suppress-connect-headers It will work for curl 7.54+ only (Debian buster+). Thank you! -- Alex Lutay Head of Quality Assurance Sipwise GmbH, Campus 21/Europaring F15 AT-2345 Brunn am Gebirge
FROM debian:buster MAINTAINER Alex Lutay <[email protected]> RUN apt-get update && \ apt-get -y install approx curl vim less openbsd-inetd squid3 procps RUN echo "9999 stream tcp nowait approx /usr/sbin/tcpd /usr/bin/env http_proxy= https_proxy= /usr/sbin/approx" > /etc/inetd.conf && \ echo "debian-http http://debian.sipwise.com/debian"; >> /etc/approx/approx.conf && \ echo "debian-https https://debian.sipwise.com/debian"; >> /etc/approx/approx.conf && \ echo '$interval 1' >> /etc/approx/approx.conf && \ echo '$verbose true' >> /etc/approx/approx.conf && \ echo '$debug true' >> /etc/approx/approx.conf && \ \ sed -i 's/^#http_access allow localnet/http_access allow localnet/' /etc/squid/squid.conf && \ echo "http_port 3129" >> /etc/squid/squid.conf && \ echo "acl sipwise dstdomain debian.sipwise.com" >> /etc/squid/squid.conf && \ echo "always_direct allow sipwise" >> /etc/squid/squid.conf WORKDIR /root/ ################################################################################ # Instructions for usage # ---------------------- # When you want to build the base image from scratch: # % docker build --no-cache --tag="approx-buster" . # % docker run --rm -i -t approx-buster:latest bash # # Inside docker: # # 1) run: service rsyslog start # approx logs now in /var/log/syslog # run: service openbsd-inetd start # now approx cache service is listening on port 9999 # run: service squid start # start local proxy # # 2) run: curl http://127.0.0.1:9999/debian-http/dists/buster/main/binary-amd64/Release # # The file 'Release' with proper content has been created in cache: # > root@e7e3277d030a:~# cat /var/cache/approx/debian-http/dists/buster/main/binary-amd64/Release # > Component: main # > Origin: Debian # > Architecture: amd64 # > Description: Debian buster mirror # > root@e7e3277d030a:~# # # All OK here. This is an expected behaviour, approx has cached the file properly. # P.S The original URL is http://debian.sipwise.com/debian/dists/buster/main/binary-amd64/Release # # # 3) Let's test approx behind HTTP proxy first # # - change: sed -i "s#http_proxy= #http_proxy=http://127.0.0.1:3128/ #" /etc/inetd.conf # - restart: service openbsd-inetd restart # - remove: rm /var/cache/approx/debian-http/dists/buster/main/binary-amd64/Release # NOTE: the files will be returned from cache if available, remove it to request new one from proxy! # # - test: curl http://127.0.0.1:9999/debian-http/dists/buster/main/binary-amd64/Release # - check: cat /var/cache/approx/debian-http/dists/buster/main/binary-amd64/Release # > Component: main # > Origin: Debian # > Architecture: amd64 # > Description: Debian buster mirror # > root@3aab8dda86e2:~# # ALL OK here too, you can repeate curl command if necessary # # JFYI squid logs in /var/log/squid/access.log: # > 127.0.0.1 TCP_MISS/200 408 GET http://debian.sipwise.com/debian/dists/buster/main/binary-amd64/Release - HIER_DIRECT/192.168.208.45 # # # 4) Let's test approx behind HTTPS proxy # # - change: sed -i "s#https_proxy= #https_proxy=http://127.0.0.1:3129/ #" /etc/inetd.conf # - restart: service openbsd-inetd restart # - remove: rm -f /var/cache/approx/debian-https/dists/buster/main/binary-amd64/Release # NOTE: the files will be returned from cache if available, remove it to use proxy! # # - test: curl http://127.0.0.1:9999/debian-https/dists/buster/main/binary-amd64/Release # NOTE: 1) curl connects http (NOT httpS) as approx doesn't support incoming httpS # 2) "debian-https" is in use here, see approx.conf: "debian-httpS httpS://debian.sipwise.com/debian" # # - my test: curl http://127.0.0.1:9999/debian-https/dists/buster/main/binary-amd64/Release # > HTTP/1.1 200 OK # > Date: Thu, 31 Oct 2019 09:14:25 GMT # > Server: Apache # > Last-Modified: Thu, 06 Sep 2018 12:30:54 GMT # > ETag: "55-575330ff277aa" # > Accept-Ranges: bytes # > Content-Length: 85 # > # > Component: main # > Origin: Debian # > Architecture: amd64 # > Description: Debian buster mirror # > root@e90450035928:~# !!! Wrong here, we see headers here! and they are also in cached file: # # - check: cat /var/cache/approx/debian-https/dists/buster/main/binary-amd64/Release # > root@e90450035928:~# cat /var/cache/approx/debian-https/dists/buster/main/binary-amd64/Release # > HTTP/1.1 200 OK # > Date: Thu, 31 Oct 2019 09:14:25 GMT # > Server: Apache # > Last-Modified: Thu, 06 Sep 2018 12:30:54 GMT # > ETag: "55-575330ff277aa" # > Accept-Ranges: bytes # > Content-Length: 85 # > # > Component: main # > Origin: Debian # > Architecture: amd64 # > Description: Debian buster mirror # > root@3aab8dda86e2:~# # !!! File Release has been DAMAGED in cache !!! # # Issue explanation: # # root@e636398d4101:~# https_proxy=http://127.0.0.1:3129 /usr/bin/curl --dump-header /tmp/zzz.log --fail --silent --header "Pragma: no-cache" --include --header "If-Modified-Since: Thu, 30 Nov 2019 11:24:30 GMT" "https://deb.sipwise.com/debian/dists/buster/main/binary-amd64/Release"; # > HTTP/1.1 200 Connection established # > # > HTTP/1.1 200 OK # > Date: Thu, 31 Oct 2019 12:38:27 GMT # > Server: Apache # > Last-Modified: Thu, 06 Sep 2018 12:30:54 GMT # > ETag: "55-575330ff277aa" # > Accept-Ranges: bytes # > Content-Length: 85 # > # > Component: main # > Origin: Debian # > Architecture: amd64 # > Description: Debian buster mirror # > root@e636398d4101:~# # # The first "200 Connection established" is "connected to proxy", the second "200 OK" is connected to "debian.sipwise.com" # approx removed the first section of headers but left the second one. ################################################################################
From b5b2fb03d35bbc49ccf39661e024a5e2136daa00 Mon Sep 17 00:00:00 2001 From: Oleg Rakitskiy <[email protected]> Date: Wed, 13 Nov 2019 21:05:46 +0200 Subject: [PATCH] Fix connection established response small fix of the issue described here https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914798 new curl version starting from 7.54 supports suppressing connect header, so used this options instead of creating own workaround --- url.ml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/url.ml b/url.ml index ac185f2..99e73f2 100644 --- a/url.ml +++ b/url.ml @@ -56,7 +56,7 @@ let rate_option = let curl_command options url = Printf.sprintf - "%s --fail --silent --header \"Pragma: no-cache\" %s %s %s" + "%s --fail --silent --header \"Pragma: no-cache\" --suppress-connect-headers %s %s %s" curl_path rate_option (String.concat " " options) (quoted_string url) let head_command = curl_command ["--head"] -- 2.1.4
