[squid-users] icap not answering
Hi, i would like todo modifications on https connections and therefore enabled ssl bump in squid 4.4, now i would like to see the real traffic and icap looks like a way to watch and change that traffic. but squid is not answering to icap://127.0.0.1:1344 when using pyicap or telnet. the telnet error is: telnet 127.0.0.1 1344 Trying 127.0.0.1... telnet: Unable to connect to remote host: Connection refused which is imho good because it tells me that something is answering on that port after all. did i misconfigure something? config: debug_options 28,9 #icap icap_enable on icap_service service_req reqmod_precache bypass=1 icap://127.0.0.1:1344/reqmod adaptation_access service_req allow all icap_service service_resp respmod_precache bypass=0 icap://127.0.0.1:1344/respmod adaptation_access service_resp allow all acl localnet src 127.0.0.1/32 192.168.10.0/24 http_access allow localnet acl SSL_ports port 443 acl CONNECT method CONNECT #http_access deny !Safe_ports #http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager include /etc/squid/conf.d/* http_access allow localhost coredump_dir /var/spool/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 # default end # my config http_port 3128 accel ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/myCA.pem https_port 3129 ssl-bump intercept generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/myCA.pem sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/lib/ssl_db -M 4MB acl step1 at_step SslBump1 ssl_bump peek step1 ssl_bump bump all ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] icap not answering
thod CONNECT http_access allow localnet coredump_dir /var/spool/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 http_port 3128 accel ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/myCA.pem https_port 3129 ssl-bump intercept generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/myCA.pem sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/lib/ssl_db -M 4MB acl step1 at_step SslBump1 ssl_bump peek step1 ssl_bump bump all forwarded_for transparent any ideas whats wrong? On 03.03.19 11:11, Marcus Kool wrote: Squid is an ICAP client, not an ICAP server!, and does not repond on port 1344. Marcus On 02/03/2019 22:29, steven wrote: Hi, i would like todo modifications on https connections and therefore enabled ssl bump in squid 4.4, now i would like to see the real traffic and icap looks like a way to watch and change that traffic. but squid is not answering to icap://127.0.0.1:1344 when using pyicap or telnet. the telnet error is: telnet 127.0.0.1 1344 Trying 127.0.0.1... telnet: Unable to connect to remote host: Connection refused which is imho good because it tells me that something is answering on that port after all. did i misconfigure something? config: debug_options 28,9 #icap icap_enable on icap_service service_req reqmod_precache bypass=1 icap://127.0.0.1:1344/reqmod adaptation_access service_req allow all icap_service service_resp respmod_precache bypass=0 icap://127.0.0.1:1344/respmod adaptation_access service_resp allow all acl localnet src 127.0.0.1/32 192.168.10.0/24 http_access allow localnet acl SSL_ports port 443 acl CONNECT method CONNECT #http_access deny !Safe_ports #http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager include /etc/squid/conf.d/* http_access allow localhost coredump_dir /var/spool/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 # default end # my config http_port 3128 accel ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/myCA.pem https_port 3129 ssl-bump intercept generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/myCA.pem sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/lib/ssl_db -M 4MB acl step1 at_step SslBump1 ssl_bump peek step1 ssl_bump bump all ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] icap example server not working
hi,
after going trough rfc3507 (icap) im not really sure how a icap server
is supposed to work together with squid, for example the python icap
servers that are available via github have method names like:
example_OPTIONS, example_REQMOD, example_RESPMOD
yara_OPTIONS_, yara_
or the example on this page: https://github.com/netom/pyicap
echo_OPTIONS, echo_RESPMOD
with:
icap_service service_resp respmod_precache bypass=0
icap://127.0.0.1:1344/respmod
adaptation_access service_resp allow all
127.0.0.1 - - [06/b'Mar'/2019 23:22:58] respmod_OPTIONS not found
127.0.0.1 - - [06/b'Mar'/2019 23:22:58] code 404, message b'Not Found'
with:
icap_service service_resp respmod_precache bypass=0
icap://127.0.0.1:1344/echo
adaptation_access service_resp allow all
Exception happened during processing of request from ('127.0.0.1', 46458)
Traceback (most recent call last):
File "/usr/lib/python3.7/socketserver.py", line 650, in
process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.7/socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.7/socketserver.py", line 720, in __init__
self.handle()
File
"/home/julius/code/python/icap/venv/lib/python3.7/site-packages/pyicap.py",
line 443, in handle
self.handle_one_request()
File
"/home/julius/code/python/icap/venv/lib/python3.7/site-packages/pyicap.py",
line 494, in handle_one_request
method()
File "icaptest.py", line 22, in echo_OPTIONS
self.send_headers(False)
File
"/home/julius/code/python/icap/venv/lib/python3.7/site-packages/pyicap.py",
line 326, in send_headers
icap_header_str += k + b': ' + v + b'\r\n'
TypeError: can only concatenate str (not "bytes") to str
squid config:
icap_enable on
icap_preview_enable off
icap_send_client_ip on
icap_send_client_username on
icap_service service_req reqmod_precache bypass=1
icap://127.0.0.1:1344/respmod
adaptation_access service_req allow all
#icap_service service_resp respmod_precache bypass=0
icap://127.0.0.1:1344/response
#adaptation_access service_resp allow all
why is this simple example not working?
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] icap not answering
On 05.03.19 06:13, Amos Jeffries wrote: On 5/03/19 12:10 pm, steven wrote: Ah thank you for that clarification, the python icap servers i tested so far are not very promissing but at least theres a connection now. sadly squid does not allow http access at all, only https access. Er, that would be because the only http_port you have is configured with 'accl' - making it a reverse-proxy port. But you do not have any cache_peer configured to handle that type of traffic. So, is there any particular reason you have that port receiving 'accel' / reverse-proxy mode traffic? If not remove that mode flag and things should all work for HTTP too. removed the accel mode but still no luck with http, when opening the adress: http://squid-web-proxy-cache.1019090.n4.nabble.com/http-port-with-quot-transparent-quot-or-quot-intercept-quot-td4677133.html The following error was encountered while trying to retrieve the URL: /http-port-with-quot-transparent-quot-or-quot-intercept-quot-td4677133.html <http://squid-web-proxy-cache.1019090.n4.nabble.com/http-port-with-quot-transparent-quot-or-quot-intercept-quot-td4677133.html> invalid url in this tutorial: https://www.reddit.com/r/sysadmin/comments/a67hly/squid_proxy_a_short_guide_forward_transparent/ the guy uses two ports for http like this: |http_port 3128 # Listen on this HTTP port, intercepting requests http_port 3129 intercept and then with iptables he redirects 80 to port 3129 which does not work here :( export http_proxy=http://192.168.10.215:3140 && wget google.de # im using 3140 as intercept port. config at the end. --2019-03-10 20:20:56-- http://google.de/ Connecting to 192.168.10.215:3140... connected. Proxy request sent, awaiting response... 403 Forbidden 2019-03-10 20:20:56 ERROR 403: Forbidden. | cache.log entry: 2019/03/10 20:16:20 kid1| WARNING: Forwarding loop detected for: GET / HTTP/1.1 User-Agent: Wget/1.19.4 (linux-gnu) Accept: */* Accept-Encoding: identity Via: 1.1 backup (squid/4.4) Cache-Control: max-age=259200 Connection: keep-alive Host: google.de and with: export http_proxy=http://192.168.10.215:3129 && wget google.de no cache .log entry, wget output: --2019-03-10 20:22:42-- (try: 2) http://google.de/ Connecting to 192.168.10.215:3129... connected. Proxy request sent, awaiting response... No data received. Retrying. why does my client get a 403? grep -v '#' squid.conf icap_enable off icap_preview_enable off icap_send_client_ip on icap_send_client_username on icap_service service_req reqmod_precache bypass=1 icap://127.0.0.1:1344/request adaptation_access service_req allow all icap_service service_resp respmod_precache bypass=0 icap://127.0.0.1:1344/response adaptation_access service_resp allow all acl localnet src 192.168.10.0/24 http_access allow localnet coredump_dir /var/spool/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 http_port 3128 http_port 3140 intercept https_port 3129 ssl-bump intercept generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/myCA.pem sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/lib/ssl_db -M 4MB acl step1 at_step SslBump1 ssl_bump peek step1 ssl_bump bump all ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] parameter forward_max_retries
Hello, I'm sorry for any bother to you. But we are evaluating our solutions and need author's confirmation about the parameter forward_max_retries. Could you give me a yes or no if you have clear answer? I appreciate you response very much! Thank you! If the configuration parameter of forward_max_retries is 3, for the following cases, is the assumption true or false? Case 1: FQDN has 4 IPs from DNS result: ip1, ip2, ip3, ip4. If ip1, ip2, ip3 are failed, Squid would not try ip4 since forward_max_retries is 3. Case 2: FQDN has 2 IPs from DNS result: ip1, ip2. If ip1, ip2 are failed, Squid would give final response since no more IP can try. Best Regards, Steven ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] parameter forward_max_retries
Amos, Thanks for your so detailed explanation. 😊 . I used wrong email address. Yes, we plan to have a test. But before our lab gets ready, it would be good to know the answer. Thanks again, Steven -Original Message- From: Amos Jeffries Sent: Monday, June 3, 2019 7:05 PM To: Niu, Steven (NSB - CN/Qingdao) ; squid-users@lists.squid-cache.org Cc: Liu, Hank (NSB - CN/Qingdao) Subject: Re: [squid-users] parameter forward_max_retries On 3/06/19 9:20 pm, Niu, Steven (NSB - CN/Qingdao) wrote: > Hello, > > I'm sorry for any bother to you. But we are evaluating our solutions and need > author's confirmation about the parameter forward_max_retries. Then you had best contact the author. This is a mailing list for *users* of Squid to discuss and learn about the software from each other, and from those developers like myself who participate here. AFAIK neither the developer who designed the feature, nor the developer who re-designed it to operate differently participate in this mailing list. > Could you give me a yes or no if you have clear answer? I appreciate you > response very much! Thank you! > > If the configuration parameter of forward_max_retries is 3, for the following > cases, is the assumption true or false? > Case 1: FQDN has 4 IPs from DNS result: ip1, ip2, ip3, ip4. > If ip1, ip2, ip3 are failed, Squid would not try ip4 since > forward_max_retries is 3. > Case 2: FQDN has 2 IPs from DNS result: ip1, ip2. > If ip1, ip2 are failed, Squid would give final response since no more > IP can try. > The answer depends on which version of Squid you are using, what those IPs are, what type of HTTP message is being attempted, and how long the process is taking. In regards to Squid version: * The initial design used (n_tries > forward_max_tries). Squid up to v4.0 use this. * The redesigned code uses (n_tries >= forward_max_tries). Squid v4 and later use this. => Both versions prevent destinations being contacted when their expression produces the "true" boolean result. In regards to IPs: * Squid which are built with IPv6 disabled completely ignore IPv6 results from DNS. They are not considered in the retries list. * If the failure occurs at the TCP connect(2) level, then Squid-3.1 and older do not count these as tries and will attempt all IPs forward_max_tries times. Squid-3.2 and later will count them as tries. Please also be aware that many popular websites have dozens of IPs in their DNS responses. Setting forward_max_tries less than 20 can result in these very popular websites not being contactable. In regards to HTTP message type: * HTTP standards specify some request methods as non-idempotent. Retrying these requests is forbidden. * HTTP standards define some response status codes as final and fatal. These cannot ever be retried by an intermediary. * HTTP transactions which involve certain stateful application layer features which break HTTP requirements about statelessness cannot be retried. ==> These features impact retry behaviour of the TCP connection itself. So the non-retry outcome can be inherited from some past HTTP transaction(s) activity. * Any HTTP request with a payload that has begun delivery cannot be retried. * Any HTTP response which began but did not completely arrive cannot be retried. In regards to timing: * timeouts for TCP connect action, TCP lifetimes, HTTP forwarding, and I/O actions can terminate any given tries early and cause later ones to not happen. * TCP error handling is subject to ICMP (both v4 and v6) being enabled and working on your network and on *every* network between your proxy and the origin serves (both outbound and inbound routes). -> without ICMP the TCP errors result in 'hung' connections which usually hit timeout errors before the forwarding re-tries succeed. Overall; Squid always give a final response when that final response is known. Where that response comes from is far more complicated than your stated cases consider. Have you tried testing for yourself the particular sites you are worried about? Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
