[squid-users] Adding an extra header to TLS connection

[Robin Wood]

Hi
I'm trying to work out how to add an extra header to a TLS connection.

I've found information on how to do it on what I think is the pre-3.5
release, but I can't find any useful information on doing it on the current
version.

Could someone give me an example or point me at some documentation on how
to do it.

Thanks

Robin
___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Adding an extra header to TLS connection

[Robin Wood]

On Thu, 23 May 2024 at 18:00, Jonathan Lee  wrote:

> I do use ssl bump again it requires certificates installed on the devices,
> and or some and a splice for the others. You must also add a url list for
> items that must never be intercepted like banks etc. I agree it is not an
> easy task, it took me years to get it to work correctly for what I needed.
> When it does work it works beautifully, you can cache updates and reuse
> them, you can use clam AV on https traffic. It’s not for everyone it will
> make you a wizard level 1000 if you can get it going.
>

Jonathan, can you give me an example of it working?

Oddly, you are replying to a message from Alex that I never received.

Alex, in answer to your questions...

I'm doing some testing against a client's site, they require a custom
header to allow my connections through their WAF. I could try to do this
manually with all my tools, but it would be easier to just have Squid do it
for me and then have the tools use Squid as their proxy. I can tell them to
not do cert checking or I can use my own CA and import it into the system
store, that is not a problem.

I've tried searching for Squid and sslbump and not found anything useful
that works with the current version, that is why I'm asking here, I was
hoping someone could point me at an example that would definitely work with
the current version of Squid.

Robin


> Sent from my iPhone
>
> > On May 23, 2024, at 08:49, Alex Rousskov <
> rouss...@measurement-factory.com> wrote:
> >
> > On 2024-05-22 03:49, Robin Wood wrote:
> >
> >> I'm trying to work out how to add an extra header to a TLS connection.
> >
> > I assume that you want to add a header field to an HTTP request or
> response that is being transmitted inside a TLS connection between a TLS
> client (e.g., a user browser) and an HTTPS origin server.
> >
> > Do you control the client that originates that TLS connection (or its
> OS/environment) or the origin server? If you do not, then what you want is
> impossible -- TLS encryption exists, in part, to prevent such traffic
> modifications.
> >
> > If you control the client that originates that TLS connection (or its
> OS/environment), then you may be able to, in _some_ cases, add that header
> by configuring the client (or its OS/environment) to trust you as a
> Certificate Authority, minting your own X509 certificates, and configuring
> Squid to perform a "man in the middle" attack on client-server traffic,
> using your minted certificates. You can search for Squid SslBump to get
> more information about this feature, but the area is full of insurmountable
> difficulties and misleading advice. Avoid it if at all possible!
> >
> >
> > HTH,
> >
> > Alex.
> >
> >
> >> I've found information on how to do it on what I think is the pre-3.5
> release, but I can't find any useful information on doing it on the current
> version.
> >> Could someone give me an example or point me at some documentation on
> how to do it.
> >> Thanks
> >> Robin
> >> ___
> >> squid-users mailing list
> >> squid-users@lists.squid-cache.org
> >> https://lists.squid-cache.org/listinfo/squid-users
> >
> > ___
> > squid-users mailing list
> > squid-users@lists.squid-cache.org
> > https://lists.squid-cache.org/listinfo/squid-users
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] test ICAP server

[Robin Wood]

I'm more looking at things like always adding extra headers such as auth
tokens. I want to create some scripts that will go and get things like JWTs
so I can run dumb tools through them and not have to worry about doing it
by hand and then getting the tool to accept extra headers.

And I know there are other ways to do this, but this is more a curiosity
project now, especially as it didn't work first time and there is a new
protocol to learn.

Robin

On Fri, 7 Feb 2025, 21:12 Amos Jeffries,  wrote:

> On 8/02/25 04:35, Robin Wood wrote:
> > Hi
> > I wouldn't risk trying to do a production quality one server!
> >
> > I'm a security tester and I want a way to automatically modify traffic
> > that I'm sending to and from sites. I've got plenty of other ways to do
> > it, but as all my testing traffic already goes through a Squid box I
> > just wanted to have a play to see if I could get it to do simple
> > things like add a new header or something like that.
>
> Please be aware that Squid normalizes and performs security sanitization
> on the HTTP messages that it receives. Regardless of whether they are
> arrive from client, server, or ICAP. That means a lot of traffic
> malformation needed for proper security tests will not work at all.
>
>
> HTH
> Amos
>
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users

[squid-users] test ICAP server

[Robin Wood]

Hi
I want to write my own ICAP server, but all the sample code I've found is
failing, the majority with this error from squidclient:

squidclient -h 192.168.0.7 -p 1344 icap://192.168.0.7:1344/request_mod
ICAP/1.0 400 Bad Request

And this error when accessed through Squid itself:

ICAP protocol error.

The system returned: [No Error]

This means that some aspect of the ICAP communication failed.

I've got this in my squid.conf:

# Enable ICAP for request and response modification
icap_enable on
icap_service service_req reqmod_precache icap://192.168.0.7:1344/request_mod
icap_service service_resp respmod_precache icap://
192.168.0.7:1344/response_mod

# ICAP adaptation rules
adaptation_access service_req allow all
adaptation_access service_resp allow all

And I've tried running both Python and Ruby servers taken from various
places. A lot are logging that they've received and replied to requests:

Received request: OPTIONS icap://192.168.0.7:1344/request_mod ICAP/1.0
Responded to OPTIONS request

The servers are all listening, I've checked that:

$ ss -antp | grep 1344
LISTEN 0  4096   0.0.0.0:1344
 0.0.0.0:* users:(("ruby",pid=563029,fd=5))

The closest I've got is this one:

https://gist.github.com/digininja/a98b3567e0aeb218d37cdd337bb12f34

It seems to be working as it sees the headers and says it has returned data:

Received request: REQMOD icap://192.168.0.7:1344/request_mod ICAP/1.0
Original Headers:
Host: 192.168.0.7:1344
Date: Fri, 07 Feb 2025 10:10:30 GMT
Encapsulated: req-hdr=0, null-body=91
Preview: 0
Allow: 204, trailers
Modified request sent back

But I still get errors.

What could be wrong?

Does someone have some example server code they know works that I can test
to help debug it?

Robin
___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] test ICAP server

[Robin Wood]

I've got a bit further with this and I'm starting to understand ICAP talk.

I've got a Go server working which I can talk to through netcat:

$ nc localhost 1344
REQMOD icap://192.168.0.7/ ICAP/1.0
GET / HTTP/1.1
Host: dvwa.test
Accept: text/html, text/plain, image/gif
Accept-Encoding: gzip, compress

ICAP/1.0 200 OK
Encapsulated: req-hdr=0, null-body=141

GET / HTTP/1.1
Host: dvwa.test
Accept: text/html, text/plain, image/gif
Accept-Encoding: gzip, compress
X-Modified-By: Go ICAP Server


On the server side I see:

This is the request

REQMOD icap://192.168.0.7/ ICAP/1.0

Received request: REQMOD icap://192.168.0.7/ ICAP/1.0
Original Request Headers:
GET / HTTP/1.1
Host: dvwa.test
Accept: text/html, text/plain, image/gif
Accept-Encoding: gzip, compress


Modified request sent back

ICAP/1.0 200 OK
Encapsulated: req-hdr=0, null-body=141

GET / HTTP/1.1
Host: dvwa.test
Accept: text/html, text/plain, image/gif
Accept-Encoding: gzip, compress
X-Modified-By: Go ICAP Server


If I try to use curl through Squid pointing at that server:

$ curl "http://dvwa.test/a?xsa"; --proxy localhost:3128

dvwa.test is at 192.168.0.42.

I get this in the ICAP server:

This is the request

REQMOD icap://192.168.0.7:1344/ ICAP/1.0

Received request: REQMOD icap://192.168.0.7:1344/ ICAP/1.0
Original Request Headers:
Host: 192.168.0.7:1344
Date: Fri, 07 Feb 2025 12:26:25 GMT
Encapsulated: req-hdr=0, null-body=93
Allow: 204, trailers


Modified request sent back

ICAP/1.0 200 OK
Encapsulated: req-hdr=0, null-body=155

Host: 192.168.0.7:1344
Date: Fri, 07 Feb 2025 12:26:25 GMT
Encapsulated: req-hdr=0, null-body=93
Allow: 204, trailers
X-Modified-By: Go ICAP Server

Squid then gives the same error as before, ERR_ICAP_FAILURE.

Robin

On Fri, 7 Feb 2025 at 10:15, Robin Wood  wrote:

> Hi
> I want to write my own ICAP server, but all the sample code I've found is
> failing, the majority with this error from squidclient:
>
> squidclient -h 192.168.0.7 -p 1344 icap://192.168.0.7:1344/request_mod
> ICAP/1.0 400 Bad Request
>
> And this error when accessed through Squid itself:
>
> ICAP protocol error.
>
> The system returned: [No Error]
>
> This means that some aspect of the ICAP communication failed.
>
> I've got this in my squid.conf:
>
> # Enable ICAP for request and response modification
> icap_enable on
> icap_service service_req reqmod_precache icap://
> 192.168.0.7:1344/request_mod
> icap_service service_resp respmod_precache icap://
> 192.168.0.7:1344/response_mod
>
> # ICAP adaptation rules
> adaptation_access service_req allow all
> adaptation_access service_resp allow all
>
> And I've tried running both Python and Ruby servers taken from various
> places. A lot are logging that they've received and replied to requests:
>
> Received request: OPTIONS icap://192.168.0.7:1344/request_mod ICAP/1.0
> Responded to OPTIONS request
>
> The servers are all listening, I've checked that:
>
> $ ss -antp | grep 1344
> LISTEN 0  4096   0.0.0.0:1344
>  0.0.0.0:* users:(("ruby",pid=563029,fd=5))
>
> The closest I've got is this one:
>
> https://gist.github.com/digininja/a98b3567e0aeb218d37cdd337bb12f34
>
> It seems to be working as it sees the headers and says it has returned
> data:
>
> Received request: REQMOD icap://192.168.0.7:1344/request_mod ICAP/1.0
> Original Headers:
> Host: 192.168.0.7:1344
> Date: Fri, 07 Feb 2025 10:10:30 GMT
> Encapsulated: req-hdr=0, null-body=91
> Preview: 0
> Allow: 204, trailers
> Modified request sent back
>
> But I still get errors.
>
> What could be wrong?
>
> Does someone have some example server code they know works that I can test
> to help debug it?
>
> Robin
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] test ICAP server

[Robin Wood]

Hi
I wouldn't risk trying to do a production quality one server!

I'm a security tester and I want a way to automatically modify traffic that
I'm sending to and from sites. I've got plenty of other ways to do it, but
as all my testing traffic already goes through a Squid box I just wanted to
have a play to see if I could get it to do simple things like add a new
header or something like that.  When I saw some really simple servers
written in Python and Go I thought it would be a fun bit of learning.

I've just grabbed a copy of c-icap, the build instructions could do with
some work, I've got it built, but it wasn't obvious what needed doing. I'll
have a play, it won't be as flexible as I wanted, but I'll hopefully learn
something, and that might help debug the other servers.

Robin

On Fri, 7 Feb 2025 at 14:56, Alex Rousskov 
wrote:

> On 2025-02-07 05:15, Robin Wood wrote:
>
> > I want to write my own ICAP server
>
> FWIW, nearly all attempts to quickly write a production-quality ICAP
> server (that I have seen) have failed. The protocol is much more complex
> than it seems. In most cases, folks looking for a free ICAP server
> should be reusing existing c-icap.
>
>
> > but all the sample code I've found
> > is failing, the majority with this error from squidclient:
>
> squidclient does not really support ICAP.
>
>
> > Does someone have some example server code they know works that I can
> > test to help debug it?
>
> Consider c-icap: https://c-icap.sourceforge.net/
>
>
> HTH,
>
> Alex.
>
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Thoughts on caching aspx jsp asp cgi-bin

[Robin Wood]

I would say that it depends on what the dynamic content is. If it is public
content from a CMS and you are OK with it potentially being your cache age
out of date, then caching it rather than reloading it from the database
every page load is fine. If the pages are for anything sensitive, for
example a user's account, then definitely do not cache it.

Robin

On Tue, 31 Dec 2024 at 17:55, Jonathan Lee  wrote:

> What are your thoughts? This is in relation to ssl intercept with
> certificates installed and bump active.
>
> Keep in mind I am still a student and learning.
> Is a rule like this recommended? Does anyone have a better version of
> this?
> Sent from my iPhone
>
> On Dec 30, 2024, at 14:10, Jonathan Lee  wrote:
>
> Hello fellow Squid Users,
>
> Can you please help?
>
> What are your thoughts on this rule? Should cgi-bin aspx and jsp files be
> excluded from the web-cache? They are dynamic correct? This could help
> speed up systems right?
>
>
> acl QUERY urlpath_regex cgi-bin \? asp aspx jsp
>
>
> ## Prevent caching jsp, cgi-bin etc
>
> cache deny QUERY
>
> Ref:
> Setting up Explicit Squid Proxy
> 
> wiki.alpinelinux.org
> 
> 
> 
> 
>
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Thoughts on caching aspx jsp asp cgi-bin

[Robin Wood]

I'm going to massively over simplify things here, but you can think of it
like this.

Files with html extensions are static web pages, you write them, put them
on the server, and they are served as they are, no changes.

Asp and the others are dynamic files, they are processed by an app on the
server before they are sent to the client. This app may do nothing, so the
page comes as it was, but usually it will add content. This content could
be to create a CMS page by pulling the page content from a database, it
could be your shopping orders pulled from your account, or it could be your
current bank statement.

Caching should never be done on anything that is specific to a single user,
so it's fine to cache public CMS content with an asp extension, but not
your bank statement.

There is more to it than that, but hopefully that gives you a general idea.

Robin

On Tue, 31 Dec 2024, 23:07 Jonathan Lee,  wrote:

> Thanks I have to admit I am a student currently,
>
> I guess my last question is what do cgi-bin asp aspx asp files do inside
> of websites? The reason I ask this is the rule below I have seen the same
> rule on different websites so it must be the main ones that cause issues.
> Is it just for dynamic content?
>
> On Dec 31, 2024, at 14:47, Robin Wood  wrote:
>
> I would say that it depends on what the dynamic content is. If it is
> public content from a CMS and you are OK with it potentially being your
> cache age out of date, then caching it rather than reloading it from the
> database every page load is fine. If the pages are for anything sensitive,
> for example a user's account, then definitely do not cache it.
>
> Robin
>
> On Tue, 31 Dec 2024 at 17:55, Jonathan Lee 
> wrote:
>
>> What are your thoughts? This is in relation to ssl intercept with
>> certificates installed and bump active.
>>
>> Keep in mind I am still a student and learning.
>> Is a rule like this recommended? Does anyone have a better version of
>> this?
>> Sent from my iPhone
>>
>> On Dec 30, 2024, at 14:10, Jonathan Lee  wrote:
>>
>> Hello fellow Squid Users,
>>
>> Can you please help?
>>
>> What are your thoughts on this rule? Should cgi-bin aspx and jsp files be
>> excluded from the web-cache? They are dynamic correct? This could help
>> speed up systems right?
>>
>>
>> acl QUERY urlpath_regex cgi-bin \? asp aspx jsp
>>
>> ## Prevent caching jsp, cgi-bin etc
>> cache deny QUERY
>>
>> Ref:
>> Setting up Explicit Squid Proxy
>> <https://wiki.alpinelinux.org/wiki/Setting_up_Explicit_Squid_Proxy>
>> wiki.alpinelinux.org
>> <https://wiki.alpinelinux.org/wiki/Setting_up_Explicit_Squid_Proxy>
>> 
>> <https://wiki.alpinelinux.org/wiki/Setting_up_Explicit_Squid_Proxy>
>> <https://wiki.alpinelinux.org/wiki/Setting_up_Explicit_Squid_Proxy>
>>
>> ___
>> squid-users mailing list
>> squid-users@lists.squid-cache.org
>> https://lists.squid-cache.org/listinfo/squid-users
>>
>
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
https://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] StoreID Question

[Robin Wood]

I've not got time to read your whole email, but you are asking about
regular expressions.

^http:\/\/[^\.]+\.dl\.sourceforge\.net\/(.*)
http://dl.sourceforge.net.squid.internal/$1

What this means is to match the first URL and "capture" the bit at the end,
the bit in brackets. This then gets rewritten to the second URL with the
captured bit added on to the end, that is $1. If you captured two things in
brackets the first would be $1, the second $2.

Do some reading on regex and regular expressions. The basics are relatively
easy to understand, beyond that, it can get very complicated very quickly.

Robin

On Tue, 31 Dec 2024, 23:05 Jonathan Lee,  wrote:

> *Hello Fellow Squid Users,*
>
> *Can you please help? I have been researching this for a long time and
> cannot find any information on this "what is the $ mean” within StoreID?*
>
> *Below is my failed attempt to make StoreID work correctly. Sorry it's a
> mess. I have since disabled my customized StoreID patterns because it
> caused issues. My question is with regard to the $number part of the
> program. I disabled all the facebook and all my tests because my photos
> where showing up wrong and it would duplicate itself over everything, I
> would have to clear the cache and change items and try again below is my
> failed attempt to get it to work correctly. *
>
> *It did work sometimes however I would get issues the longer it went on
> for. I decided to stop the trial and testing of it because it was driving
> me crazy. It is a great puzzle to solve. Does anyone have any tips? I have
> some Squid text books like the Squid the definitive guide, and The Squid
> Proxy Server 3.1 guide still nothing really explains StoreID outside of the
> Squid website. Yes the website comes with a great database that does work,
> I tested some database items with Ubuntu updates inside of VMs and it
> worked and reserved them to other machines asking for the same update. So
> in my quest I thought can I also do this with Facebook… (I do not recommend
> you try it) or something else, Youtube. *
>
> *This is the Text file I have been testing and it was a failed test
> outside of Ubuntu updates however I do not use that OS anymore so it is
> removed I think I had the $ wrong I have no info on what it does some are 1
> some are 5 some are doubles $ and another of them:*
>
> *^https?:\/\/(fbcdn|scontent).*(akamaihd|fbcdn)\.net\/.*\/v\/.*\/(.*\.mp4)
> http://facebook.squid.internal/$3 *
> *^https?:\/\/fbcdn\-(static|profile)\-a\.akamaihd\.net\/static\-ak\/rsrc\.php\/((?!.*\.(?:js|css|swf)).*)
> http://facebook.squid.internal/static/$2
> *
> *^https?:\/\/(fbcdn|scontent).*(akamaihd|fbcdn)\.net\/(h|s)(profile|photos).*\/(.*\.(png|gif|jpg))(\?.+)?
> http://facebook.squid.internal/$5 *
> *^https?:\/\/fbstatic\-a\.akamaihd\.net\/rsrc\.php\/((?!.*\.(?:js|css|swf)).*)
> http://facebook.squid.internal/static/$1
> *
> *^http:\/\/.*[steampowered|steamcontent]\.com\/([^?]*)
> http://steamupdates.squid.internal/$1
> *
> *^https?\:\/\/download\.oracle\.com\/((otn\-pub|otn)\/[\d\w]+\/[\d\w]+\/[\w\d\-]+\/[\w\d\-]+\.(exe|dmg|rpm|msi|tar\.(gz|Z)))\?
> http://java.oracle.otn.ngtech.squid.internal/$1
> *
> *^https?\:\/\/([\d\w\-]+)\.oracle\.com\/(([\d\w]+)\/[\d\w]+\/[\d\w]+\/([\d\w\-]+)\/([\d\w]+\/)?[\d\w\-\.\_]+\.(dmg|msi|exe|tar\.gz|tar\.Z))\?
> http://java.oracle.download.ngtech.squid.internal/$2
> *
> *^http:\/\/[^\.]+\.phobos\.apple\.com\/(.*)
> http://appupdates.apple.squid.internal/$1
> *
> *^http:\/\/[^\.]+\.c\.android\.clients\.google\.com\/(.*)
> http://androidupdates.google.squid.internal/$1
> *
>
> *My question here is:*
> *What does this $3 mean within the the store id program?*
>
> *This is the config and refresh patterns that I was learning with for
> Squid StoreID. Much of it is “#”ed out but this is what I was using:*
>
> *#store_id_program /usr/local/libexec/squid/storeid_file_rewrite
> /var/squid/storeid/storeid_rewrite.txt*
> *#store_id_children 10 startup=5 idle=1 concurrency=0*
> *#always_direct allow all*
> *#store_id_access deny connect*
> *#store_id_access deny !getmethod*
> *#store_id_access allow rewritedoms*
> *#store_id_access deny all*
>
> *refresh_all_ims on*
> *reload_into_ims on*
> *max_stale 20 years*
> *minimum_expiry_time 0*
>
> *#refresh_pattern -i ^http.*squid\.internal.* 43200 100% 79900
> override-expire override-lastmod ignore-reload ignore-no-store
> ignore-must-revalidate ignore-private ignore-auth*
>
> *#FACEBOOK*
> *#refresh_pattern ^https.*.facebook.com/*  10080
> 80% 43200*
>
> *#FACEBOOK IMAGES  *
> *#refresh_pattern -i pixel.facebook.com..(jpg|pn