Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

[Klaus Brandl]

On Wednesday 29 July 2020 14:50:11 Amos Jeffries wrote:
> On 29/07/20 1:07 pm, Contato - KONNTROL wrote:
> > Hello Everyone,
> > Greetings.
> > 
> > Background:
> > OS - FreeBSD 12.1
> > SQUID ver 4.10
> > OpenSSL 1.0.2u
> > 
> > I am trying to use SQUID in front of E2Guardian (content filter) with the
> > following configuration at the SQUID side.
> > 
> > ###
> > cache_peer 127.0.0.1 parent 8080 0 login=*:password
> > client_persistent_connections on
> > always_direct deny all
> > never_direct allow all
> > ###
> > 
> > It works fine till the point  SQUID exhausts all E2Guardian
> > threads/workers, no matter the amount you set. If 1000, SQUID is opening
> > 1000 connections. If 10.000, squid also opens 10.000 connections.
> > I tried the directive "client_persistent_connections on and off" with no
> > success.
> > Even using a single browser for testing purposes, for some reason SQUID
> > opens thousands of connections against the E2guardian.
> > I did a wireshark capture to "see" what is  happening and it seems like a
> > lot of ACK/SYN with no payload.
> > 
> > Any idea? Maybe I am using a wrong configuration.
> 
> You are. BUT, I think you have a forwarding loop happening so the
> correct config for limiting connections will not help.
> 
> You should be able to test for loops by enabling the Via header. If your
> squid.conf contains "via off" remove that line. Assuming e2g is not
> removing that header Squid will reject loops with an error message.

Setting an other "visible_hostname" may also help.

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen
tel +49 89 991950-0, fax -999, www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch
Amtsgericht Muenchen HRB 98238
genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

[Amos Jeffries]

On 29/07/20 8:29 pm, Klaus Brandl wrote:
> 
> Setting an other "visible_hostname" may also help.
> 

Why do you think the hostname has any relation to the problem?

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

[Klaus Brandl]

On Wednesday 29 July 2020 23:03:43 Amos Jeffries wrote:
> On 29/07/20 8:29 pm, Klaus Brandl wrote:
> > Setting an other "visible_hostname" may also help.
> 
> Why do you think the hostname has any relation to the problem?

because we had also a forwarding loop by connection 2 squids on the same host 
together via a parent statement. Then in the via header there was the same 
hostname 2 times, and this issued squid to detect a forwarding loop.
Setting an other visible_hostname on one of the squids solved this problem.

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen
tel +49 89 991950-0, fax -999, www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch
Amtsgericht Muenchen HRB 98238
genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

[Matus UHLAR - fantomas]

On 29/07/20 8:29 pm, Klaus Brandl wrote:
> Setting an other "visible_hostname" may also help.



On Wednesday 29 July 2020 23:03:43 Amos Jeffries wrote:

Why do you think the hostname has any relation to the problem?


On 29.07.20 13:58, Klaus Brandl wrote:

because we had also a forwarding loop by connection 2 squids on the same host
together via a parent statement. Then in the via header there was the same
hostname 2 times, and this issued squid to detect a forwarding loop.
Setting an other visible_hostname on one of the squids solved this problem.


there's a specific setting "unique_hostname" for these cases.

#  TAG: unique_hostname
#   If you want to have multiple machines with the same
#   'visible_hostname' you must give each machine a different
#   'unique_hostname' so forwarding loops can be detected.
#Default:
# Copy the value from visible_hostname


I just wonder why is this not set to $HOSTNAME by default, so setting same
visible_hostname on different servers would not require setting different
unique_hostname

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
The only substitute for good manners is fast reflexes.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

[Amos Jeffries]

On 29/07/20 11:58 pm, Klaus Brandl wrote:
> On Wednesday 29 July 2020 23:03:43 Amos Jeffries wrote:
>> On 29/07/20 8:29 pm, Klaus Brandl wrote:
>>> Setting an other "visible_hostname" may also help.
>>
>> Why do you think the hostname has any relation to the problem?
> 
> because we had also a forwarding loop by connection 2 squids on the same host 
> together via a parent statement. Then in the via header there was the same 
> hostname 2 times, and this issued squid to detect a forwarding loop.
> Setting an other visible_hostname on one of the squids solved this problem.>

In your situation the parent proxy was wrongly reporting loops since it
saw its own name coming out of the child proxy. That is not a real loop,
just a misconfiguration on your part to begin with.

NP: Klaus, unique_hostname is probably a better solution to your
problem. That lets loop detection work properly but both proxies send
URLs containing the shared visible_hostname to clients when they need to
reference proxy resources.


For this thread the OP has only one Squid and it is first in the proxy
chain.

For a loop to happen the peer must already be accepting traffic from
Squid with its current visible/public hostname. Only after that the
traffic might loop back to Squid to begin another circle. So setting the
Squid hostname to a different value will not stop any real loops, only
alter the string placed in the Via header each cycle.

Right now in the troubleshooting we are trying to get loops to show up
to see whether that is the hidden problem.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

[Klaus Brandl]

On Thursday 30 July 2020 00:24:52 Amos Jeffries wrote:
> On 29/07/20 11:58 pm, Klaus Brandl wrote:
> > On Wednesday 29 July 2020 23:03:43 Amos Jeffries wrote:
> >> On 29/07/20 8:29 pm, Klaus Brandl wrote:
> >>> Setting an other "visible_hostname" may also help.
> >> 
> >> Why do you think the hostname has any relation to the problem?
> > 
> > because we had also a forwarding loop by connection 2 squids on the same
> > host together via a parent statement. Then in the via header there was
> > the same hostname 2 times, and this issued squid to detect a forwarding
> > loop. Setting an other visible_hostname on one of the squids solved this
> > problem.>
> In your situation the parent proxy was wrongly reporting loops since it
> saw its own name coming out of the child proxy. That is not a real loop,
> just a misconfiguration on your part to begin with.
> 
> NP: Klaus, unique_hostname is probably a better solution to your
> problem. That lets loop detection work properly but both proxies send
> URLs containing the shared visible_hostname to clients when they need to
> reference proxy resources.

ok, thank you, i will use this next time we have this problem.

> 
> 
> For this thread the OP has only one Squid and it is first in the proxy
> chain.

i saw the "cache_peer 127.0.0.1 parent 8080..." entry and it remembered me to 
our problem, sorry.

> 
> For a loop to happen the peer must already be accepting traffic from
> Squid with its current visible/public hostname. Only after that the
> traffic might loop back to Squid to begin another circle. So setting the
> Squid hostname to a different value will not stop any real loops, only
> alter the string placed in the Via header each cycle.
> 
> Right now in the troubleshooting we are trying to get loops to show up
> to see whether that is the hidden problem.
> 
> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen
tel +49 89 991950-0, fax -999, www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch
Amtsgericht Muenchen HRB 98238
genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

[Contato - KONNTROL]

Thanks Amos, Klaus and Fantomas.
I already send another message with a lot of info and an attachment, that is 
now under moderator's review. If blocked, I will delete the attachment and send 
it again.

Thanks
Fabricio.


-Original Message-
From: squid-users  On Behalf Of 
Klaus Brandl
Sent: Wednesday, July 29, 2020 9:37 AM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] SQUID with cache_peer config + E2guardian - too many 
connections

On Thursday 30 July 2020 00:24:52 Amos Jeffries wrote:
> On 29/07/20 11:58 pm, Klaus Brandl wrote:
> > On Wednesday 29 July 2020 23:03:43 Amos Jeffries wrote:
> >> On 29/07/20 8:29 pm, Klaus Brandl wrote:
> >>> Setting an other "visible_hostname" may also help.
> >> 
> >> Why do you think the hostname has any relation to the problem?
> > 
> > because we had also a forwarding loop by connection 2 squids on the 
> > same host together via a parent statement. Then in the via header 
> > there was the same hostname 2 times, and this issued squid to detect 
> > a forwarding loop. Setting an other visible_hostname on one of the 
> > squids solved this problem.>
> In your situation the parent proxy was wrongly reporting loops since 
> it saw its own name coming out of the child proxy. That is not a real 
> loop, just a misconfiguration on your part to begin with.
> 
> NP: Klaus, unique_hostname is probably a better solution to your 
> problem. That lets loop detection work properly but both proxies send 
> URLs containing the shared visible_hostname to clients when they need 
> to reference proxy resources.

ok, thank you, i will use this next time we have this problem.

> 
> 
> For this thread the OP has only one Squid and it is first in the proxy 
> chain.

i saw the "cache_peer 127.0.0.1 parent 8080..." entry and it remembered me to 
our problem, sorry.

> 
> For a loop to happen the peer must already be accepting traffic from 
> Squid with its current visible/public hostname. Only after that the 
> traffic might loop back to Squid to begin another circle. So setting 
> the Squid hostname to a different value will not stop any real loops, 
> only alter the string placed in the Via header each cycle.
> 
> Right now in the troubleshooting we are trying to get loops to show up 
> to see whether that is the hidden problem.
> 
> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen tel +49 89 991950-0, fax -999, 
www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch Amtsgericht Muenchen HRB 98238 
genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

[Contato - KONNTROL]

Hello Amos/Klaus/Fantomas,
Thanks for your help.

I have searched for the "via off"  and "via on" - The directive was not present 
on the config file but I am assuming "via on" it's the default option. Anyway I 
added it to the config file. No Success.
I also checked the visible_hostname, as suggested by Klaus (Thanks Klaus for 
the help!!) . It was set to localhost, then I changed to something different. 
No success as well.

So, I decided to make another test using Wireshark. For that, I put the squid 
down before starting the capture, then started Squid after capturing.
What I can see is squid sending thousands of requests (like a machine gun) 
against the E2G (loopback interface on port 8080) with thousands of "408 
Request Time Out" entries.
I also see the following HTTP GET: 
"http://127.0.0.1:8080/squid-internal-dynamic/netdb";  by the way,  8080 is E2G 
port. Not sure what it is. The 408 above are probably because of this calling.

Attached you can see the capture file, just in case you have wireshark or any 
other software able to read .cap file. Don't worry, there is nothing 
confidential on the file. That's a LAB environment.
That is really confusing me.

Thank You very Much!

Regards
Fabricio.



-Original Message-
From: squid-users  On Behalf Of Amos 
Jeffries
Sent: Tuesday, July 28, 2020 11:50 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] SQUID with cache_peer config + E2guardian - too many 
connections

On 29/07/20 1:07 pm, Contato - KONNTROL wrote:
> Hello Everyone,
> Greetings.
> 
> Background:
> OS - FreeBSD 12.1
> SQUID ver 4.10
> OpenSSL 1.0.2u
> 
> I am trying to use SQUID in front of E2Guardian (content filter) with 
> the following configuration at the SQUID side.
> 
> ###
> cache_peer 127.0.0.1 parent 8080 0 login=*:password 
> client_persistent_connections on always_direct deny all never_direct 
> allow all ###
> 
> It works fine till the point  SQUID exhausts all E2Guardian 
> threads/workers, no matter the amount you set. If 1000, SQUID is 
> opening 1000 connections. If 10.000, squid also opens 10.000 connections.
> I tried the directive "client_persistent_connections on and off" with 
> no success.
> Even using a single browser for testing purposes, for some reason 
> SQUID opens thousands of connections against the E2guardian.
> I did a wireshark capture to "see" what is  happening and it seems 
> like a lot of ACK/SYN with no payload.
> 
> Any idea? Maybe I am using a wrong configuration.
> 

You are. BUT, I think you have a forwarding loop happening so the correct 
config for limiting connections will not help.

You should be able to test for loops by enabling the Via header. If your 
squid.conf contains "via off" remove that line. Assuming e2g is not removing 
that header Squid will reject loops with an error message.


Check that the traffic leaving e2g is not going back into Squid. With the setup 
described e2g should be connecting directly to upstream/Internet servers - it 
should have no settings about Squid except those for processing the 
X-Forwarded-For header.

If you are intercepting traffic to deliver it to Squid make sure the 
connections leaving e2g are not being caught by those firewall rules.


If you are certain there is no loop the cache_peer max-conn=N is the way to 
limit the connections made to a peer. This will only help if the problem is 
high traffic flow. It will not help if there is a forwarding loop happening.


> By the way, I am using SQUID in front of E2Guardian cause I use 
> Kerberos authentication (not supported by E2guardian) with FORWARDX option 
> enable.
> 

Sure. You may want to look at the features of e2g you are using and see whether 
Squid can do them instead. The idea there being to make deny decisions early as 
possible to minimize the total amount of processing work those transactions 
consume.
 You may find you can get rid of e2g entirely, which will improve overall 
performance and reduce management headaches from layers of proxy.


Cheers
Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
<>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

[Klaus Brandl]

tried to add the option "no-netdb-exchange" on your cache_peer line?

On Wednesday 29 July 2020 11:50:20 Contato - KONNTROL wrote:
> Hello Amos/Klaus/Fantomas,
> Thanks for your help.
> 
> I have searched for the "via off"  and "via on" - The directive was not
> present on the config file but I am assuming "via on" it's the default
> option. Anyway I added it to the config file. No Success. I also checked
> the visible_hostname, as suggested by Klaus (Thanks Klaus for the help!!) .
> It was set to localhost, then I changed to something different. No success
> as well.
> 
> So, I decided to make another test using Wireshark. For that, I put the
> squid down before starting the capture, then started Squid after capturing.
> What I can see is squid sending thousands of requests (like a machine gun)
> against the E2G (loopback interface on port 8080) with thousands of "408
> Request Time Out" entries. I also see the following HTTP GET:
> "http://127.0.0.1:8080/squid-internal-dynamic/netdb";  by the way,  8080 is
> E2G port. Not sure what it is. The 408 above are probably because of this
> calling.
> 
> Attached you can see the capture file, just in case you have wireshark or
> any other software able to read .cap file. Don't worry, there is nothing
> confidential on the file. That's a LAB environment. That is really
> confusing me.
> 
> Thank You very Much!
> 
> Regards
> Fabricio.
> 
> 

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen
tel +49 89 991950-0, fax -999, www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch
Amtsgericht Muenchen HRB 98238
genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid and multipart form decode

[Ryan Le]

Even though it looks like TeChunkedParser is getting all the
additional headers I can't seem to create ACL or output them using
logformat. I was trying to request these headers with
req_mime_type/resp_mime_type. and alos had log_mime_hdrs on and then in
logformat just had all.

On Thu, Jul 23, 2020 at 11:46 AM Ryan Le  wrote:

> Thanks,
>
> I have been looking at the squid debug and can see that it is getting the
> multipart.
>
> POST http://bb.com
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0)
> Gecko/20100101 Firefox/78.0
> Accept: application/json
> Accept-Language: en-US,en;q=0.5
> Accept-Encoding: gzip, deflate
> Referer: http://b.com
> Cache-Control: no-cache
> X-Requested-With: XMLHttpRequest
> Content-Type: multipart/form-data;
> boundary=---328901485836611227811186534509
> Content-Length: 1245
> Origin: http://b.com
> Cookie: cookie
> Host: bbb.com
> Via: ICAP/1.0
>
> 4dd
> -328901485836611227811186534509
> Content-Disposition: form-data; name="action"
>
> frm_submit_dropzone
> -328901485836611227811186534509
> Content-Disposition: form-data; name="field_id"
>
> 8
> -328901485836611227811186534509
> Content-Disposition: form-data; name="form_id"
>
> 5
> -328901485836611227811186534509
> Content-Disposition: form-data; name="nonce"
>
> e1aca92777
> -328901485836611227811186534509
> Content-Disposition: form-data; name="file8"; filename="translate.zip"
> Content-Type: application/x-zip-compressed
>
> On Thu, Jul 23, 2020 at 11:16 AM Alex Rousskov <
> rouss...@measurement-factory.com> wrote:
>
>> On 7/23/20 9:22 AM, Ryan Le wrote:
>> > I have been trying to configure squid to decode and send multipart form
>> > data to another service. Is there an acl or build parameter needed for
>> > multipart form data support?
>>
>> No, there is no need to allow any specific Content-Type, including
>> multipart. Squid does not know anything about multipart/form-data. If a
>> multipart/form-data message is well-formed from HTTP point of view, then
>> Squid will process it as any other message, including passing it to
>> ICAP/eCAP (where configured).
>>
>> Cheers,
>>
>> Alex.
>>
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid and multipart form decode

[Alex Rousskov]

On 7/29/20 11:38 AM, Ryan Le wrote:
> Even though it looks like TeChunkedParser is getting all the
> additional headers 

TeChunkedParser has nothing to do with multipart/form-data bodies.
TeChunkedParser parses chunked encoding, and even then it is applied to
remove _transfer_ encoding, not to interpret the actual resource content
inside the chunks.

I am not sure, but it looks like you have pasted a part of an ICAP
message. TeChunkedParser is used to parse chunked transfer encoding used
for a part of the ICAP message body. Beyond decoding those chunks, it is
all opaque data to Squid.

To avoid misunderstanding, in your pasted example, the contents of the
first chunk starts with these two lines:

> -328901485836611227811186534509
> Content-Disposition: form-data; name="action"

It does _not_ start with the "Content-Disposition:..." line or the
"frm_submit_dropzone" line.


> I can't seem to create ACL or output them using
> logformat. I was trying to request these headers with
> req_mime_type/resp_mime_type. 

If by "them" you mean MIME headers inside multipart parts, then Squid
does not see them and does not operate on them. The insides of each
chunk is opaque data to Squid.


> and alos had log_mime_hdrs on and then in
> logformat just had all.

You should be able to log the HTTP request header values using %>h or
%>ha. You will not be able to log or match any message body snippets,
including things like MIME Content-Disposition values. Squid does not
look inside the body of the POSTed resource.


If you need further help, you may want to clarify what you are trying to
achieve. You said "send multipart form data to another service". Are you
trying to _route_ request messages based on multipart form _contents_?


HTH,

Alex.


> On Thu, Jul 23, 2020 at 11:46 AM Ryan Le wrote:
> 
> Thanks, 
> 
> I have been looking at the squid debug and can see that it is
> getting the multipart.
> 
> POST http://bb.com
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0)
> Gecko/20100101 Firefox/78.0
> Accept: application/json
> Accept-Language: en-US,en;q=0.5
> Accept-Encoding: gzip, deflate
> Referer: http://b.com
> Cache-Control: no-cache
> X-Requested-With: XMLHttpRequest
> Content-Type: multipart/form-data;
> boundary=---328901485836611227811186534509
> Content-Length: 1245
> Origin: http://b.com
> Cookie: cookie
> Host: bbb.com 
> Via: ICAP/1.0 
> 
> 4dd
> -328901485836611227811186534509
> Content-Disposition: form-data; name="action"
> 
> frm_submit_dropzone
> -328901485836611227811186534509
> Content-Disposition: form-data; name="field_id"
> 
> 8
> -328901485836611227811186534509
> Content-Disposition: form-data; name="form_id"
> 
> 5
> -328901485836611227811186534509
> Content-Disposition: form-data; name="nonce"
> 
> e1aca92777
> -328901485836611227811186534509
> Content-Disposition: form-data; name="file8"; filename="translate.zip"
> Content-Type: application/x-zip-compressed
> 
> On Thu, Jul 23, 2020 at 11:16 AM Alex Rousskov
>  > wrote:
> 
> On 7/23/20 9:22 AM, Ryan Le wrote:
> > I have been trying to configure squid to decode and send
> multipart form
> > data to another service. Is there an acl or build parameter
> needed for
> > multipart form data support?
> 
> No, there is no need to allow any specific Content-Type, including
> multipart. Squid does not know anything about
> multipart/form-data. If a
> multipart/form-data message is well-formed from HTTP point of
> view, then
> Squid will process it as any other message, including passing it to
> ICAP/eCAP (where configured).
> 
> Cheers,
> 
> Alex.
> 

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections - RESOLVED

[Contato - KONNTROL]

Klaus, You got it!  Thanks a lot!!

I just added the directive like:  

cache_peer 127.0.0.1 parent 8080 0 login=*:password no-netdb-exchange
always_direct deny all
never_direct allow all

It worked fine now.  All those thousands of connections disappeared.
Just curious what is that "option" of "netdb-exchange".  Where can I find 
further info about it?

Thanks everyone!! Closed-Resolved.
Fabricio.




-Original Message-
From: squid-users  On Behalf Of 
Klaus Brandl
Sent: Wednesday, July 29, 2020 12:27 PM
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] SQUID with cache_peer config + E2guardian - too many 
connections

tried to add the option "no-netdb-exchange" on your cache_peer line?

On Wednesday 29 July 2020 11:50:20 Contato - KONNTROL wrote:
> Hello Amos/Klaus/Fantomas,
> Thanks for your help.
> 
> I have searched for the "via off"  and "via on" - The directive was 
> not present on the config file but I am assuming "via on" it's the 
> default option. Anyway I added it to the config file. No Success. I 
> also checked the visible_hostname, as suggested by Klaus (Thanks Klaus for 
> the help!!) .
> It was set to localhost, then I changed to something different. No 
> success as well.
> 
> So, I decided to make another test using Wireshark. For that, I put 
> the squid down before starting the capture, then started Squid after 
> capturing.
> What I can see is squid sending thousands of requests (like a machine 
> gun) against the E2G (loopback interface on port 8080) with thousands 
> of "408 Request Time Out" entries. I also see the following HTTP GET:
> "http://127.0.0.1:8080/squid-internal-dynamic/netdb";  by the way,  
> 8080 is E2G port. Not sure what it is. The 408 above are probably 
> because of this calling.
> 
> Attached you can see the capture file, just in case you have wireshark 
> or any other software able to read .cap file. Don't worry, there is 
> nothing confidential on the file. That's a LAB environment. That is 
> really confusing me.
> 
> Thank You very Much!
> 
> Regards
> Fabricio.
> 
> 

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen tel +49 89 991950-0, fax -999, 
www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch Amtsgericht Muenchen HRB 98238 
genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid and multipart form decode

[Ryan Le]

I do apologize I do not have logs of that specific file but I have an
example from the same site and doing the same post.

> Even though it looks like TeChunkedParser is getting all the
> additional headers

>TeChunkedParser has nothing to do with multipart/form-data bodies.
>TeChunkedParser parses chunked encoding, and even then it is applied to
>remove _transfer_ encoding, not to interpret the actual resource content
>inside the chunks.

I do see it in two locations.

2020/07/26 23:11:12.921 kid6| 74,9| TeChunkedParser.cc(45) parse: Parse
buf={length=3667, data='e47
-351645264024548376901231954897
Content-Disposition: form-data; name="action"

frm_submit_dropzone
-351645264024548376901231954897
Content-Disposition: form-data; name="field_id"

8
-351645264024548376901231954897
Content-Disposition: form-data; name="form_id"

5
-351645264024548376901231954897
Content-Disposition: form-data; name="nonce"

6bb20c0bd7
-351645264024548376901231954897
Content-Disposition: form-data; name="file8"; filename="file.zip"
Content-Type: application/x-zip-compressed

As well as the following location

2020/07/26 23:11:12.921 kid6| 58,9| HttpMsg.cc(198) parse: HttpMsg::parse
success (689 bytes) near 'POST http://.com/post HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0)
Gecko/20100101 Firefox/78.0
Accept: application/json
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cache-Control: no-cache
X-Requested-With: XMLHttpRequest
Content-Type: multipart/form-data;
boundary=---351645264024548376901231954897
Content-Length: 3655
Origin: http://bb.com
Referer: http://b.com.com/http-post/
Cookie: _ga=GA1.2.1194289608.1595640198; _gid=GA1.2.252592555.1595804428;
_gat_gtag_UA_47458108_3=1
Host: bbb.com
Via: ICAP/1.0 (C-ICAP/0.5.6 ICAP service )

e47
-351645264024548376901231954897
Content-Disposition: form-data; name="action"

frm_submit_dropzone
-351645264024548376901231954897
Content-Disposition: form-data; name="field_id"

8
-351645264024548376901231954897
Content-Disposition: form-data; name="form_id"

5
-351645264024548376901231954897
Content-Disposition: form-data; name="nonce"

6bb20c0bd7
-351645264024548376901231954897
Content-Disposition: form-data; name="file8"; filename="file.zip"
Content-Type: application/x-zip-compressed


>I am not sure, but it looks like you have pasted a part of an ICAP
>message. TeChunkedParser is used to parse chunked transfer encoding used
>for a part of the ICAP message body. Beyond decoding those chunks, it is
>all opaque data to Squid.

Thanks for that information.


>To avoid misunderstanding, in your pasted example, the contents of the
>first chunk starts with these two lines:

> -328901485836611227811186534509
> Content-Disposition: form-data; name="action"

>It does _not_ start with the "Content-Disposition:..." line or the
>"frm_submit_dropzone" line.


> I can't seem to create ACL or output them using
> logformat. I was trying to request these headers with
> req_mime_type/resp_mime_type.

>If by "them" you mean MIME headers inside multipart parts, then Squid
>does not see them and does not operate on them. The insides of each
>chunk is opaque data to Squid.


> and also had log_mime_hdrs on and then in
> logformat just had all.

>You should be able to log the HTTP request header values using %>h or
>%>ha. You will not be able to log or match any message body snippets,
>including things like MIME Content-Disposition values. Squid does not
>look inside the body of the POSTed resource.

I will test with the two examples given and see what they return.


>If you need further help, you may want to clarify what you are trying to
>achieve. You said "send multipart form data to another service". Are you
>trying to _route_ request messages based on multipart form _contents_?

What I am ultimately trying to accomplish is to see the best way to get
more detail and have an action on sites
that are posting using multipart/form-data as the Content-Type header. This
is mainly to separate action taken on an actual form being submitted
versus a file being submitted or as you stated route request messages based
on the content.
Whether that be logformat with headers and passing the headers to a custom
external service or within squid itself.

On Wed, Jul 29, 2020 at 12:16 PM Alex Rousskov <
rouss...@measurement-factory.com> wrote:

> On 7/29/20 11:38 AM, Ryan Le wrote:
> > Even though it looks like TeChunkedParser is getting all the
> > additional headers
>
> TeChunkedParser has nothing to do with multipart/form-data bodies.
> TeChunkedParser parses chunked encoding, and even then it is applied to
> remove _transfer_ encoding, 

Re: [squid-users] Squid and multipart form decode

[Alex Rousskov]

On 7/29/20 12:47 PM, Ryan Le wrote:

>>> Even though it looks like TeChunkedParser is getting all the
>>> additional headers

>> TeChunkedParser has nothing to do with multipart/form-data bodies.

> I do see it in two locations.

> 2020/07/26 23:11:12.921 kid6| 74,9| TeChunkedParser.cc(45) parse: Parse
> buf={length=3667, data='e47
> -351645264024548376901231954897
> Content-Disposition: form-data; name="action"
...

You see the parser reporting the raw input buffer that it is about to
parse. The parser will treat everything you see after the first "e47"
line (which specifies the chunk size in hex) as opaque body bytes (until
the start of the next chunk metadata).


> As well as the following location

> 2020/07/26 23:11:12.921 kid6| 58,9| HttpMsg.cc(198) parse:
> HttpMsg::parse success (689 bytes) near 'POST http://.com/post HTTP/1.1
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0)
...

Same problem. It is just a raw input buffer dump.


> What I am ultimately trying to accomplish is to see the best way to get
> more detail and have an action on sites 
> that are posting using multipart/form-data as the Content-Type header.

ACL-driven actions based on the Content-Type header value should work
fine. Logging of the Content-Type header value to access.log should work
fine.  If something does not work, please provide a specific non-working
configuration example.

Some of your earlier messages sounded like you want Squid to act based
on MIME headers inside the message body or even the values of HTML form
entries. Squid cannot do that on its own. To analyze message bodies
(even in read-only mode), you will need a custom ICAP or eCAP service:
https://wiki.squid-cache.org/SquidFaq/ContentAdaptation


HTH,

Alex.



> On Wed, Jul 29, 2020 at 12:16 PM Alex Rousskov wrote:
> 
> On 7/29/20 11:38 AM, Ryan Le wrote:
> > Even though it looks like TeChunkedParser is getting all the
> > additional headers
> 
> TeChunkedParser has nothing to do with multipart/form-data bodies.
> TeChunkedParser parses chunked encoding, and even then it is applied to
> remove _transfer_ encoding, not to interpret the actual resource content
> inside the chunks.
> 
> I am not sure, but it looks like you have pasted a part of an ICAP
> message. TeChunkedParser is used to parse chunked transfer encoding used
> for a part of the ICAP message body. Beyond decoding those chunks, it is
> all opaque data to Squid.
> 
> To avoid misunderstanding, in your pasted example, the contents of the
> first chunk starts with these two lines:
> 
> > -328901485836611227811186534509
> > Content-Disposition: form-data; name="action"
> 
> It does _not_ start with the "Content-Disposition:..." line or the
> "frm_submit_dropzone" line.
> 
> 
> > I can't seem to create ACL or output them using
> > logformat. I was trying to request these headers with
> > req_mime_type/resp_mime_type.
> 
> If by "them" you mean MIME headers inside multipart parts, then Squid
> does not see them and does not operate on them. The insides of each
> chunk is opaque data to Squid.
> 
> 
> > and alos had log_mime_hdrs on and then in
> > logformat just had all.
> 
> You should be able to log the HTTP request header values using %>h or
> %>ha. You will not be able to log or match any message body snippets,
> including things like MIME Content-Disposition values. Squid does not
> look inside the body of the POSTed resource.
> 
> 
> If you need further help, you may want to clarify what you are trying to
> achieve. You said "send multipart form data to another service". Are you
> trying to _route_ request messages based on multipart form _contents_?
> 
> 
> HTH,
> 
> Alex.
> 
> 
> > On Thu, Jul 23, 2020 at 11:46 AM Ryan Le wrote:
> >
> >     Thanks, 
> >
> >     I have been looking at the squid debug and can see that it is
> >     getting the multipart.
> >
> >     POST http://bb.com
> >     User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0)
> >     Gecko/20100101 Firefox/78.0
> >     Accept: application/json
> >     Accept-Language: en-US,en;q=0.5
> >     Accept-Encoding: gzip, deflate
> >     Referer: http://b.com
> >     Cache-Control: no-cache
> >     X-Requested-With: XMLHttpRequest
> >     Content-Type: multipart/form-data;
> >     boundary=---328901485836611227811186534509
> >     Content-Length: 1245
> >     Origin: http://b.com
> >     Cookie: cookie
> >     Host: bbb.com  
> >     Via: ICAP/1.0 
> >
> >     4dd
> >     -328901485836611227811186534509
> >     Content-Disposition: form-data; name="action"
> >
> >     frm_submit_dropzone
> >  

[squid-users] Caching https data

[Darwin O'Connor]

I run a transit prediction web app . It 
connects to a variety of web APIs to collect the real time data it 
needs. The apps activities are split among many processes. They 
currently uses libcurl to connect to squid for caching (often for as 
little as 10-30 seconds) and benefits of connection sharing, but some of 
the APIs use https, so in that case the data passes through squid 
without the benefits of caching or connection sharing.


I would like to configure squid to connect to these servers securely and 
pass it unencrypted to clients. Security isn't really an issue since 
this step is all within the one server. I'll have to configure libcurl 
to allow unencrypted data.


___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users