Thanks for that Alex. I do have another related question regarding Squid 206 handling.
I have a RESPMOD icap server that supports Allow:206 in the OPTIONS response. I can see that for html webpages, squid sends Allow: 204, 206, trailers in RESPMOD requests, and my server can handle that fine. But for files, specifically for zip files, squid sends Allow: trailers , showing no support for 206. Even if I reply with a 206, I get TCP reset and connection is closed. Do you know why squid doesn't Allow 206 for files? ============================ Here is my configured OPTIONS response. ICAP/1.0 200 OK Methods: RESPMOD Service: PyICAP Server 1.0 Preview: 0 Transfer-Complete: * Max-Connections: 100 Options-TTL: 3600 Allow: 206 ISTag: "CmLYfciETxyLwZW3XYsPwJdCXvVehd" Date: b'Mon', 16 b'Sep' 2019 20:50:52 GMT Server: BaseICAP/1.0 Python/3.7.3 ================================================== And here is the RESPMOD request. RESPMOD icap://127.0.0.1:13440/example ICAP/1.0 Host: 127.0.0.1:13440 Date: Mon, 16 Sep 2019 20:51:00 GMT Encapsulated: req-hdr=0, res-hdr=337, res-body=579 Allow: trailers X-Client-IP: 192.168.0.6 GET http://203.0.113.1/file.zip HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate DNT: 1 Upgrade-Insecure-Requests: 1 Host: 203.0.113.1 HTTP/1.1 200 OK Date: Mon, 16 Sep 2019 20:51:00 GMT Server: Apache/2.4.6 (CentOS) Last-Modified: Thu, 29 Aug 2019 13:54:58 GMT ETag: "989680-59141d9f0557a" Accept-Ranges: bytes Content-Length: 10000000 Content-Type: application/zip On Mon, Sep 16, 2019 at 3:54 PM Alex Rousskov < rouss...@measurement-factory.com> wrote: > On 9/16/19 2:58 PM, Felipe Arturo Polanco wrote: > > > In our case we don't need to modify the initial 10MB, just scan it for > > virus and if found, send a reset back to squid to not transmit the file. > > Yes, my original response was based on that assumption. > > In summary, you can ask Squid to own the 10MB prefix (i.e. do a huge > ICAP Preview) or you can ask the ICAP server to own the 10MB prefix > (i.e. extend ICAP 206). The choice balances various trade-offs, > including different overheads and different risks related to Squid > development associated with each option. Pick your poison :-). > > Alex. > > > > On Mon, Sep 16, 2019 at 11:54 AM Alex Rousskov wrote: > > > > On 9/16/19 10:37 AM, Felipe Arturo Polanco wrote: > > > > > We would like to add some logic to our custom made ICAP server, > one of > > > these logics is to analyze up to 10MB of data of a given file and > > if the > > > file is larger than that, squid should not keep sending it to icap, > > > basically, a 204 message should be returned. > > > > > squid has a cap of 64K of preview size that limit us in the aspect. > > > > True. > > > > > > > Is there a way to extend this limit? Modifying the source code > > perhaps? > > > > Yes, of course. > > > > > > > If so, what are the disadvantages of doing so? > > > > Simply increasing BodyPipe::MaxCapacity might open you up to > > difficult-to-find vulnerabilities where some old Squid code still > > assumes that the buffers are limited by 64KB and crashes/asserts when > > that assumption becomes false. FWIW, I have not audited all > > BodyPipe-related code and do not know whether such bad code exists > > today. > > > > Increasing BodyPipe::MaxCapacity will also increase Squid RAM > > requirements if your Squid receives requests (and/or ICAP > > REQMOD-generated HTTP responses) that exceed 64KB body limit. I am > not > > sure about regular responses -- their accumulation is limited by > > other/independent restrictions. > > > > > > > Would it require more RAM per response > > > > [You should worry about "per message" accumulation -- requests will > use > > the same limit, even if you do not have REQMOD services.] > > > > Yes, but only for messages that exceed the limit. > > > > > > > or is RAM dynamically allocated as the file is being received? > > > > Yes, the RAM in question is allocated dynamically on "as needed" > basis. > > > > YMMV, but Squid may slow down a lot while dealing with the associated > > reallocations because of the too-small 64KB increment used for each > such > > reallocation. > > > > > > An alternative solution to consider here is extending ICAP features > to > > allow for splicing of the 10MB prefix (analyzed _and_ returned back > by > > the ICAP server) and the remaining virgin response suffix (paused > > without excessive accumulation on the Squid side). This would be > similar > > to the existing ICAP 206 use-original-body="10MB" feature[1], but > would > > require additional negotiation and changes to disable storage of the > > 10MB prefix on the Squid side. > > > > This alternative can be implemented without worrying about hidden > 64KB > > body buffer assumptions, but the implementation will not be trivial. > > > > [1] > > > http://www.icap-forum.org/documents/specification/draft-icap-ext-partial-content-07.txt > > > > > > HTH, > > > > Alex. > > > >
_______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
