Re: [squid-users] Caching http google deb files

2016-10-23 Thread garryd
On 2016-10-22 23:18, Heiler Bemerguy wrote: I've never used ICAP, and I think hacking the code is way faster than creating/using a separate service for that. And I'm not sure, but I don't think I can manage to get this done with current squid's options. Hi, For this case I also suggest to use

Re: [squid-users] Caching http google deb files

2016-10-22 Thread Eliezer Croitoru
merguy [mailto:heiler.bemer...@cinbesa.com.br] Sent: Saturday, October 22, 2016 21:18 To: Eliezer Croitoru Cc: squid-us...@squid-cache.org Subject: Re: [squid-users] Caching http google deb files Hi Eliezer I've never used ICAP, and I think hacking the code is way faster than creating/using a separate servi

Re: [squid-users] Caching http google deb files

2016-10-22 Thread Heiler Bemerguy
liezer Eliezer Croitoru <http://ngtech.co.il/lmgtfy/> Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Heiler Bemerguy Sent: Friday, October 21, 2016 18:21 To: squid-us...

Re: [squid-users] Caching http google deb files

2016-10-21 Thread Yuri Voinov
o:squid-users-boun...@lists.squid-cache.org] On > > > Behalf Of Heiler Bemerguy > > > Sent: Friday, October 21, 2016 18:21 > > > To: squid-us...@squid-cache.org > > > Subject: Re: [squid-users] Caching http google deb files > > > > > &g

Re: [squid-users] Caching http google deb files

2016-10-21 Thread Yuri Voinov
System Administrator > Mobile: +972-5-28704261 > Email: elie...@ngtech.co.il > > > From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On > Behalf Of Heiler Bemerguy > Sent: Friday, October 21, 2016 18:21 > To: squid-us...@squid-cache.org > Subject: Re: [s

Re: [squid-users] Caching http google deb files

2016-10-21 Thread Eliezer Croitoru
tfy/> Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Heiler Bemerguy Sent: Friday, October 21, 2016 18:21 To: squid-us...@squid-cache.org Subject: Re: [squid-users] Caching http

Re: [squid-users] Caching http google deb files

2016-10-21 Thread Heiler Bemerguy
Hello, I've limited the "vary" usage and gained some hits by making these modifications (in blue) to the http.cc code: while (strListGetItem(&vary, ',', &item, &ilen, &pos)) { SBuf name(item, ilen); if (name == asterisk) { */** vstr.clear(); break; **/ * *

Re: [squid-users] Caching http google deb files

2016-10-07 Thread Alex Rousskov
On 10/07/2016 03:26 AM, Hardik Dangar wrote: > 2) I want to build a module which will first handle Vary: * requests and > convert it into Vary: Accept-Encoding or something similar but only for > the ACL's specified by cache administrator. If you want to convert/change the Vary response header, yo

Re: [squid-users] Caching http google deb files

2016-10-07 Thread Hardik Dangar
Hey Alex, I totally get that Vary code is different, I have been trying to understand squid code for last few days although my C, C++ skills are very limited I am able to understand bits and pieces here and there. I have also read HTTP 1.1 specs for cache ( https://tools.ietf.org/html/rfc2616#sect

Re: [squid-users] Caching http google deb files

2016-10-06 Thread Alex Rousskov
On 10/06/2016 11:14 AM, Linda A. Walsh wrote: > Alex Rousskov wrote: >> We can, but ignoring Vary requires more/different work than adding >> another refresh_pattern option. Vary is not a refresh mechanism so >> different code areas need to be modified to ignore (but still forward!) >> Vary. >

Re: [squid-users] Caching http google deb files

2016-10-06 Thread Linda A. Walsh
Alex Rousskov wrote: We can, but ignoring Vary requires more/different work than adding another refresh_pattern option. Vary is not a refresh mechanism so different code areas need to be modified to ignore (but still forward!) Vary. I can't say for certain, but I'd give it a 75% shot

Re: [squid-users] Caching http google deb files

2016-10-05 Thread Hardik Dangar
Hey Anthony, I have used apt-cacher-ng, but it can't save git repos or npm repos. Also i have used apt-cacher-ng, it used to work great until 12.02 but when we had started to have mixed setup [ ubuntu 13,14.04 and others ] we got issues within our setup and one point issues became so daily we deci

Re: [squid-users] Caching http google deb files

2016-10-05 Thread Antony Stone
On Wednesday 05 October 2016 at 20:40:46, Hardik Dangar wrote: > Hey Jok, > > Thanks for the suggetion but the big issue with that is i have to download > whole repository about ( 80-120 GB ) first and then each week i need to > download 20 to 25 GB. This is not true for apt-cacher-ng. You inst

Re: [squid-users] Caching http google deb files

2016-10-05 Thread Hardik Dangar
Hey Jok, Thanks for the suggetion but the big issue with that is i have to download whole repository about ( 80-120 GB ) first and then each week i need to download 20 to 25 GB. We hardly use any of that except few popular repos. big issue i always have with most of them is third party repo's. sq

Re: [squid-users] Caching http google deb files

2016-10-05 Thread Jok Thuau
This is sort of off-topic, but have you considered using a deb repo mirroring software? (it would mean that you need to update your clients to point to that rather than google, but that's not really difficult). software like aptly (aptly.info) are really good about this (though a little hard to get

Re: [squid-users] Caching http google deb files

2016-10-05 Thread Alex Rousskov
On 10/05/2016 08:33 AM, Hardik Dangar wrote: > One more thing can you tell me if we are already violating http via > options like nocache, ignore-no-store ignore-private ignore-reload, why > can't we do the same for Vary header ? We can, but ignoring Vary requires more/different work than adding

Re: [squid-users] Caching http google deb files

2016-10-05 Thread Hardik Dangar
Hey Amos, oh, i actually built archive mode squid by getting help at here, http://bugs.squid-cache.org/show_bug.cgi?id=4604 I was thinking if we have option vary_mode just like archive mode to set it for particular domain like, acl dlsslgoogle srcdomain dl-ssl.google.com vary_mode allow dlsslgoo

Re: [squid-users] Caching http google deb files

2016-10-05 Thread Amos Jeffries
On 5/10/2016 11:27 p.m., Hardik Dangar wrote: > Hey Amos, > > I have implemented your patch at > > and added following to my squid.conf > archive_mode allow all > > and my refresh pattern is, > refresh_pattern dl-ssl.google.com/.*\.(deb|zip|tar|rpm) 129600 100% 129600 > ignore-reload ignore-no-s

Re: [squid-users] Caching http google deb files

2016-10-05 Thread Hardik Dangar
Hey Amos, I have implemented your patch at and added following to my squid.conf archive_mode allow all and my refresh pattern is, refresh_pattern dl-ssl.google.com/.*\.(deb|zip|tar|rpm) 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignor$ but i am still not a

Re: [squid-users] Caching http google deb files

2016-10-04 Thread Hardik Dangar
Wow, i couldn't think about that. google might need tracking data that could be the reason they have blindly put vary * header. oh Irony, company which talks to all of us on how to deliver content is trying to do such thing. I have looked at your patch but how do i enable that ? do i need to write

Re: [squid-users] Caching http google deb files

2016-10-04 Thread Amos Jeffries
On 5/10/2016 2:34 a.m., Hardik Dangar wrote: > Hey Amos, > > We have about 50 clients which downloads same google chrome update every 2 > or 3 days means 2.4 gb. although response says vary but requested file is > same and all is downloaded via apt update. > > Is there any option just like ignore

Re: [squid-users] Caching http google deb files

2016-10-04 Thread Hardik Dangar
Hey Amos, after referring to one of your old posts i found, we can use reply_header_replace to replace headers. Is it possible to replace vary * header with something appropriate? or i need to look at squid's source code to ignore vary header and recompile ? On Tue, Oct 4, 2016 at 7:04 PM,

Re: [squid-users] Caching http google deb files

2016-10-04 Thread Hardik Dangar
Hey Amos, We have about 50 clients which downloads same google chrome update every 2 or 3 days means 2.4 gb. although response says vary but requested file is same and all is downloaded via apt update. Is there any option just like ignore-no-store? I know i am asking for too much but it seems ver

Re: [squid-users] Caching http google deb files

2016-10-04 Thread Amos Jeffries
On 5/10/2016 2:05 a.m., Hardik Dangar wrote: > Hello, > > I am trying to cache following deb files as its most requested file in > network. ( google chrome almost every few days many clients update it ). > > http://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb > http://dl-ssl.

[squid-users] Caching http google deb files

2016-10-04 Thread Hardik Dangar
Hello, I am trying to cache following deb files as its most requested file in network. ( google chrome almost every few days many clients update it ). http://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.