Re: [squid-users] Large text ACL lists

2016-10-03 Thread Darren
Hi Nishant Thanks for the lead, I will have a look. Redis is also interesting in this case due to its ability to scan keys and iterate through keys with a wildcard and cursors. Redis looks like it's just what I need as I need to swap in and out sets of sites on demand. I have also been using P

Re: [squid-users] Large text ACL lists

2016-10-03 Thread Bob Cochran
On 10/02/2016 02:16 AM, Nishant Sharma wrote: Hi, On 2 October 2016 9:54:52 AM IST, Darren wrote: Hi I have now opened the Pandora box of writing my own helper as per Bobs suggestion. We are working on a redirector which we are currently using at around 100 geographically distributed squids

Re: [squid-users] Large text ACL lists

2016-10-01 Thread Nishant Sharma
Hi, On 2 October 2016 9:54:52 AM IST, Darren wrote: >Hi > >I have now opened the Pandora box of writing my own helper as per Bobs >suggestion.  We are working on a redirector which we are currently using at around 100 geographically distributed squids. These squid are running on OpenWRT and Pf

Re: [squid-users] Large text ACL lists

2016-10-01 Thread Darren
Hi I have now opened the Pandora box of writing my own helper as per Bobs suggestion.  I am playing with the idea of pre-processing my acl lists and using memcached as a KV store. This way I should be able to update ACL members whilst keeping everything as available as possible. I would updat

Re: [squid-users] Large text ACL lists

2016-10-01 Thread Amos Jeffries
On 2/10/2016 2:08 p.m., Benjamin E. Nichols wrote: > I wouldnt advise reconfigure for when you update your blacklists, sure > it sounds great, but in reality, as I said, in my experience, only > sometimes will it actually reload the acl from disk, sometimes it wont. > Youll do a reconfigure and di

Re: [squid-users] Large text ACL lists

2016-10-01 Thread Benjamin E. Nichols
I wouldnt advise reconfigure for when you update your blacklists, sure it sounds great, but in reality, as I said, in my experience, only sometimes will it actually reload the acl from disk, sometimes it wont. Youll do a reconfigure and discover your squid is still running the old acls which p

Re: [squid-users] Large text ACL lists

2016-10-01 Thread Alex Rousskov
On 09/30/2016 07:02 PM, Darren wrote: > If I have to reload the ACL lists do I restart squid or is there a way > to update without impacting the users to much? You can reconfigure Squid instead of restarting it. Reconfiguration is usually better than a complete restart as far as user impact is co

Re: [squid-users] Large text ACL lists

2016-09-30 Thread Bob Cochran
Darren, Have you also considered writing your own redirector/rewriter in a language like python? There seems to be a nice starting example in the "Squid Book", which I was able to get working along with extending it. Good luck, Bob On 09/29/2016 05:44 AM, Darren wrote: Hi All I have bee

Re: [squid-users] Large text ACL lists

2016-09-30 Thread Benjamin E. Nichols
Also if you are going to use Squid Native ACL blacklists and reload while you are updating, its a good idea to have a parent proxy configured, so that your traffic/users wont be interrupted, squid will default to the next available proxy while its unavailable/reloading the blacklists and forwar

Re: [squid-users] Large text ACL lists

2016-09-30 Thread Benjamin E. Nichols
I would recommend you stop squid and start it, simply doing a -k reconfigure is a bad idea, because sometimes squid will not reload the new blacklists, I have no idea why it is unpredictable in this manner or if they have fixed this problem, I didnt write the software, but what I do know, in my

Re: [squid-users] Large text ACL lists

2016-09-30 Thread Darren
One further question If I have to reload the ACL lists do I restart squid or is there a way to update without impacting the users to much? In some of the scenarios, some acl lists may change frequently thanks again. Sent from Mailbird [http://www.getmailbird.com/?utm_source=Mailbird&utm_med

Re: [squid-users] Large text ACL lists

2016-09-30 Thread Darren
Hi My main issue with squid guard is that when I try and block say www.facebook.com and the user goes to https://www.facebook.com, squidguard only sees the initial CONNECT as the target IP so doesn't match against the domain entry. If squidguard did a reverse DNS lookup, I could keep using tha

Re: [squid-users] Large text ACL lists

2016-09-30 Thread Yuri Voinov
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Amos, I'm afraid that this is not a solution. Block lists have become so huge that only their compression and / or placement in an external database (as Marcus) can save the situation. 30.09.2016 12:59, Amos Jeffries пишет: > On 30/09/2016 6:58 p

Re: [squid-users] Large text ACL lists

2016-09-30 Thread Amos Jeffries
On 30/09/2016 6:58 p.m., Darren wrote: > Thank you Amos > > The resources I save not running multiple Squidguards will make more > ram available as you say and having a simpler setup is never a bad > thing either. > > Just to clarify, so when squid fires up, it caches the ACL file into > ram in

Re: [squid-users] Large text ACL lists

2016-09-29 Thread Darren
Thank you Amos The resources I save not running multiple Squidguards will  make more ram available as you say and having a simpler setup is never a bad thing either. Just to clarify, so when squid fires up, it caches the ACL file into ram  in it's entirety and then does some optimizations? If t

Re: [squid-users] Large text ACL lists

2016-09-29 Thread Amos Jeffries
That is more than enough please. Some people on this list are competitors. There will necessarily be private issues between people and/or organisations. And that is exactly where those issues should stay. Private. It benefits us all to interact politely on the list(s) no matter what is going on

Re: [squid-users] Large text ACL lists

2016-09-29 Thread Amos Jeffries
On 29/09/2016 10:44 p.m., Darren wrote: > Hi All > > I have been tinkering with Squidguard for a while, using it to manage > ACL lists and time limits etc. > > While it works OK, it's not in active development and has it's > issues. > > What are the limitations with just pumping ACL lists direct

Re: [squid-users] Large text ACL lists

2016-09-29 Thread Benjamin E. Nichols
Dear Mr Alex Rousskov. Please kindly take your opinions, take them and shove them directly up your bloated arrogant ass. I have little need to cater to you, or to dignify your mindless criticism of my opinions, which only serve to demonstrate that your ego is larger than you are sir. Sign

Re: [squid-users] Large text ACL lists

2016-09-29 Thread Alex Rousskov
On 09/29/2016 03:42 PM, Benjamin E. Nichols wrote: > Well, forgive me for bad mouthing the developers here, but I think this > is a good reason. It is not. Badmouthing, for any reason, has no positive side effects and may have many negative ones. > it would be better > to actually have something

Re: [squid-users] Large text ACL lists

2016-09-29 Thread Benjamin E. Nichols
Well, forgive me for bad mouthing the developers here, but I think this is a good reason. You see, you are going to have to eliminate all the redundant subdomains in your blacklists, because they are going to crash modern versions of squid. And to do this I would recommend using an older versi

Re: [squid-users] Large text ACL lists

2016-09-29 Thread Darren
Hi What I am trying to do is to simplify everything and remove the external re-writers from the workflow due to the fact that they are either old with sporadic development or wrap their own lists into the solution. I am also producing my own ACL lists for this project so third party blacklists

Re: [squid-users] Large text ACL lists

2016-09-29 Thread Benjamin E. Nichols
The other issue is that shalla and urlblacklist produce garbage blacklists, and neither of them are actively developing or improving the backend technology required to product high quality blacklists. We are the leading publisher of blacklists tailored for Web Filtering Purposes. We are also

Re: [squid-users] Large text ACL lists

2016-09-29 Thread Antony Stone
On Thursday 29 September 2016 at 11:44:28, Darren wrote: > Hi All > > I have been tinkering with Squidguard for a while, using it to manage ACL > lists and time limits etc. > > While it works OK, it's not in active development and has its issues. Have you considered https://www.urlfilterdb.com/

[squid-users] Large text ACL lists

2016-09-29 Thread Darren
Hi All I have been tinkering with Squidguard for a while, using it to manage ACL lists and time limits etc. While it works OK, it's not in active development and has it's issues. What are the limitations with just pumping ACL lists directly into Squid and letting it do all the work internally