Re: transparent proxy

2019-06-24 Thread Alan Carroll
See if this helps - among other things it has a check list of issues to check. https://www.dropbox.com/sh/h7erczfbt8ug8kn/AADJAqNz_xizurIHE6hx8Q8ka This is available from the wiki, under "Presentations", back from 2013. On Sun, Jun 23, 2019 at 11:54 PM Dk Jack wrote: > Hi Susan, > yes, I've enab

Re: Fetching URL contents in a plugin

2019-06-24 Thread Alan Carroll
I am literally working with this right now, I will send an update when I have the code in a good state. Turns out that, for no apparent reason, TSIOBufferSizedCreate asserts if the requested size is more than 32K. Although in practice that's likely to be enough. Perhaps TSHttpTxnEffectiveUrlStringG

Re: transparent proxy

2019-06-24 Thread SUSAN HINRICHS
Rp_filter is a reverse proxy filter. If enabled, which it is by default, it will drop packets that show up on unexpected interfaces. May not be an issue in the bridge case, but something to look into. On Sun, Jun 23, 2019, 11:54 PM Dk Jack wrote: > Hi Susan, > yes, I've enabled ip_forward. What

Re: Fetching URL contents in a plugin

2019-06-24 Thread Alan Carroll
My mistake, TSHttpTxnEffectiveUrlStringGet is documented here - https://docs.trafficserver.apache.org/en/8.0.x/developer-guide/api/functions/TSUrlStringGet.en.html#c.TSHttpTxnEffectiveUrlStringGet On Mon, Jun 24, 2019 at 10:08 AM Alan Carroll < solidwallofc...@verizonmedia.com> wrote: > I am lite

Re: transparent proxy

2019-06-24 Thread Dk Jack
Ah! Yeah, I think you are taking about reverse path filter. I remember checking for it. But I’ll check again. Dk. > On Jun 24, 2019, at 8:31 AM, SUSAN HINRICHS wrote: > > Rp_filter is a reverse proxy filter. If enabled, which it is by default, it > will drop packets that show up on unexpecte

Re: Implementing Rate-limiting in forward proxy mode

2019-06-24 Thread Weixi Li (BLOOMBERG/ PRINCETON)
Hi Dk, Thanks a lot for the example. It is really inspiring. I have a question though: in case of "bucket->consume()" returning false, the code would return a 429 response. In our use-case, we don't want to deny the client with a 429 (but want to queue the request instead). How to handle the fa

Re: Implementing Rate-limiting in forward proxy mode

2019-06-24 Thread Leif Hedstrom
> On Jun 24, 2019, at 13:12, Weixi Li (BLOOMBERG/ PRINCETON) > wrote: > > Hi Dk, > > Thanks a lot for the example. It is really inspiring. I have a question > though: in case of "bucket->consume()" returning false, the code would return > a 429 response. In our use-case, we don't want to d

Re: Implementing Rate-limiting in forward proxy mode

2019-06-24 Thread Dk Jack
Also, stalling will not be good, if you have CDN upstream from you that is pipelining requests on the same connection for multiple clients. In our plugin we tried delay request, but we couldn’t solve http 1.1 pipeline issue. Hence ended up going with the simple approach of deny. Dk. > On Jun

Re: transparent proxy

2019-06-24 Thread Alan Carroll
I don't think this went through so I'm sending it again. These are slides from an ApacheCon talk I gave about transparency. Among the details is a nice check list of things to check when it doesn't work. https://www.dropbox.com/sh/h7erczfbt8ug8kn/AADJAqNz_xizurIHE6hx8Q8ka?preview=ApacheCon-2013.p

Re: Implementing Rate-limiting in forward proxy mode

2019-06-24 Thread Leif Hedstrom
You definitely would not want to stall the client after you’ve made an origin connection. I.e. you only want to stall the UA. — Leif > On Jun 24, 2019, at 14:42, Dk Jack wrote: > > Also, stalling will not be good, if you have CDN upstream from you that is > pipelining requests on the same co

[API] TSHttpHdrEffectiveUrlBuffGet

2019-06-24 Thread Alan Carroll
Triggered by a recent email exchange on this subject, and Watl's proposal in this are, and due to the fact that I was doing plugin work that also needed that sort of API, I realized the current API is rather dismal in this regard. I propose adding this function. TSReturnCode TSHttpHdrEffectiveURL

Re: transparent proxy

2019-06-24 Thread Dk Jack
I disabled rp_filter. However, I see no change in behavior. The kernel seems to be dropping it after picking it up from the interface. I can see the SYN when I do tcpdump on the physical interface. However, I don't see it on bridge interface (br0). Not sure if that offers a clue. I flushed all the

Re: Implementing Rate-limiting in forward proxy mode

2019-06-24 Thread Dk Jack
Leif, I was talking about the case where ATS is deployed after CDN. client --> CDN Network ---> ATS --> Origin-Server In most such deployments, the connection between the CDN and ATS is HTTP 1.1. and it is pipelined. In that case, you cannot stall the clients because you could run into head-o

Re: Implementing Rate-limiting in forward proxy mode

2019-06-24 Thread Sudheer Vinukonda
Interesting setup - But, wouldn’t the CDN have connection pools to ATS already? Curious to know if enabling HTTP pipelining provides benefits beyond what connection pools can give. That said, I don’t think it makes a lot of sense to implement rate limiting for _end clients_ in this sort of a se

Re: Implementing Rate-limiting in forward proxy mode

2019-06-24 Thread Dk Jack
I suspect connection pools would be using the http pipeline feature to multiplex multiple client connections on to the pool connections towards ATS. In my case, rate-limiting was implemented as a security feature i.e rate-limiting malicious IPs etc. Moreover, we don't have control over it at the en

Re: Implementing Rate-limiting in forward proxy mode

2019-06-24 Thread Sudheer Vinukonda
> On Jun 24, 2019, at 8:33 PM, Dk Jack wrote: > > I suspect connection pools would be using the http pipeline feature to > multiplex multiple client connections on to the pool connections towards > ATS. Right. But, you can turn off the pipelining support on ATS, if there isn’t a clear benefi

Re: Implementing Rate-limiting in forward proxy mode

2019-06-24 Thread Dk Jack
Perhaps. I think we are talking two different things. I was trying to say, it’s a bad idea to stall a client connection when multiple clients are getting multiplexed on the same connection (between cdn and ats). From that perspective, the result is the same on the client side of ats. Stalling on

Re: Implementing Rate-limiting in forward proxy mode

2019-06-24 Thread Sudheer Vinukonda
Agree. However, it’s not an issue in the typical set ups when ATS is not behind a CDN (and I think Weixi’s use case didn’t appear to be one such either?). Note that, besides rate limiting, there are other practical examples in production, where user requests are “queued” for various use cases w