Thanks Devon! So just to clarify our request flow is:
Client > CDN > Go Reverse Proxy > Origin Our Go Reverse Proxy has historically been responsible for adding caching headers (e.g. Cache-Control and Surrogate-Control) when the origins have failed to do so (as a way to ensure things are cached appropriately). It's unclear to me why you should be setting an etag header if you're a > proxy. > That's why when it came to looking at setting serve stale defaults for our origins (e.g. stale-while-revalidate and stale-if-error) I realized that somewhere along the chain an appropriate ETag/Last-Modified should be set and that's why I started wondering if our proxy should be responsible for setting them. Even then I felt like setting Last-Modified was way outside the responsibility of our proxy, but that maybe setting of ETag would have sufficed. Unless you're serving from the filesystem handler (which does > implement IMS/INM), you'll need to implement these yourself. > I think your other related answers might explain to me why the go reverse proxy doesn't support conditional requests, in that it's NOT a 'caching proxy' and so being able to handle that revalidation logic wouldn't make sense. > Note that you _could_ simply proxy this to the origin and let it > handle the validation. This is often overkill for what people actually > need, but it is guaranteed to work. > OK, so as we are indeed just proxying the request pretty much 'as is' to the origin, i.e. the CDN is making the revalidation conditional request when our stale-while-revalidate TTL expires, I'm guessing (I appreciate this is the 'basics' of how a proxy works, but I want to talk it through in case I'm mistaken in any way!) the go proxy will transparently keep that information for the origin to respond with the appropriate ETag/Last-Modified, and the go proxy again will transparently pass back their response through to the CDN to then update its cache if it indeed got a `200 OK` from origin or to continue serving stale if the origin returned a `304 Not Modified` (and in either case I expect the origin should send ETag/Last-Modified headers regardless of 200/304 status'). A hash function over the body of the response would constitute strong > validation. I'm not sure why you'd need to mix in the path; there's > nothing wrong with serving the exact same content between two > endpoints, and the ETag is tied to a response object. > Ah ok, so I was thinking along these lines, but was getting confused between content that is cached vs content that is rendered at 'runtime' (e.g. I was getting confused with the response containing a <script> tag that might dynamically change the adverts on the page depending on the client and wondering if that meant it wasn't "strong" validation just hashing the server response body, but I guess it's redundant thinking like that because the *actual* cached content is what's compared as far as the hash is concerned and not what the client-side scripting is modifying. On Tuesday, 6 August 2019 15:48:49 UTC+1, Devon H. O'Dell wrote: > > Hi Mark, > > Whether or not your proxy is caching, you may find RFC7234[1] relevant > in addressing some of your questions (as well as many you may later > encounter). I think you may find section 5.2 to be of particular > interest, though any proxy author should be familiar with the full > text. > > Op di 6 aug. 2019 om 05:14 schreef <mark...@gmail.com <javascript:>>: > > > > Hello, > > > > I'm using Go's standard library reverse proxy and I'm trying to figure > out if the standard library HTTP web server (e.g. http.ListenAndServe) > implements the relevant conditional request handling logic for > ETag/Last-Modified headers. > > > > I did some Googling and noticed the HTTP file system request handler ( > https://golang.org/src/net/http/fs.go) does implement that logic, but I > couldn't find the same for the HTTP web server. > > > > I also couldn't find any examples of setting ETags/Last-Modified (other > than this basic implementation for setting ETags: > https://github.com/go-http-utils/etag/blob/master/etag.go). > > > > What's confusing me there is the concept of "strong" and "weak" > validation and how certain scenarios might influence whether an ETag is > marked as either strong or weak (depending on the implementation -- see > https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests#Validators). > > > > > > So to recap, my questions are (and I appreciate some of these are > outside the scope of just Go -- so apologies if that's not allowed in this > forum): > > I think this is a fine question for this list, which isn't necessarily > constrained to questions about Go, but also for how to achieve things > while using Go. Lines get blurred since many technologies touch each > other. I don't think any apologies are necessary :). > > > 1. Should I set ETag/Last-Modified in a proxy? Last-Modified feels like > it's not the responsibility of the proxy but the origin, where as an ETag > is something I feel is "ok" to do in the proxy as a 'fallback' (as we > already set 'serve stale' caching headers on behalf of our origins if they > neglect to include them). > > ETag and Last-Modified should be sent by the origin to any proxy to > let the proxy know when the content is stale (assuming the proxy is > caching). The only case in which a proxy might set these things is if > there are configurations provided by the content owner that allow the > proxy to determine what the lifetime of the response object is outside > of response headers. This is most useful in cases where the content is > synthetically generated by the proxy as a result of the content > owner's configuration. If you don't have such a system in place, your > proxy should never be generating these response headers, and you > should be working with your customers / users to help them understand > when to set cache control headers. > > > 2. Do I need to implement `If-None-Match` and `If-Modified-Since` > behaviours myself (i.e. is it not provided by the Go standard library's > HTTP web server)? > > Unless you're serving from the filesystem handler (which does > implement IMS/INM), you'll need to implement these yourself. > > Note that you _could_ simply proxy this to the origin and let it > handle the validation. This is often overkill for what people actually > need, but it is guaranteed to work. > > One trick that many CDN providers leverage is to offer their customers > the option to serve the stale object while revalidating it. If that > option is set, an asynchronous revalidation request is spawned -- new > requests are blocked on the completion of that request -- and the > potentially stale content is served to the original requester without > blocking that request on revalidation. > > > 3. I was planning on setting an ETag header on the response from within > httputil.ReverseProxy#ModifyResponse but wasn't sure if that would be the > correct place to set it. > > It's unclear to me why you should be setting an etag header if you're a > proxy. > > > 4. What constitutes a strong/weak validator (e.g. would a simple hash > function generating a digest of the URL path + response body suffice)? > > A hash function over the body of the response would constitute strong > validation. I'm not sure why you'd need to mix in the path; there's > nothing wrong with serving the exact same content between two > endpoints, and the ETag is tied to a response object. > > Weak validation is signified by an additional "W/" in the etag > identifier. In practice, this means that you mustn't use weak > identifiers for serving byte-range requests. Weak identifiers may be > more useful for dynamically generated content where you might for > example have a date added in, or an ad server link that is rotated > each time the page is served, or a counter, or something like this. An > example of weak validation would be something that is version and > encoding based -- each time the content changes materially, you'd > increment the version, and some identifier for the content-encoding > would also be mixed in. > > > Thanks for any help/insights/opinions y'all can share with me. > > > > Kind regards, > > Mark > > > > Hope that helps! > > Kind regards, > > --dho > > [1]: https://tools.ietf.org/html/rfc7234 > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a8da59d3-c905-4e13-8d15-3792a36c2f61%40googlegroups.com.
