I think API is needed. If there is no plugin exists, gzip/unzip is not necessary. I imagine that the one API will guarantee `unzip transform` deal with the body first, another API will guarantee `zip transform` deal with the body last.
On Wed, Mar 25, 2015 at 2:37 AM, Leif Hedstrom <zw...@apache.org> wrote: > > > On Mar 24, 2015, at 1:48 PM, Brian Geffon <bri...@apache.org> wrote: > > > > Yes they get called in the order they are registered for better or worse. > > Think of it this way, three transformations that all register read > request > > and read response headers, the first will run first for both and the last > > will run last for both. This is the fundamental problem. > > > > Nonetheless, I think this argument is independent of the fact that gzip > > should be in the core anyway because it is such a common operation and > > there are known issues with transformations related to performance. > > > I agree. I was trying to understand what the problem is, such that we can > solve this in some reasonable way. This sort of boils down to the lack of > communications between plugins in general, which is why we do have both > @-headers, transaction data and the txn / ssn slots. But there’s no generic > way to communicate intent like this across plugins. And certainly we’re > lacking a “priority” or “ordering” of registered hooks. > > I’d imagine what you are solving for gzip by putting it into core is only > a subset of the big picture problem? How are the gzip / ungzip processing > configured? Via config file or via new APIs or hooks? > > — Leif > > > > > Brian > > > > On Tuesday, March 24, 2015, Leif Hedstrom <zw...@apache.org> wrote: > > > >> Hmmm I'm not anywhere near a computer but continuations should get > called > >> in the order they were added. Is that not the case ? In the case of > global > >> plugins that's the order in the plugin.config. In remap plugins, it's > the > >> order in the remap rule. Where it gets iffy is when plugins adds new > >> continuations as part of the handler of another callback. > >> > >> Maybe I'm still just confused :) > >> > >> -- Leif > >> > >> > >> > >>> On Mar 24, 2015, at 1:20 PM, Dzmitry Markovich < > dmarkov...@linkedin.com > >> <javascript:;>> wrote: > >>> > >>> Setting up unzip plugin at first and zip plugin at last will not help. > >> We tried this already. > >>> There is no clear way to control that first response hook will go to > >> unzip plugin and last response hook will go to zip plugin. > >>> > >>> Also, there might be a case when based on some run-time logic plugins > in > >> the chain will not touch the response body (it is already gzipped and > it is > >> static content file - js/css) in this case we should not execute unzip > and > >> zip plugin at all. But if it is just unzip and gzip plugin we cant > >> extrapolate this logic and we will execute gzip/gunzip where it is not > >> necessary. > >>> > >>> And at this point only ATS know about the "body/data" hooks registered > >> for transaction, so ATS will know whether it is necessary to unzip > >> response, because there are "body/data" hooks registered. And on the way > >> out to the client, based on Accept-Encoding header ATS will zip the > body if > >> necessary. > >>> > >>> -Dzmitry > >>> > >>> ________________________________________ > >>> From: gang li [portl4t...@gmail.com <javascript:;>] > >>> Sent: Tuesday, March 24, 2015 6:38 AM > >>> To: dev@trafficserver.apache.org <javascript:;> > >>> Cc: Roland Zink; Brian Geffon; Cynthia Gu > >>> Subject: Re: Adding Gzip/Gunzip feature in ATS core > >>> > >>> I got this problem before, I think we can just set the unzip plugin at > >>> first and zip plugin at last. > >>> > >>>> On Tue, Mar 24, 2015 at 8:39 PM, Leif Hedstrom <zw...@apache.org > >> <javascript:;>> wrote: > >>>> > >>>> Makes sense. So it's about knowing if a plugin needs the plain text or > >>>> not. I'm not arguing against gzip in the core, we should do that for > >>>> performance reasons. > >>>> > >>>> Not quite sure I see the argument for which plugin does the gzip > though, > >>>> that sounds somewhat of a configuration issue. Something w suck at :) > >> The > >>>> order of global plugins is well defined, but maybe txn hooks makes > this > >>>> trickier too. > >>>> > >>>> Cheers, > >>>> > >>>> -- Leif > >>>> > >>>> > >>>> > >>>>> On Mar 24, 2015, at 7:50 AM, Roland Zink <rola...@flashnetworks.com > >> <javascript:;>> > >>>> wrote: > >>>>> > >>>>> I think they can see the updated headers but this doesn't help. A > >> plugin > >>>> can't know if the next plugin in the chain needs the plain text or > even > >> if > >>>> there is a next plugin. So every plugin is doing the gzip at the end. > >> The > >>>> next plugin in the chain then will ungzip again. > >>>>> > >>>>> Regards, > >>>>> Roland > >>>>> > >>>>> -----Original Message----- > >>>>> From: Leif Hedstrom [mailto:zw...@apache.org <javascript:;>] > >>>>> Sent: Tuesday, March 24, 2015 12:43 PM > >>>>> To: dev@trafficserver.apache.org <javascript:;> > >>>>> Cc: Brian Geffon; Cynthia Gu > >>>>> Subject: Re: Adding Gzip/Gunzip feature in ATS core > >>>>> > >>>>> I guess I don't understand why the chaining makes a difference. > That's > >>>> what I was asking in the previous email. If two plugins are chained, > one > >>>> should be able to detect gzip or not gzip the same way it does as if > it > >> was > >>>> not chained. What am I missing ? Does each chained plugin not see the > >>>> updated header? Do they all see the same unmodified headers? That > seems > >>>> unfortunate to say the least if it's so ;/. > >>>>> > >>>>> -- Leif > >>>>> > >>>>> > >>>>> > >>>>>> On Mar 24, 2015, at 12:39 AM, Shu Kit Chan <chanshu...@gmail.com > >> <javascript:;>> > >>>> wrote: > >>>>>> > >>>>>> ESI should handle these situations correctly. e.g. it won't gzip the > >>>>>> response if it is already gzipped. > >>>>>> So it works fine in a standalone way. > >>>>>> > >>>>>> The problems begin when we starts to chain a few of these standalone > >>>>>> transformation plugins together. Each of them will try to do the > right > >>>>>> thing when they are used in a standalone way and perform gunzip/gzip > >>>>>> correctly. So chaining them together means I will be unnecessarily > >>>>>> doing multiple gunzip/gzip on the contents. > >>>>>> > >>>>>> The other way is make them plugins be aware of each other and work > >>>>>> together and don't unnecessarily do gunzip/gzip. But then this makes > >>>>>> the plugins no longer standalone > >>>>>> > >>>>>> +1 on Dzmitry's idea. > >>>>>> > >>>>>> Kit > >>>>>> > >>>>>>> On Mon, Mar 23, 2015 at 5:11 PM, Leif Hedstrom <zw...@apache.org > >> <javascript:;>> > >>>> wrote: > >>>>>>> > >>>>>>> > >>>>>>>>> On Mar 23, 2015, at 6:08 PM, Dzmitry Markovich > >>>>>>>> <dmarkov...@linkedin.com.INVALID> wrote: > >>>>>>>> > >>>>>>>> Hello ATS experts, > >>>>>>>> > >>>>>>>> Today it is very common that http data that goes to the wire is > >>>>>>> compressed. And we think it is a time to standardize this process > in > >>>>>>> ATS core, since it is very common operation. > >>>>>>>> > >>>>>>>> Today multiple plugins running on the same tier (that operates > with > >>>>>>> response body) do not have enough flexibility to handle > gzip/gunzip - > >>>>>>> and most of the times simply every plugin do gzip/gunzip. This lead > >>>>>>> to the situation when we do gzip/gunzip multiple times while we > >>>>>>> process the http response. This leads to the bad CPU utilization > and > >>>> performance. > >>>>>>>> > >>>>>>>> Yes, there is gzip/gunzip logic in atscpp API - so plugins can > >>>>>>>> simply > >>>>>>> use those. But today's ATS architecture does not allow us to fully > >>>>>>> control the order of hook callbacks for every plugin per request - > it > >>>>>>> means there is no non-hacky way to prevent multiple gzip/gunzip > calls > >>>>>>> while processing the request. And there is no way to do that with > the > >>>>>>> assumption that plugins dnot know about each other. > >>>>>>>> > >>>>>>>> > >>>>>>>> Here is our high level proposal, just to start the conversation > >> going: > >>>>>>>> - Have gzip/gunzip logic landed on ATS core, so engineers can ask > >>>>>>>> ATS > >>>>>>> (vi config parameter) to take care about gzip/gunzip logic for them > >>>>>>>> 1. Only ATS know when the body arrived to first plugin - so ATS at > >>>>>>> this point ungzip the body; > >>>>>>>> 2. Only ATS knows when body is processed by all plugins and should > >>>>>>>> go > >>>>>>> over the wire - so ATS at this point gzip the body if client > >> supported > >>>> it. > >>>>>>> > >>>>>>> > >>>>>>> Moving gzip to the core seems a generally good idea. Probably in > some > >>>>>>> extensible way such that we can add future compression encoding (I > >>>>>>> think Chrome has support for some better ones?). > >>>>>>> > >>>>>>> Now, I’m slightly confused, and/or concerned, that our plugins do > not > >>>>>>> handle this well. I would have imagined that something doing gzip > >>>>>>> would not do so if the content comes back with Content-Encoding: > >>>>>>> gzip. So shouldn’t e.g. ESI detect if the gzip plugin has already > >> done > >>>> so? > >>>>>>> > >>>>>>> Is there a reason why a plugin *can’t* detect this? If not, we > should > >>>>>>> fix the plugins regardless of this RFE, it seems like a broken > >>>>>>> behavior if a plugin can gzip something that’s already CE: gzip. > >>>>>>> > >>>>>>> Cheers, > >>>>>>> > >>>>>>> — Leif > >
