+1 to the API in the core. Although I'm generally against features in the core-- this one makes a lot of sense. We only want/need to uncompress the data if the specific transaction has a transformation hook. In the core we can check and then uncompress it once for all of them, then re-compress it once they have all run. The problem we've run into locally (as Dzmitry mentioned) is that each plugin uncompresses and recompresses the body-- which is less than ideal. This also means that plugins don't have to attempt to coordinate with eachother-- which introduces a few problems (1) no good way to coordinate (as Leif mentioned), and (2) this becomes *very* difficult to test-- since you now have to test each plugin's coordination against every version of all other plugins.
On Tue, Mar 24, 2015 at 6:25 PM, gang li <portl4t...@gmail.com> wrote: > I think API is needed. If there is no plugin exists, gzip/unzip is not > necessary. I imagine that the one API will guarantee `unzip transform` deal > with the body first, another API will guarantee `zip transform` deal with > the body last. > > On Wed, Mar 25, 2015 at 2:37 AM, Leif Hedstrom <zw...@apache.org> wrote: > > > > > > On Mar 24, 2015, at 1:48 PM, Brian Geffon <bri...@apache.org> wrote: > > > > > > Yes they get called in the order they are registered for better or > worse. > > > Think of it this way, three transformations that all register read > > request > > > and read response headers, the first will run first for both and the > last > > > will run last for both. This is the fundamental problem. > > > > > > Nonetheless, I think this argument is independent of the fact that gzip > > > should be in the core anyway because it is such a common operation and > > > there are known issues with transformations related to performance. > > > > > > I agree. I was trying to understand what the problem is, such that we can > > solve this in some reasonable way. This sort of boils down to the lack of > > communications between plugins in general, which is why we do have both > > @-headers, transaction data and the txn / ssn slots. But there’s no > generic > > way to communicate intent like this across plugins. And certainly we’re > > lacking a “priority” or “ordering” of registered hooks. > > > > I’d imagine what you are solving for gzip by putting it into core is only > > a subset of the big picture problem? How are the gzip / ungzip processing > > configured? Via config file or via new APIs or hooks? > > > > — Leif > > > > > > > > Brian > > > > > > On Tuesday, March 24, 2015, Leif Hedstrom <zw...@apache.org> wrote: > > > > > >> Hmmm I'm not anywhere near a computer but continuations should get > > called > > >> in the order they were added. Is that not the case ? In the case of > > global > > >> plugins that's the order in the plugin.config. In remap plugins, it's > > the > > >> order in the remap rule. Where it gets iffy is when plugins adds new > > >> continuations as part of the handler of another callback. > > >> > > >> Maybe I'm still just confused :) > > >> > > >> -- Leif > > >> > > >> > > >> > > >>> On Mar 24, 2015, at 1:20 PM, Dzmitry Markovich < > > dmarkov...@linkedin.com > > >> <javascript:;>> wrote: > > >>> > > >>> Setting up unzip plugin at first and zip plugin at last will not > help. > > >> We tried this already. > > >>> There is no clear way to control that first response hook will go to > > >> unzip plugin and last response hook will go to zip plugin. > > >>> > > >>> Also, there might be a case when based on some run-time logic plugins > > in > > >> the chain will not touch the response body (it is already gzipped and > > it is > > >> static content file - js/css) in this case we should not execute > unzip > > and > > >> zip plugin at all. But if it is just unzip and gzip plugin we cant > > >> extrapolate this logic and we will execute gzip/gunzip where it is not > > >> necessary. > > >>> > > >>> And at this point only ATS know about the "body/data" hooks > registered > > >> for transaction, so ATS will know whether it is necessary to unzip > > >> response, because there are "body/data" hooks registered. And on the > way > > >> out to the client, based on Accept-Encoding header ATS will zip the > > body if > > >> necessary. > > >>> > > >>> -Dzmitry > > >>> > > >>> ________________________________________ > > >>> From: gang li [portl4t...@gmail.com <javascript:;>] > > >>> Sent: Tuesday, March 24, 2015 6:38 AM > > >>> To: dev@trafficserver.apache.org <javascript:;> > > >>> Cc: Roland Zink; Brian Geffon; Cynthia Gu > > >>> Subject: Re: Adding Gzip/Gunzip feature in ATS core > > >>> > > >>> I got this problem before, I think we can just set the unzip plugin > at > > >>> first and zip plugin at last. > > >>> > > >>>> On Tue, Mar 24, 2015 at 8:39 PM, Leif Hedstrom <zw...@apache.org > > >> <javascript:;>> wrote: > > >>>> > > >>>> Makes sense. So it's about knowing if a plugin needs the plain text > or > > >>>> not. I'm not arguing against gzip in the core, we should do that for > > >>>> performance reasons. > > >>>> > > >>>> Not quite sure I see the argument for which plugin does the gzip > > though, > > >>>> that sounds somewhat of a configuration issue. Something w suck at > :) > > >> The > > >>>> order of global plugins is well defined, but maybe txn hooks makes > > this > > >>>> trickier too. > > >>>> > > >>>> Cheers, > > >>>> > > >>>> -- Leif > > >>>> > > >>>> > > >>>> > > >>>>> On Mar 24, 2015, at 7:50 AM, Roland Zink < > rola...@flashnetworks.com > > >> <javascript:;>> > > >>>> wrote: > > >>>>> > > >>>>> I think they can see the updated headers but this doesn't help. A > > >> plugin > > >>>> can't know if the next plugin in the chain needs the plain text or > > even > > >> if > > >>>> there is a next plugin. So every plugin is doing the gzip at the > end. > > >> The > > >>>> next plugin in the chain then will ungzip again. > > >>>>> > > >>>>> Regards, > > >>>>> Roland > > >>>>> > > >>>>> -----Original Message----- > > >>>>> From: Leif Hedstrom [mailto:zw...@apache.org <javascript:;>] > > >>>>> Sent: Tuesday, March 24, 2015 12:43 PM > > >>>>> To: dev@trafficserver.apache.org <javascript:;> > > >>>>> Cc: Brian Geffon; Cynthia Gu > > >>>>> Subject: Re: Adding Gzip/Gunzip feature in ATS core > > >>>>> > > >>>>> I guess I don't understand why the chaining makes a difference. > > That's > > >>>> what I was asking in the previous email. If two plugins are chained, > > one > > >>>> should be able to detect gzip or not gzip the same way it does as if > > it > > >> was > > >>>> not chained. What am I missing ? Does each chained plugin not see > the > > >>>> updated header? Do they all see the same unmodified headers? That > > seems > > >>>> unfortunate to say the least if it's so ;/. > > >>>>> > > >>>>> -- Leif > > >>>>> > > >>>>> > > >>>>> > > >>>>>> On Mar 24, 2015, at 12:39 AM, Shu Kit Chan <chanshu...@gmail.com > > >> <javascript:;>> > > >>>> wrote: > > >>>>>> > > >>>>>> ESI should handle these situations correctly. e.g. it won't gzip > the > > >>>>>> response if it is already gzipped. > > >>>>>> So it works fine in a standalone way. > > >>>>>> > > >>>>>> The problems begin when we starts to chain a few of these > standalone > > >>>>>> transformation plugins together. Each of them will try to do the > > right > > >>>>>> thing when they are used in a standalone way and perform > gunzip/gzip > > >>>>>> correctly. So chaining them together means I will be unnecessarily > > >>>>>> doing multiple gunzip/gzip on the contents. > > >>>>>> > > >>>>>> The other way is make them plugins be aware of each other and work > > >>>>>> together and don't unnecessarily do gunzip/gzip. But then this > makes > > >>>>>> the plugins no longer standalone > > >>>>>> > > >>>>>> +1 on Dzmitry's idea. > > >>>>>> > > >>>>>> Kit > > >>>>>> > > >>>>>>> On Mon, Mar 23, 2015 at 5:11 PM, Leif Hedstrom <zw...@apache.org > > >> <javascript:;>> > > >>>> wrote: > > >>>>>>> > > >>>>>>> > > >>>>>>>>> On Mar 23, 2015, at 6:08 PM, Dzmitry Markovich > > >>>>>>>> <dmarkov...@linkedin.com.INVALID> wrote: > > >>>>>>>> > > >>>>>>>> Hello ATS experts, > > >>>>>>>> > > >>>>>>>> Today it is very common that http data that goes to the wire is > > >>>>>>> compressed. And we think it is a time to standardize this process > > in > > >>>>>>> ATS core, since it is very common operation. > > >>>>>>>> > > >>>>>>>> Today multiple plugins running on the same tier (that operates > > with > > >>>>>>> response body) do not have enough flexibility to handle > > gzip/gunzip - > > >>>>>>> and most of the times simply every plugin do gzip/gunzip. This > lead > > >>>>>>> to the situation when we do gzip/gunzip multiple times while we > > >>>>>>> process the http response. This leads to the bad CPU utilization > > and > > >>>> performance. > > >>>>>>>> > > >>>>>>>> Yes, there is gzip/gunzip logic in atscpp API - so plugins can > > >>>>>>>> simply > > >>>>>>> use those. But today's ATS architecture does not allow us to > fully > > >>>>>>> control the order of hook callbacks for every plugin per request > - > > it > > >>>>>>> means there is no non-hacky way to prevent multiple gzip/gunzip > > calls > > >>>>>>> while processing the request. And there is no way to do that with > > the > > >>>>>>> assumption that plugins dnot know about each other. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Here is our high level proposal, just to start the conversation > > >> going: > > >>>>>>>> - Have gzip/gunzip logic landed on ATS core, so engineers can > ask > > >>>>>>>> ATS > > >>>>>>> (vi config parameter) to take care about gzip/gunzip logic for > them > > >>>>>>>> 1. Only ATS know when the body arrived to first plugin - so ATS > at > > >>>>>>> this point ungzip the body; > > >>>>>>>> 2. Only ATS knows when body is processed by all plugins and > should > > >>>>>>>> go > > >>>>>>> over the wire - so ATS at this point gzip the body if client > > >> supported > > >>>> it. > > >>>>>>> > > >>>>>>> > > >>>>>>> Moving gzip to the core seems a generally good idea. Probably in > > some > > >>>>>>> extensible way such that we can add future compression encoding > (I > > >>>>>>> think Chrome has support for some better ones?). > > >>>>>>> > > >>>>>>> Now, I’m slightly confused, and/or concerned, that our plugins do > > not > > >>>>>>> handle this well. I would have imagined that something doing gzip > > >>>>>>> would not do so if the content comes back with Content-Encoding: > > >>>>>>> gzip. So shouldn’t e.g. ESI detect if the gzip plugin has already > > >> done > > >>>> so? > > >>>>>>> > > >>>>>>> Is there a reason why a plugin *can’t* detect this? If not, we > > should > > >>>>>>> fix the plugins regardless of this RFE, it seems like a broken > > >>>>>>> behavior if a plugin can gzip something that’s already CE: gzip. > > >>>>>>> > > >>>>>>> Cheers, > > >>>>>>> > > >>>>>>> — Leif > > > > >
