Hi Willy, Thank you for the great suggestion! I agree that it would be more general for other use cases to have support for only including or excluding certain headers.
I can look into the implementation and send a patch for that if no other concerns are there. We did try the lua implementation, but we observed there would be a lot more malloc/free that impacted performance and took a lot of CPUs when QPS was high (100k+). Ruei-Bang ________________________________ From: Willy Tarreau <w...@1wt.eu> Sent: Thursday, October 12, 2023 8:56 PM To: Ruei-Bang Chen <ruec...@linkedin.com> Cc: haproxy@formilux.org <haproxy@formilux.org> Subject: Re: Request for feedback: Add fetcher for all headers excluding cookies Hi Ruei-Bang, On Thu, Oct 12, 2023 at 09:44:22PM +0000, Ruei-Bang Chen wrote: > Hi team, > > I am writing to gather feedback on an idea before doing the implementation. > > > We have a use-case where we need all headers except for cookies. Currently, > the fetcher "req.hdrs" / "res.hdrs" returns all headers including cookie / > set-cookie headers. > > However, the cookie header itself can get pretty huge and cause unnecessary > overhead when we want to store / send / parse the string variable. > > In a high qps environment, this will cause big degradation to the latency of > the requests, so the proposal is to add a new fetcher for sending the headers > excluding cookies. > > Let me know what you think. If there are no concerns, I can send a patch. I'm not necessarily against it but I fear that it opens a can of worms. Others will say that User-Agent is often too verbose and will want to skip it, then it will be Referer, then X-Pad, then all Connection and Cache-control headers etc. So if we go in that direction, we'd need to keep the exclusion list configurable. For example we could imagine that req.hdrs takes an optional list of arguments that are either a restricted list of headers to extract, or to exclude depending on the syntax. I think we can support up to about 10 arguments so maybe that could do the job. For example: req.hdrs() => all req.hdrs(host,referer,x-forwarded-for) => only these req.hdrs(!cookie,referer,connection) => all but these Alternately, have you tried to make your own sample-fetch function in Lua to deal with the headers you want ? I'm pretty sure that it was earlier used like this before becoming a native function. Thanks, Willy
