Sure, it would be good to have that as an option. It's just that it would need to be a completely different "operating mode" for the proxy, since more than the logging, is the parsing of each frame / command that will be "expensive".
It should be possible to do that by controlling the handlers in the netty channel pipeline, by having different handlers based on the config. -- Matteo Merli <matteo.me...@gmail.com> On Thu, Jan 31, 2019 at 6:38 AM Sijie Guo <guosi...@gmail.com> wrote: > > I think if this feature is controlled under some flags/settings. It should be > a good tradeoff between performance concerns and debuggability/visibility. > > As what Yao said, people can choose what level of details that they would > like to see. It is similar to log4j's debug level. > > Thanks, > Sijie > > On Thu, Jan 31, 2019 at 2:30 PM sun yao <foreversun...@yahoo.com> wrote: >> >> Hi Sijie, Matteo and Joe , >> >> Thanks for your kind and professional opinions . >> >> For performance concern from Joe, what I think is like this, different >> log levels for different detail outputs and let dev decide which is good for >> them, and I also mentioned in PIP about we can disable this feature by >> parameter in some critical scenarios. >> >> For protocol implement from Matteo and Joe, we could try to implement >> it for we need to get from the proxy, like what it did in other proxies, >> proxysql and twemproxy , or any others. >> >> The point for this feature is we could have more details for our >> traffic, not only for performances, and also resources usage and msg details >> for debug and audit, like client issues, proxy issue or broker issues or ... >> >> Thanks. >> >> >> >> >> >> On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F >> <joefranc...@gmail.com> wrote: >> >> >> I run Pulsar proxy in production, and the same concern here. >> >> I don't think we can get any of these metrics unless we start parsing >> protocol, and definitely its going to make everything slower, and create >> additional memory and GC pressures. >> >> Joe >> >> >> On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <matteo.me...@gmail.com> >> wrote: >> >> > Missed to comment on this :) >> > >> > One issue might arise from the fact that proxy is not actually parsing >> > each and every request. >> > >> > The proxy only "speaks" Pulsar protocol initial Connect/Connected >> > handshake, in which the proxy forwards the client credentials and >> > route it through the appropriate broker. >> > >> > After the initial handshake, the proxy is essentially degrading itself >> > into a 1-1 TCP proxy, patching 2 TCP connections through >> > without checking the commands anymore. That's the reason we only have >> > metrics around bytes/s and "operation/s" where >> > operation maps to a "buffer" we're getting from a socket (with no >> > direct relation to IP packets). >> > >> > >> > -- >> > Matteo Merli >> > <matteo.me...@gmail.com> >> > >> > On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <guosi...@gmail.com> wrote: >> > > >> > > Yao, >> > > >> > > Thank you for your proposal! The proposal looks good to me +1. >> > > >> > > In general, the ASF is using lazy consensus for a lot of things, like >> > > adopting PIPs. basically, if there is no objection coming up within a >> > > period (typically 1~2 days), you are good to pull the trigger and send >> > PRs >> > > :-) >> > > >> > > - Sijie >> > > >> > > On Sun, Jan 27, 2019 at 11:39 AM sun yao <foreversun...@yahoo.com >> > .invalid> >> > > wrote: >> > > >> > > > >> > > > >> > > > Hi folks, >> > > > Pulsar Proxy is almost a gateway for all pulsar requests, it >> > would >> > > > be helpful if it can record more details for the traffic, like source, >> > > > target, session id, response time(different stage) for each request, >> > even >> > > > for message body. I am proposing an improve for pulsar proxy, for >> > > > more information : PIP: >> > > > >> > https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages >> > > > Feel free to let me know any ideas or suggestions for >> > this >> > > > feature. Thanks. >> > > > >> >
