I think if this feature is controlled under some flags/settings. It should be a good tradeoff between performance concerns and debuggability/visibility.
As what Yao said, people can choose what level of details that they would like to see. It is similar to log4j's debug level. Thanks, Sijie On Thu, Jan 31, 2019 at 2:30 PM sun yao <foreversun...@yahoo.com> wrote: > Hi Sijie, Matteo and Joe , > > Thanks for your kind and professional opinions . > > For performance concern from Joe, what I think is like this, different > log levels for different detail outputs and let dev decide which is good > for them, and I also mentioned in PIP about we can disable this feature by > parameter in some critical scenarios. > > For protocol implement from Matteo and Joe, we could try to > implement it for we need to get from the proxy, like what it did in other > proxies, proxysql <https://github.com/sysown/proxysql/wiki/Query-Logging> > and twemproxy > <https://github.com/twitter/twemproxy/blob/c5c725d2b45715e378909ebc909b0492591c4715/notes/recommendation.md> > , > or any others. > > The point for this feature is we could have more details for our > traffic, not only for performances, and also resources usage and msg > details for debug and audit, like client issues, proxy issue or broker > issues or ... > > Thanks. > > > > > > On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F < > joefranc...@gmail.com> wrote: > > > I run Pulsar proxy in production, and the same concern here. > > I don't think we can get any of these metrics unless we start parsing > protocol, and definitely its going to make everything slower, and create > additional memory and GC pressures. > > Joe > > > On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <matteo.me...@gmail.com> > wrote: > > > Missed to comment on this :) > > > > One issue might arise from the fact that proxy is not actually parsing > > each and every request. > > > > The proxy only "speaks" Pulsar protocol initial Connect/Connected > > handshake, in which the proxy forwards the client credentials and > > route it through the appropriate broker. > > > > After the initial handshake, the proxy is essentially degrading itself > > into a 1-1 TCP proxy, patching 2 TCP connections through > > without checking the commands anymore. That's the reason we only have > > metrics around bytes/s and "operation/s" where > > operation maps to a "buffer" we're getting from a socket (with no > > direct relation to IP packets). > > > > > > -- > > Matteo Merli > > <matteo.me...@gmail.com> > > > > On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <guosi...@gmail.com> wrote: > > > > > > Yao, > > > > > > Thank you for your proposal! The proposal looks good to me +1. > > > > > > In general, the ASF is using lazy consensus for a lot of things, like > > > adopting PIPs. basically, if there is no objection coming up within a > > > period (typically 1~2 days), you are good to pull the trigger and send > > PRs > > > :-) > > > > > > - Sijie > > > > > > On Sun, Jan 27, 2019 at 11:39 AM sun yao <foreversun...@yahoo.com > > .invalid> > > > wrote: > > > > > > > > > > > > > > > Hi folks, > > > > Pulsar Proxy is almost a gateway for all pulsar requests, it > > would > > > > be helpful if it can record more details for the traffic, like > source, > > > > target, session id, response time(different stage) for each request, > > even > > > > for message body. I am proposing an improve for pulsar proxy, > for > > > > more information : PIP: > > > > > > > https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages > > > > Feel free to let me know any ideas or suggestions for > > this > > > > feature. Thanks. > > > > > > >
