Re: PIP 28: Improve pulsar proxy log messages

2019-02-01 Thread sun yao
Hi Dave,      Noted, thanks for providing this so good name . On Thursday, January 31, 2019, 11:40:01 PM PST, Dave Fisher wrote: Hi Sun Yao, When you started this thread writing that the proxy is almost like a gateway. You were correct. I think that what is wanted is for Proxies to usu

Re: PIP 28: Improve pulsar proxy log messages

2019-01-31 Thread Dave Fisher
Hi Sun Yao, When you started this thread writing that the proxy is almost like a gateway. You were correct. I think that what is wanted is for Proxies to usually be lightweight / highly performant, and optionally become a highly resilient Gateway. Let’s call this new feature Gateway mode. Re

Re: PIP 28: Improve pulsar proxy log messages

2019-01-31 Thread sun yao
Hi Matteo,       You are right, will give it a shot by using "netty".      Thanks again. Sam On Thursday, January 31, 2019, 6:53:03 AM PST, Matteo Merli wrote: Sure, it would be good to have that as an option. It's just that it would need to be a completely different "operating mod

Re: PIP 28: Improve pulsar proxy log messages

2019-01-31 Thread Matteo Merli
Sure, it would be good to have that as an option. It's just that it would need to be a completely different "operating mode" for the proxy, since more than the logging, is the parsing of each frame / command that will be "expensive". It should be possible to do that by controlling the handlers in

Re: PIP 28: Improve pulsar proxy log messages

2019-01-31 Thread Sijie Guo
I think if this feature is controlled under some flags/settings. It should be a good tradeoff between performance concerns and debuggability/visibility. As what Yao said, people can choose what level of details that they would like to see. It is similar to log4j's debug level. Thanks, Sijie On T

Re: PIP 28: Improve pulsar proxy log messages

2019-01-30 Thread sun yao
Hi Sijie, Matteo and Joe ,     Thanks for your kind and professional opinions .     For performance concern from Joe, what I think is like this, different log levels for different detail outputs and let dev decide which is good for them, and I also mentioned in PIP about we can disable this featu

Re: PIP 28: Improve pulsar proxy log messages

2019-01-30 Thread Joe F
I run Pulsar proxy in production, and the same concern here. I don't think we can get any of these metrics unless we start parsing protocol, and definitely its going to make everything slower, and create additional memory and GC pressures. Joe On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli wrot

Re: PIP 28: Improve pulsar proxy log messages

2019-01-30 Thread Matteo Merli
Missed to comment on this :) One issue might arise from the fact that proxy is not actually parsing each and every request. The proxy only "speaks" Pulsar protocol initial Connect/Connected handshake, in which the proxy forwards the client credentials and route it through the appropriate broker.

Re: PIP 28: Improve pulsar proxy log messages

2019-01-30 Thread Sijie Guo
Yao, Thank you for your proposal! The proposal looks good to me +1. In general, the ASF is using lazy consensus for a lot of things, like adopting PIPs. basically, if there is no objection coming up within a period (typically 1~2 days), you are good to pull the trigger and send PRs :-) - Sijie

PIP 28: Improve pulsar proxy log messages

2019-01-26 Thread sun yao
Hi folks,       Pulsar Proxy is almost a gateway for all pulsar requests, it would be helpful if it can record more details for the traffic, like source, target, session id, response time(different stage) for each request, even for message body.       I am proposing an improve for pulsar proxy