Hi Matteo,
You are right, will give it a shot by using "netty".
Thanks again.
Sam
On Thursday, January 31, 2019, 6:53:03 AM PST, Matteo Merli
<[email protected]> wrote:
Sure, it would be good to have that as an option.
It's just that it would need to be a completely different "operating
mode" for the proxy,
since more than the logging, is the parsing of each frame / command
that will be
"expensive".
It should be possible to do that by controlling the handlers in the
netty channel pipeline,
by having different handlers based on the config.
--
Matteo Merli
<[email protected]>
On Thu, Jan 31, 2019 at 6:38 AM Sijie Guo <[email protected]> wrote:
>
> I think if this feature is controlled under some flags/settings. It should be
> a good tradeoff between performance concerns and debuggability/visibility.
>
> As what Yao said, people can choose what level of details that they would
> like to see. It is similar to log4j's debug level.
>
> Thanks,
> Sijie
>
> On Thu, Jan 31, 2019 at 2:30 PM sun yao <[email protected]> wrote:
>>
>> Hi Sijie, Matteo and Joe ,
>>
>> Thanks for your kind and professional opinions .
>>
>> For performance concern from Joe, what I think is like this, different
>>log levels for different detail outputs and let dev decide which is good for
>>them, and I also mentioned in PIP about we can disable this feature by
>>parameter in some critical scenarios.
>>
>> For protocol implement from Matteo and Joe, we could try to implement
>>it for we need to get from the proxy, like what it did in other proxies,
>>proxysql and twemproxy , or any others.
>>
>> The point for this feature is we could have more details for our traffic,
>>not only for performances, and also resources usage and msg details for debug
>>and audit, like client issues, proxy issue or broker issues or ...
>>
>> Thanks.
>>
>>
>>
>>
>>
>> On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F
>> <[email protected]> wrote:
>>
>>
>> I run Pulsar proxy in production, and the same concern here.
>>
>> I don't think we can get any of these metrics unless we start parsing
>> protocol, and definitely its going to make everything slower, and create
>> additional memory and GC pressures.
>>
>> Joe
>>
>>
>> On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <[email protected]>
>> wrote:
>>
>> > Missed to comment on this :)
>> >
>> > One issue might arise from the fact that proxy is not actually parsing
>> > each and every request.
>> >
>> > The proxy only "speaks" Pulsar protocol initial Connect/Connected
>> > handshake, in which the proxy forwards the client credentials and
>> > route it through the appropriate broker.
>> >
>> > After the initial handshake, the proxy is essentially degrading itself
>> > into a 1-1 TCP proxy, patching 2 TCP connections through
>> > without checking the commands anymore. That's the reason we only have
>> > metrics around bytes/s and "operation/s" where
>> > operation maps to a "buffer" we're getting from a socket (with no
>> > direct relation to IP packets).
>> >
>> >
>> > --
>> > Matteo Merli
>> > <[email protected]>
>> >
>> > On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <[email protected]> wrote:
>> > >
>> > > Yao,
>> > >
>> > > Thank you for your proposal! The proposal looks good to me +1.
>> > >
>> > > In general, the ASF is using lazy consensus for a lot of things, like
>> > > adopting PIPs. basically, if there is no objection coming up within a
>> > > period (typically 1~2 days), you are good to pull the trigger and send
>> > PRs
>> > > :-)
>> > >
>> > > - Sijie
>> > >
>> > > On Sun, Jan 27, 2019 at 11:39 AM sun yao <[email protected]
>> > .invalid>
>> > > wrote:
>> > >
>> > > >
>> > > >
>> > > > Hi folks,
>> > > > Pulsar Proxy is almost a gateway for all pulsar requests, it
>> > would
>> > > > be helpful if it can record more details for the traffic, like source,
>> > > > target, session id, response time(different stage) for each request,
>> > even
>> > > > for message body. I am proposing an improve for pulsar proxy, for
>> > > > more information : PIP:
>> > > >
>> > https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
>> > > > Feel free to let me know any ideas or suggestions for
>> > this
>> > > > feature. Thanks.
>> > > >
>> >
