Hi Dave,
Noted, thanks for providing this so good name .
On Thursday, January 31, 2019, 11:40:01 PM PST, Dave Fisher
<[email protected]> wrote:
Hi Sun Yao,
When you started this thread writing that the proxy is almost like a gateway.
You were correct. I think that what is wanted is for Proxies to usually be
lightweight / highly performant, and optionally become a highly resilient
Gateway.
Let’s call this new feature Gateway mode.
Regards,
Dave
Sent from my iPhone
> On Jan 31, 2019, at 8:34 PM, sun yao <[email protected]> wrote:
>
>
>
> Hi Matteo,
> You are right, will give it a shot by using "netty".
> Thanks again.
>
> Sam
>
> On Thursday, January 31, 2019, 6:53:03 AM PST, Matteo Merli
><[email protected]> wrote:
>
> Sure, it would be good to have that as an option.
>
> It's just that it would need to be a completely different "operating
> mode" for the proxy,
> since more than the logging, is the parsing of each frame / command
> that will be
> "expensive".
>
> It should be possible to do that by controlling the handlers in the
> netty channel pipeline,
> by having different handlers based on the config.
>
>
> --
> Matteo Merli
> <[email protected]>
>
>> On Thu, Jan 31, 2019 at 6:38 AM Sijie Guo <[email protected]> wrote:
>>
>> I think if this feature is controlled under some flags/settings. It should
>> be a good tradeoff between performance concerns and debuggability/visibility.
>>
>> As what Yao said, people can choose what level of details that they would
>> like to see. It is similar to log4j's debug level.
>>
>> Thanks,
>> Sijie
>>
>>> On Thu, Jan 31, 2019 at 2:30 PM sun yao <[email protected]> wrote:
>>>
>>> Hi Sijie, Matteo and Joe ,
>>>
>>> Thanks for your kind and professional opinions .
>>>
>>> For performance concern from Joe, what I think is like this, different
>>>log levels for different detail outputs and let dev decide which is good for
>>>them, and I also mentioned in PIP about we can disable this feature by
>>>parameter in some critical scenarios.
>>>
>>> For protocol implement from Matteo and Joe, we could try to implement
>>>it for we need to get from the proxy, like what it did in other proxies,
>>>proxysql and twemproxy , or any others.
>>>
>>> The point for this feature is we could have more details for our
>>>traffic, not only for performances, and also resources usage and msg details
>>>for debug and audit, like client issues, proxy issue or broker issues or ...
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>> On Wednesday, January 30, 2019, 11:26:07 AM PST, Joe F
>>> <[email protected]> wrote:
>>>
>>>
>>> I run Pulsar proxy in production, and the same concern here.
>>>
>>> I don't think we can get any of these metrics unless we start parsing
>>> protocol, and definitely its going to make everything slower, and create
>>> additional memory and GC pressures.
>>>
>>> Joe
>>>
>>>
>>> On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli <[email protected]>
>>> wrote:
>>>
>>>> Missed to comment on this :)
>>>>
>>>> One issue might arise from the fact that proxy is not actually parsing
>>>> each and every request.
>>>>
>>>> The proxy only "speaks" Pulsar protocol initial Connect/Connected
>>>> handshake, in which the proxy forwards the client credentials and
>>>> route it through the appropriate broker.
>>>>
>>>> After the initial handshake, the proxy is essentially degrading itself
>>>> into a 1-1 TCP proxy, patching 2 TCP connections through
>>>> without checking the commands anymore. That's the reason we only have
>>>> metrics around bytes/s and "operation/s" where
>>>> operation maps to a "buffer" we're getting from a socket (with no
>>>> direct relation to IP packets).
>>>>
>>>>
>>>> --
>>>> Matteo Merli
>>>> <[email protected]>
>>>>
>>>>> On Wed, Jan 30, 2019 at 9:19 AM Sijie Guo <[email protected]> wrote:
>>>>>
>>>>> Yao,
>>>>>
>>>>> Thank you for your proposal! The proposal looks good to me +1.
>>>>>
>>>>> In general, the ASF is using lazy consensus for a lot of things, like
>>>>> adopting PIPs. basically, if there is no objection coming up within a
>>>>> period (typically 1~2 days), you are good to pull the trigger and send
>>>> PRs
>>>>> :-)
>>>>>
>>>>> - Sijie
>>>>>
>>>>> On Sun, Jan 27, 2019 at 11:39 AM sun yao <[email protected]
>>>> .invalid>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Hi folks,
>>>>>> Pulsar Proxy is almost a gateway for all pulsar requests, it
>>>> would
>>>>>> be helpful if it can record more details for the traffic, like source,
>>>>>> target, session id, response time(different stage) for each request,
>>>> even
>>>>>> for message body. I am proposing an improve for pulsar proxy, for
>>>>>> more information : PIP:
>>>>>>
>>>> https://github.com/apache/pulsar/wiki/PIP-28%3A-Improve-pulsar-proxy-log-messages
>>>>>> Feel free to let me know any ideas or suggestions for
>>>> this
>>>>>> feature. Thanks.
>>>>>>
