* So I think we could not replace completely the Pulsar protocol by gRPC.
Sure, I understand that it’s unlikely possible to implement a full replacement. I’m rather talking about it as a better, more type-safe alternative to WebSocket and REST APIs, with a similar amount of supported features. https://pulsar.apache.org/docs/3.1.x/client-libraries-websocket/ https://pulsar.apache.org/docs/3.1.x/client-libraries-rest/ I’m not talking about the performance at all. I think, in the worst case it should be at least the same performance as the WebSocket protocol implementation. From my understanding, it should be simpler to maintain it than REST or WebSocket protocols due to type safety, isn’t it? * You have some kind of type hinting with the OpenAPI spec that you can use to generate client SDKs (eg. with openapi-generator. Sure, but openapi + Java annotations by its nature are very error-prone. About a year ago when I needed to use Pulsar Admin API via REST, I reported 4 bugs about wrong fields in the response object and wrong status codes. I suspect there are many more such errors. The current approach I see in the following way: We use strongly typed Java => then we remove all type information and re-define it semi-manually via OpenAPI annotations (and introduce tons of errors) => then again trying to make it strongly typed by running OpenAPI client generator. I’d prefer the approach where it’s easy and safe to generate a client for any language without bugs introduced by Java API => REST => OpenAPI client conversions. In rare cases where REST is still needed (its bash scripts mostly, or am I missing anything?), it’s not hard to run/embed https://github.com/grpc-ecosystem/grpc-gateway I think it’s much easier to implement it once in Java, and then generate a client for target languages. The amount of work should be about the same as implementing it for any other language, like here https://github.com/apache/pulsar-client-go/tree/master/pulsaradmin/pkg/admin The difference is that this way you don’t need to re-implement it for any other language anymore. Best, Kiryl From: Christophe Bornet <bornet.ch...@gmail.com> Date: Wednesday, November 1, 2023 at 12:43 AM To: dev@pulsar.apache.org <dev@pulsar.apache.org> Subject: Re: Question about Pulsar gRPC client(s) Hi Kiryl, Thanks for mentioning pulsar-grpc. Indeed, using gRPC simplifies the implementation of the networking logic (keep-alive, reconnection, flow control,…). On the other hand, the Java gRPC implementation makes a lot of buffer copies to cleanly separate the network and app layers but that takes a toll on performance. Compared to that, the broker Pulsar protocol impl is optimized to not do copies between the consumer/producer endpoints and the bookkeeper client. So I think we could not replace completely the Pulsar protocol by gRPC. We could maybe support both but it’s a lot of work to maintain both protocols. (I kind of gave up maintaining pulsar-grpc because of the amount of work compared to the number of users, but if there’s interest I can reconsider). Another possibility would be to do a proxy instead of a low-level protocol handler. A bit like the WebSocket proxy. This would be far less work to maintain as it would use the Pulsar protocol to communicate with the brokers. It could be done as a Proxy extension. Compared to the WS proxy, this would provide type safety, discovery, and so on… As for the Admin, it’s a bit the same. It would be a bunch of work to support both gRPC and REST. You have some kind of type hinting with the OpenAPI spec that you can use to generate client SDKs (eg. with openapi-generator. I wonder what others have to say. Christophe Le mar. 31 oct. 2023 à 19:57, Kiryl Valkovich <kiryl_valkovich@teal.tools> a écrit : > Hi! Am I understanding it right, that if this project > https://github.com/cbornet/pulsar-grpc is merged to the apache/pulsar > repo, we could easily cover non-mainstream platforms that are supported by > gRPC, but don't have ready-to-use Pulsar clients? > > https://github.com/apache/pulsar/wiki/PIP-59:-gPRC-Protocol-Handler > > Similar to already supported language-agnostic client interfaces - REST > and WebSocket. > > Actively maintained gRPC libraries I found (19, or 15 if considering JVM > languages and web as duplicates): > - [C# / .NET](https://grpc.io/docs/languages/csharp/) > - [C++](https://grpc.io/docs/languages/cpp/) > - [Dart](https://grpc.io/docs/languages/dart/) > - [Go](https://grpc.io/docs/languages/go/) > - [Java](https://grpc.io/docs/languages/java/) > - [Kotlin](https://grpc.io/docs/languages/kotlin/) > - [Node](https://grpc.io/docs/languages/node/) > - [Objective-C](https://grpc.io/docs/languages/objective-c/) > - [PHP](https://grpc.io/docs/languages/php/) > - [Python](https://grpc.io/docs/languages/python/) > - [Ruby](https://grpc.io/docs/languages/ruby/) > - [OCaml](https://github.com/dialohq/ocaml-grpc) > - [Haskell](https://github.com/awakesecurity/gRPC-haskell) > - [Elixir](https://github.com/elixir-grpc/grpc) > - [Rust](https://github.com/hyperium/tonic) > - [Scala](https://scalapb.github.io/) > - [Swift](https://github.com/grpc/grpc-swift) > - Web client: https://github.com/grpc/grpc-web > - Web client 2: https://connectrpc.com/docs/web/getting-started > > Actively maintained Pulsar libraries (8): > - Java > - C++ > - Python > - Go > - Node.js > - C# > - PHP > - Rust > > Is there any reason for not merging it to the apache/pulsar? > > I would definitely prefer to work with a statically typed gRPC client > instead of REST or WebSocket. > > By the way, the same can work for the Pulsar Admin API. Implement the gRPC > server once in Java, and we have full-featured native statically typed > (where applicable :)) Pulsar Admin clients for any platform. > > Best, > Kiryl >
