Hi, Thanks for Sijie's proposal. I resonate Rajan's comment that this project should become a peripheral to Pulsar because of the stateless nature of HTTP and scaling up a web service inside Pulsar might be affecting its current design. For that reason, I developed a project with an HTTP interface to Pulsar and webhook notification that pushes messages to consumers.
It is written in Go. The current repo is at https://github.com/kafkaesque-io/pulsar-beam It can be deployed in standalone or inside a Pulsar cluster. I know someone has started using it in their production trial. It makes sense for the HTTP interface to become an optional add-on to Pulsar. I would like to contribute this back to Apache Software Foundation as a standalone repo like Pulsar Manager. What do you think? Let me know how I can help. Cheers Ming On Wed, 13 May 2020 at 17:00, Rajan Dhabalia <rdhaba...@apache.org> wrote: > *Hi,I would like to add little history about Websocket vs REST API > implementation in Pulsar. Before open-sourcing Pulsar and WebSocket proxy > in Yahoo, we had implemented and evaluated both WebSocket and REST-HTTP > APIs options and then decided to not introduce REST-API and kept Websocket > proxy solution in Pulsar. Let me add a few points to understand why we > should avoid adding REST API in Pulsar. 1. One of the advantages of REST > architecture is statelessness which means that any server can handle any > request. However, it’s true for Websocket protocol as well but Websoket is > a bi-directional protocol with full duplex communication and client-server > can communicate on a single TCP connection which avoids different overheads > such as authentication on every request that needs to be done on REST-Http > api.2. If client doesn’t use persist connection correctly on REST-HTTP then > every produce/consume request can go to a different server and it will > create a lot of complexity to manage the control flow, producer/consumer > queuing at REST-server which internally manages pulsar-client, clean up > unused resources, etc.3. Performance and scalability: Websocket is highly > scalable and provides higher performance and throughput compare to > REST-HTTP.4. Almost all programming languages and web-browsers > <https://caniuse.com/#feat=websockets> support WebSocket.5. Pulsar is > known > for performance and scalability, so, we try to select solutions that work > with scale and provide better performance.6. Pulsar broker gives an option > to start WebSocket proxy along with broker and that doesn’t require extra > component to manage. So, if one doesn’t want to manage additional component > then one can start with the broker process as well. 7. Other messaging > system also moving away from their legacy REST-HTTP api to WebSocket > because of performance and scalability concerns.* > *So, I think let’s not add this feature in Pulsar but it can be added a > utility project outside of pulsar if it is needed for specific usecases.* > > *Thanks,* > > *Rajan* > > On Wed, May 13, 2020 at 3:25 AM Sijie Guo <si...@apache.org> wrote: > > > Hi all, > > > > Some of the people have been asking for a REST-proxy like equivalent > > component in Pulsar. Pulsar already has a web service that provides the > > management API over a Pulsar cluster. We don't actually need a separate > > component or service for supporting streaming events in and out of Pulsar > > using HTTP. We can just add endpoints to the existing web server to > support > > those functionalities. > > > > I'd like to start a discussion on adding REST endpoints to the current > web > > server for producing, consuming and reading messages. So people can use > > HTTP for streaming events in and out of Pulsar without setting up > > additional components. > > > > The full proposal is written at > > > > > https://github.com/apache/pulsar/wiki/PIP-64:-Introduce-REST-endpoints-for-producing,-consuming-and-reading-messages > > . > > > > Feedback and comments are welcome! > > > > - Sijie > > > -- Powered by Pulsar Engineering, Kafkaesque
