Hi Reed, Something will have to produce the POST request body into Kafka. We do this at the Wikimedia Foundation with a service called EventGate <https://github.com/wikimedia/eventgate>. I've got a 3 part blog series in which the 3rd entry <https://techblog.wikimedia.org/2020/09/24/wikimedias-event-data-platform-event-intake/> explains how we use EventGate.
-Andrew Otto SRE, Wikimedia Foundation On Tue, Apr 27, 2021 at 11:42 PM Ran Lupovich <ranlupov...@gmail.com> wrote: > Hi, have a look for Rest Proxy component as part of the kafka eco system > > בתאריך יום ד׳, 28 באפר׳ 2021, 01:27, מאת Reed Villanueva < > villanuevar...@gmail.com>: > > > What is the best-practice/kafka way to get http(s) POST requests into a > > Kafka topic (kafka v2.0.0 installed on a HDP cluster)? > > Have never used kafka before and would like to know the best way that > this > > should be done. > > Basically, we have a public URL that is going to receive requests from a > > specific external URL based on event hooks ( > > https://developers.acuityscheduling.com/docs/webhooks) and I want to get > > these requests into a kafka topic. > > I've seen this ( > https://docs.confluent.io/3.0.0/kafka-rest/docs/intro.html > > ), > > but am a bit confused (again, have never used kafka before). Will there > > need to be an always-on producer to read from these event hooks to > produce > > into a topic? What is the best practice way to do this to account for > > whatever common fault tolerances that should be built into a kafka > producer > > for this kind of live event feed? No way to just automatically dump the > > requests into the topic (and avoid having to ensure such a simple > > forwarding producer is always alive (and thus not forever missing the > data > > that came in during that downtime))? > > > > Thank you > > >
