I have a doc ( https://docs.google.com/document/d/1-WxZTNu9RrLhh5O7Dl5PbnKqz3e5gm1x3gDBBhszVF8/edit#heading=h.n02teqc95avo) on writing an IO in Beam.
Some of it is specific to using SplittableDoFn, but most of it is applicable to doing IO at scale in beam in general. I hope this helps John On Mon, Jun 26, 2023 at 10:57 AM Juan Romero <jsrf...@gmail.com> wrote: > According what you said guys, creating a Http io connector makes sense if > you have scenarios like these: > - I want to monitor the number of http successful or failed requests in a > window of time. > - I want to track transactions that require multiple http requests to be > done and we need to save a state. > - Implement idempotency to avoid duplicates in the http request in a > window of time. > > Respect with the functionality to distribute the incoming request to > different nodes, I think that in the current world it is more > responsibility for the API side, because this is responsibility of a load > balancer that receive the request and distribute them across multiple > containers or VM. > > I want to validate your opinion with the examples I gave or if you have > other more scenarios that help me to clarify my ideas. > > Thanks guys! > > > > > > El El vie, 23 de jun. de 2023 a la(s) 10:46 p.m., Jean-Baptiste Onofré < > j...@nanthrax.net> escribió: > >> Hi, >> >> While ago (at the very early stage of Beam :)), I proposed to create a >> HTTP/REST source/sink (we should still have the Jira :)). >> However, we didn't have a consensus in terms of features (I proposed >> something very simple). Splittable DoFn didn't exist at that time. >> >> So, if we want to move forward on HTTP/REST, we have to list the >> features and expected behavior. >> >> Regards >> JB >> >> On Sat, Jun 24, 2023 at 1:47 AM Juan Romero <jsrf...@gmail.com> wrote: >> > >> > Hi guys. I have a doubt related with it make sense to create an HTTP >> connector in Apache Beam or simply I can create a PArdo Function that make >> the http request. I want to know which advantages I would have creating an >> IO HTTP connector. >> >
