I have a doc (
https://docs.google.com/document/d/1-WxZTNu9RrLhh5O7Dl5PbnKqz3e5gm1x3gDBBhszVF8/edit#heading=h.n02teqc95avo)
on writing an IO in Beam.
Some of it is specific to using SplittableDoFn, but most of it is
applicable to doing IO at scale in beam in general.
I hope this helps
John
On Mo
According what you said guys, creating a Http io connector makes sense if
you have scenarios like these:
- I want to monitor the number of http successful or failed requests in a
window of time.
- I want to track transactions that require multiple http requests to be
done and we need to save a sta
Hi,
While ago (at the very early stage of Beam :)), I proposed to create a
HTTP/REST source/sink (we should still have the Jira :)).
However, we didn't have a consensus in terms of features (I proposed
something very simple). Splittable DoFn didn't exist at that time.
So, if we want to move forwa
Connectors are written using ParDos. A connector (source) may use a source
framework (Splittable DoFn is the recommended framework currently) or may
be written using regular ParDos. The main advantages of a source framework
are various features provided by such frameworks (progress reporting,
dynam
