Re: Getting Started With Implementing a Runner

If Beam Runner Authoring Guide is rather high-level for you, then, at fist, I’d suggest to answer two questions for yourself: - Am I going to implement a portable runner or native one? - Which SDK I should use for this runner? Then, depending on answers, I’d suggest to take as an example one of t

Re: Getting Started With Implementing a Runner

On Fri, Jun 23, 2023, 7:37 AM Alexey Romanenko wrote: > If Beam Runner Authoring Guide is rather high-level for you, then, at > fist, I’d suggest to answer two questions for yourself: > - Am I going to implement a portable runner or native one? > The answer to this should be portable, as non-por

Re: Getting Started With Implementing a Runner

> On 23 Jun 2023, at 17:40, Robert Bradshaw via user > wrote: > > On Fri, Jun 23, 2023, 7:37 AM Alexey Romanenko > wrote: >> If Beam Runner Authoring Guide is rather high-level for you, then, at fist, >> I’d suggest to answer two questions for yourself: >> -

Re: Getting Started With Implementing a Runner

Thanks all for the responses! If Beam Runner Authoring Guide is rather high-level for you, then, at fist, > I’d suggest to answer two questions for yourself: > - Am I going to implement a portable runner or native one? > Portable sounds great, but the answer depends on how much additional cost it

Re: Getting Started With Implementing a Runner

On Fri, Jun 23, 2023 at 11:15 AM Joey Tran wrote: > Thanks all for the responses! > > If Beam Runner Authoring Guide is rather high-level for you, then, at >> fist, I’d suggest to answer two questions for yourself: >> - Am I going to implement a portable runner or native one? >> > > Portable soun

Re: Getting Started With Implementing a Runner

> > Totally doable by one person, especially given the limited feature set you > mention above. > https://docs.google.com/presentation/d/1Cso0XP9dmj77OD9Bd53C1M3W1sPJF0ZnA20gzb2BPhE > is > a good starting point as to what the relationship between a Runner and the > SDK is at a level of detail suff

Re: Getting Started With Implementing a Runner

On Fri, Jun 23, 2023 at 1:43 PM Joey Tran wrote: > Totally doable by one person, especially given the limited feature set you >> mention above. >> https://docs.google.com/presentation/d/1Cso0XP9dmj77OD9Bd53C1M3W1sPJF0ZnA20gzb2BPhE >> is >> a good starting point as to what the relationship betwee

Create IO connector for HTTP or ParDO

Hi guys. I have a doubt related with it make sense to create an HTTP connector in Apache Beam or simply I can create a PArdo Function that make the http request. I want to know which advantages I would have creating an IO HTTP connector.

Re: Getting Started With Implementing a Runner

Another advantage of a portable runner would be that it will be using well defined and backwards compatible Beam portable APIs to communicate with SDKs. I think this is specially important for runners that do not live in the Beam repo since otherwise future SDK releases could break your runner in s

Re: Create IO connector for HTTP or ParDO

Connectors are written using ParDos. A connector (source) may use a source framework (Splittable DoFn is the recommended framework currently) or may be written using regular ParDos. The main advantages of a source framework are various features provided by such frameworks (progress reporting, dynam

Re: Create IO connector for HTTP or ParDO

Hi, While ago (at the very early stage of Beam :)), I proposed to create a HTTP/REST source/sink (we should still have the Jira :)). However, we didn't have a consensus in terms of features (I proposed something very simple). Splittable DoFn didn't exist at that time. So, if we want to move forwa