Hello all, I've addressed the areas with comments with updated explanations and responses where necessary.
Please do have a quick read if you have time. I shall follow-up with these datas as markdown changes to beam site in a couple of days for feedback. Thanks! Herman Mak | Customer Engineer, Hong Kong, Google Cloud | herman...@google.com | +852-3923-5417 On Sat, Dec 17, 2022 at 2:13 AM Andrew Pilloud <apill...@google.com> wrote: > By "Relational" I mean things like: Column Pruning, Filter Pushdown, Table > Statistics, Partition Metadata, Metastore. We have a bunch of one-off > implementations in various IOs (mostly BigQueryIO) and have been waiting > for IO standards to push them out to all IOs. This was section "F5 - > Relational" from https://s.apache.org/beam-io-api-standard-documentation > > On Thu, Dec 15, 2022 at 6:50 PM Herman Mak <herman...@google.com> wrote: > >> Hey all, >> >> Firstly apologies for the confusion. >> >> The scope of this effort is to *finalize and have this added to the Beam >> public documentation* to be used as a PR reference once we have resolved >> the comments. >> YES this document is a continuation of the below docs with some >> additional components such as testing! >> >> The idea is to convert this to a MD file and add a page under "Developing >> new I/O connectors" with some small cleanup work around this area in other >> pages. >> [image: image.png] >> >> >> >> >> Docs that this is a continuation of: >> https://s.apache.org/beam-io-api-standard-documentation >> https://s.apache.org/beam-io-api-standard >> >> >> @Andrew Pilloud <apill...@google.com> Totally not intending to start >> from the beginning here, by relational do you mean having this hosting in >> the Beam confluence? >> >> Thanks all, and keep the feedback to the docs coming >> >> Herman Mak | Customer Engineer, Hong Kong, Google Cloud | >> herman...@google.com | +852-3923-5417 <+852%203923%205417> >> >> >> >> >> >> On Fri, Dec 16, 2022 at 1:36 AM Chamikara Jayalath <chamik...@google.com> >> wrote: >> >>> >>> >>> On Thu, Dec 15, 2022, 8:33 AM Alexey Romanenko <aromanenko....@gmail.com> >>> wrote: >>> >>>> Cham, do you remember what was a reason to not finalise that doc? >>>> >>> >>> I think this is a continuation of those docs (so we are trying to >>> finalize) but probably Herman can explain better. >>> >>> >>>> Personally, I find having such standards very useful (if they are >>>> flexible during a time, of course), especially for new developers and PR >>>> reviewers, and it’d be great to finally have such doc as a part of >>>> contribution guide. >>>> >>> >>> +1 >>> >>> Thanks, >>> Cham >>> >>>> >>>> — >>>> Alexey >>>> >>>> On 13 Dec 2022, at 04:32, Chamikara Jayalath via dev < >>>> dev@beam.apache.org> wrote: >>>> >>>> Yeah, I don't think either finalized or documented (in the Website) the >>>> previous iteration. This doc seems to contain details from the documents >>>> shared in the previous iteration. >>>> >>>> Thanks, >>>> Cham >>>> >>>> >>>> >>>> On Mon, Dec 12, 2022 at 6:49 PM Robert Burke <rob...@frantil.com> >>>> wrote: >>>> >>>>> I think ultimately: until the docs a clearly available on the Beam >>>>> site itself, it's not documentation. See also, design docs, previous >>>>> emails, and similar. >>>>> >>>>> On Mon, Dec 12, 2022, 6:07 PM Andrew Pilloud via dev < >>>>> dev@beam.apache.org> wrote: >>>>> >>>>>> I believe the previous iteration was here: >>>>>> https://lists.apache.org/thread/3o8glwkn70kqjrf6wm4dyf8bt27s52hk >>>>>> >>>>>> The associated docs are: >>>>>> https://s.apache.org/beam-io-api-standard-documentation >>>>>> https://s.apache.org/beam-io-api-standard >>>>>> >>>>>> This is missing all the relational stuff that was in those docs, this >>>>>> appears to be another attempt starting from the beginning? >>>>>> >>>>>> Andrew >>>>>> >>>>>> >>>>>> On Mon, Dec 12, 2022 at 9:57 AM Alexey Romanenko < >>>>>> aromanenko....@gmail.com> wrote: >>>>>> >>>>>>> Thanks for writing this! >>>>>>> >>>>>>> IIRC, the similar design doc was sent for review here a while ago. >>>>>>> Is this just an updated version and a new one? >>>>>>> >>>>>>> — >>>>>>> Alexey >>>>>>> >>>>>>> On 11 Dec 2022, at 15:16, Herman Mak via dev <dev@beam.apache.org> >>>>>>> wrote: >>>>>>> >>>>>>> Hello Everyone, >>>>>>> >>>>>>> *TLDR* >>>>>>> >>>>>>> Should we adopt a set of standards that Connector I/Os should adhere >>>>>>> to? >>>>>>> Attached is a first version of a Beam I/O Standards guideline that >>>>>>> includes opinionated best practices across important components of a >>>>>>> Connector I/O, namely Documentation, Development and Testing. >>>>>>> >>>>>>> *The Long Version* >>>>>>> >>>>>>> Apache Beam is a unified open-source programming model for both >>>>>>> batch and streaming. It runs on multiple platform runners and integrates >>>>>>> with over 50 services using individually developed I/O Connectors >>>>>>> <https://beam.apache.org/documentation/io/connectors/>. >>>>>>> >>>>>>> Given that Apache Beam connectors are written by many different >>>>>>> developers and at varying points in time, they vary in syntax style, >>>>>>> documentation completeness and testing done. For a new adopter of Apache >>>>>>> Beam, that can definitely cause some uncertainty. >>>>>>> >>>>>>> So should we adopt a set of standards that Connector I/Os should >>>>>>> adhere to? >>>>>>> Attached is a first version, in Doc format, of a Beam I/O Standards >>>>>>> guideline that includes opinionated best practices across important >>>>>>> components of a Connector I/O, namely Documentation, Development and >>>>>>> Testing. And the aim is to incorporate this into the documentation and >>>>>>> to >>>>>>> have it referenced as standards for new Connector I/Os (and ideally have >>>>>>> existing Connectors upgraded over time). If it looks helpful, the >>>>>>> immediate >>>>>>> next step is that we can convert it into a .md as a PR into the Beam >>>>>>> repo! >>>>>>> >>>>>>> Thanks and looking forward to feedbacks and discussion, >>>>>>> >>>>>>> [PUBLIC] Beam I/O Standards >>>>>>> <https://docs.google.com/document/d/1BCTpSZDUjK90hYZjcn8aAnPd9vuRfj8YU1j3mpSgRwI/edit?usp=drive_web> >>>>>>> >>>>>>> Herman Mak | Customer Engineer, Hong Kong, Google Cloud | >>>>>>> herman...@google.com | +852-3923-5417 <+852%203923%205417> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>
