On Fri, May 11, 2018 at 11:46 AM Lukasz Cwik <[email protected]> wrote:
> > On Fri, May 11, 2018 at 11:40 AM Kenneth Knowles <[email protected]> wrote: > >> This is great. "The Beam Vision in a spreadsheet" and/or what the >> capability matrix wishes it always had been. >> >> - I don't know how to interpret the DirectRunner column. Is it that it >> uses ye olde proto round trip? Another level is that it actually directly >> links in the SDK harness as a dep and uses the exact code paths (seems like >> overkill). >> >> > Its up to the direct runner here to decide what level of execution is > actually done via portability APIs but it is meant to be a single process > to ease debugging for users. > Yea so I guess the column is more just "what works?" and not "what works with portability?" in this case. Just a clarification - either way is fine by me. I wasn't sure if the column was to track progress on making the direct runners respect the model or whatnot. Without a proto round trip, a DirectRunner can easily have non-model behaviors by using information that it shouldn't. - For runners*SDK pairs that don't have a batch/streaming distinction how >> about collapsing the columns? >> >> > Runners may not have a distinction but the portability framework may > require more work from a runner to support a use case. A good example of > this is side input readiness checking for streaming pipelines. > What do you mean the portability framework? Do you mean an SDK harness? Or that the protos do not express enough information? Kenn - Anyone have spreadsheet-fu to do a permanent global automatic >> hyperlinking of BEAM-xxxx? >> >> Kenn >> >> On Fri, May 11, 2018 at 10:38 AM Henning Rohde <[email protected]> >> wrote: >> >>> Hi everyone, >>> >>> While the portability framework moves forward, it is often hard to >>> figure out exactly what is supported to work at any given time. There >>> are still many irregularities, TODOs, bugs and small differences between >>> batch and streaming and the portable SDK and runner implementations. >>> For example, the answer to the question "Does Wordcount run portably?" >>> depends on the SDK, Runner and where the output is written. >>> >>> To this end, I've started a spreadsheet to better track the "swiss >>> cheese" of what works portably: >>> >>> >>> https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI/edit?usp=sharing >>> >>> Note that is is a work in progress. The intended audience is for >>> everyone working on or interested in portability. I am hoping we can >>> populate, expand and maintain the information as a community, until the >>> portability framework support is mature enough to allow SDKs and runners to >>> be considered independently. >>> >>> Comments and suggestions welcome! >>> >>> Thanks, >>> Henning >>> >>> >>> >>>
