Turns out much of the work was completed to populate and consume the urn + payloads.
I have deprecated the single "url" field in enviornment with https://github.com/apache/beam/pull/8213 which will allow us to close of BEAM-5433. On Mon, Apr 1, 2019 at 1:48 PM Lukasz Cwik <lc...@google.com> wrote: > Yes, need to use the new fields everywhere and then deprecate the old > fields. > > On Mon, Apr 1, 2019 at 1:33 PM Kenneth Knowles <k...@apache.org> wrote: > >> >> >> On Mon, Apr 1, 2019 at 8:59 AM Lukasz Cwik <lc...@google.com> wrote: >> >>> To clarify, docker isn't the only environment type we are using. We have >>> a process based and "existing" environment mode that don't fit the current >>> protobuf and is being worked around. >>> >> >> Ah, understood. >> >> >>> The idea would be to move to a URN + payload model like our PTransforms >>> and coders with a docker specific one. Using the URN + payload would allow >>> us to have a versioned way to update the environment specifications and >>> deprecate/remove things that are ill defined. >>> >> >> Makes sense to me. It looks like this migration path is already in place >> in `message Environment` in beam_runner_api.proto, with `message >> StandardEnvironments` enumerating some URNs and corresponding payload >> messages just below. So is the gap just getting the two portable runners to >> look at the new fields? >> >> Kenn >> >> >>> On Fri, Mar 29, 2019 at 6:41 PM Kenneth Knowles <k...@apache.org> wrote: >>> >>>> >>>> >>>> On Thu, Mar 28, 2019 at 9:30 AM Lukasz Cwik <lc...@google.com> wrote: >>>> >>>>> The intention is that these kinds of hints such as CPU and/or memory >>>>> should be embedded in the environment specification that is associated >>>>> with >>>>> the transforms that need resource hints. >>>>> >>>>> The environment spec is woefully ill prepared as it only has a docker >>>>> URL right now. >>>>> >>>> >>>> FWIW I think this is actually "extremely well prepared" :-) >>>> >>>> Protobuf is great for adding fields when you need more but removing is >>>> nearly impossible once deployed, so it is best to do the absolute minimum >>>> until you need to expand. >>>> >>>> Kenn >>>> >>>> >>>>> >>>>> On Thu, Mar 28, 2019 at 8:45 AM Robert Burke <rob...@frantil.com> >>>>> wrote: >>>>> >>>>>> A question came over the beam-go slack that I wasn't able to answer, >>>>>> in particular for Dataflow*, is there a way to increase how much of a >>>>>> Portable FnAPI worker is dedicated for the SDK side, vs the Runner side? >>>>>> >>>>>> My assumption is that runners should manage it, and have the Runner >>>>>> Harness side be as lightweight as possible, to operate under reasonable >>>>>> memory bounds, allowing the user-code more room to spread, since it's >>>>>> largely unknown. >>>>>> >>>>>> I saw there's the Provisioning API >>>>>> <https://github.com/apache/beam/blob/master/model/fn-execution/src/main/proto/beam_provision_api.proto#L52> >>>>>> which to communicates resource limits to the SDK side, but is there a way >>>>>> to make the request (probably on job start up) in the other direction? >>>>>> >>>>>> I imagine it has to do with the container boot code, but I have only >>>>>> vague knowledge of how that works at present. >>>>>> >>>>>> If there's a portable way for it, that's ideal, but I suspect this >>>>>> will be require a Dataflow specific answer. >>>>>> >>>>>> Thanks! >>>>>> Robert B >>>>>> >>>>>> *Dataflow doesn't support the Go SDK, but the Go SDK supports >>>>>> Dataflow. >>>>>> >>>>>
