Hi Robert,
Thanks for your confirmation that Fn API is already ready for supporting
the MultimapUserState use cases, really appreciate it! And totally agree
that how to integrate it depends on the runner's implementation.
A follow-up question:
- is there any runner that already implemented these Multimap protocols?
- I didn't find a runner(e.g. Dataflow Flink, Spark, and Samza)
defined handlers(codes like [1]) for handling Multimap state
requests, so I
think the answer is NO. But I wanted to double confirm with you.
- Having this question just wanted to know: 1) if the existing Fn
APIs for MultimapUserState are already used in production 2) if we can
build/abstract some generic layers(like what Beam did in
StateRequestHandlers now) to benefit other runners
Have a good weekend!
On Fri, Feb 24, 2023 at 1:18 PM Robert Burke <rob...@frantil.com> wrote:
> The runners should be able to support Multimap User State portably over
> the FnApi already.
>
>
> https://github.com/apache/beam/blob/master/model/fn-execution/src/main/proto/org/apache/beam/model/fn_execution/v1/beam_fn_api.proto#L937
>
> How that's supported on each SDK is a different matter though.
>
>
> On Fri, Feb 24, 2023, 12:57 PM Alan Zhang <shuai....@gmail.com> wrote:
>
>> Appreciate it if anyone can help confirm and share thoughts.
>>
>> On Wed, Feb 22, 2023 at 11:46 PM Alan Zhang <shuai....@gmail.com> wrote:
>>
>>> Hi Beam devs.
>>>
>>> According to the Fn State API design doc[1], the state type
>>> MultimapUserState is intended for supporting MapState/SetState. And the
>>> implementation[2] for this state type is ready on the SDK harness side.
>>> Each runner will be responsible for integrating it if they want to leverage
>>> it.
>>>
>>> Today Beam uses StateRequestHandlers to define handler interfaces for
>>> other state types, e.g. MultimapSideInputHandler for
>>> MultimapSideInput, BagUserStateHandler for BagUserState, etc.[3] This is
>>> great since each runner can implement these handler interfaces then the Fn
>>> state API integration is done.
>>>
>>> In order to support MapState/SetState, I think we will need to provide
>>> a MultimapUserStateHandler interface in StateRequestHandlers and allow the
>>> runners to implement it.
>>>
>>> What do you think?
>>>
>>> Feel free to correct me if there is any incorrect understanding since
>>> I'm new to the Beam world.
>>>
>>> Btw, I saw Flink Python used MultimapSideInput to support MapState[4]
>>> but I think this is not recommended since MultimapUserState is available
>>> today. But please correct me if I'm wrong.
>>>
>>>
>>> [1] https://s.apache.org/beam-fn-state-api-and-bundle-processing
>>> <https://s.apache.org/beam-fn-state-api-and-bundle-processing>
>>> [2] https://github.com/apache/beam/pull/15238
>>> [3]
>>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandlers.java#L192
>>> [4]
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-153%3A+Support+state+access+in+Python+DataStream+API
>>> --
>>> Thanks,
>>> Alan
>>>
>>
>>
>> --
>> Thanks,
>> Alan
>>
>
--
Thanks,
Alan
