Thanks! Looking forward to that. On Tue, Mar 31, 2020 at 1:02 AM Mark Niehe <mark.ni...@segment.com> wrote:
> Hi Gordan and Seth, > > Thanks for explanation and opening up the ticket. I'll add some details in > the ticket to explain what we're trying to do which will hopefully add some > context. > > -- > <http://segment.com/> > Mark Niehe · Software Engineer > Integrations > <https://segment.com/catalog?utm_source=signature&utm_medium=email> · > Blog <https://segment.com/blog?utm_source=signature&utm_medium=email> · > We're > Hiring! <https://segment.com/jobs?utm_source=signature&utm_medium=email> > > On Mon, Mar 30, 2020 at 1:04 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> It seems like Seth's reply didn't make it to the mailing lists somehow. >> Forwarding his reply below: >> >> ---------- Forwarded message --------- >> From: Seth Wiesman <sjwies...@gmail.com> >> Date: Thu, Mar 26, 2020 at 5:16 AM >> Subject: Re: Lack of KeyedBroadcastStateBootstrapFunction >> To: Dawid Wysakowicz <dwysakow...@apache.org> >> Cc: <user@flink.apache.org>, Tzu-Li (Gordon) Tai <tzuli...@apache.org> >> >> >> As Dawid mentioned, you can implement your own operator using the >> transform method to do this yourself. Unfortunately, that is fairly low >> level and would require you to understand some flink amount internals. >> >> The real problem is that the state processor api does not support two >> input operators. We originally skipped that because there were a number of >> open questions about how best to do it and it wasn't clear that it would be >> a necessary feature. Typically, flink users use two input operators to do >> some sort of join. And when bootstrapping state, you typically only want to >> pre-fill one side of that join. KeyedBroadcastState is clearly a good >> counter-argument to that. >> >> I've opened a ticket for the feature if you would like to comment there. >> >> https://issues.apache.org/jira/browse/FLINK-16784 >> >> On Tue, Mar 24, 2020 at 9:17 AM Dawid Wysakowicz <dwysakow...@apache.org> >> wrote: >> >>> Hi, >>> >>> I am not very familiar with the State Processor API, but from a brief >>> look at it, I think you are right. I think the State Processor API does not >>> support mixing different kinds of states in a single operator for now. At >>> least not in a nice way. Probably you could implement the >>> KeyedBroadcastStateBootstrapFunction yourself and us it with >>> KeyedOperatorTransformation#transform(org.apache.flink.state.api.SavepointWriterOperatorFactory). >>> I understand this is probably not the easiest task. >>> >>> I am not aware if there are plans to support that out of the box, but I >>> cc'ed Gordon and Seth who if I remember correctly worked on that API. I >>> hope they might give you some more insights. >>> >>> Best, >>> >>> Dawid >>> On 23/03/2020 17:36, Mark Niehe wrote: >>> >>> Hey all, >>> >>> I have another question about the State Processor API. I can't seem to >>> find a way to create a KeyedBroadcastStateBootstrapFunction operator. The >>> two options currently available to bootstrap a savepoint with state are >>> KeyedStateBootstrapFunction and BroadcastStateBootstrapFunction. Because >>> these are the only two options, it's not possible to bootstrap both keyed >>> and broadcast state for the same operator. Are there any plans to add that >>> functionality or did I miss it entirely when going through the API docs? >>> >>> Thanks, >>> -- >>> <http://segment.com/> >>> Mark Niehe · Software Engineer >>> Integrations >>> <https://segment.com/catalog?utm_source=signature&utm_medium=email> · >>> Blog <https://segment.com/blog?utm_source=signature&utm_medium=email> >>> · We're Hiring! >>> <https://segment.com/jobs?utm_source=signature&utm_medium=email> >>> >>>
