Thanks Gordon and Filip, I appreciate your help on this one. On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote:
> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the efforts, > Galen and Filip! > > We should be ready to kick off a new release for StateFun with the Flink > version upgrade. > I'll cut off a release branch now on apache/flink-statefun for > release-3.3.x to move things forward. > @Galen, @Filip if you want to, after the release branch is cut, you could > probably upgrade the master branch to Flink 1.16.x as well. > > Afterwards we should decide who is available to drive the actual release > process for 3.3.0. > There's quite a few steps that would require committer write access. > Unless someone else is up for this earlier, I'll have some availability > towards the end of next week to help drive this. > > Thanks, > Gordon > > On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <ga...@cvillewarrens.com> > wrote: > > > Yes, that makes sense. > > > > PR is here: [FLINK-29814][statefun] Change supported Flink version to > > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun > > (github.com) <https://github.com/apache/flink-statefun/pull/319>. > > > > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <trohrm...@apache.org> > > wrote: > > > > > I think there might still be value in supporting 1.15 since not > everyone > > > upgrades Flink very fast. Hopefully, for Statefun the diff between > Flink > > > 1.15 and 1.16 boils down to changing the Flink dependencies. > > > > > > Cheers, > > > Till > > > > > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <ga...@cvillewarrens.com> > > > wrote: > > > > > >> Sure thing. One question -- Flink 1.16 was just released a few days > ago. > > >> Should I support 1.15, or just go straight to 1.16? > > >> > > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <trohrm...@apache.org> > > >> wrote: > > >> > > >>> Hi folks, > > >>> > > >>> if you can open a PR for supporting Flink 1.15 Galen, then this would > > be > > >>> awesome. I've assigned you to this ticket. The next thing after > merging > > >>> this PR would be creating a new StateFun release. Once we have merged > > the > > >>> PR, let's check who can help with it the fastest. > > >>> > > >>> Cheers, > > >>> Till > > >>> > > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren < > ga...@cvillewarrens.com> > > >>> wrote: > > >>> > > >>>> Yes, I could do that. > > >>>> > > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki < > > >>>> filip.karni...@gmail.com> wrote: > > >>>> > > >>>>> Hi All > > >>>>> > > >>>>> So what's the play here? > > >>>>> > > >>>>> Galen, what do you think about taking this on? Perhaps ++Till would > > >>>>> assign this jira to you (with your permission) given he's helped me > > out > > >>>>> with statefun work before > > >>>>> https://issues.apache.org/jira/browse/FLINK-29814 > > >>>>> > > >>>>> I can try to move to move statefun to flink 1.16 when it's out > > >>>>> > > >>>>> > > >>>>> Kind regards > > >>>>> Fil > > >>>>> > > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki < > > filip.karni...@gmail.com> > > >>>>> wrote: > > >>>>> > > >>>>>> Hi All > > >>>>>> > > >>>>>> Our use case is that we need to process elements for the same key > > >>>>>> sequentially, and this processing involves async operations. > > >>>>>> > > >>>>>> If any part of the processing fails, we store the offending and > all > > >>>>>> subsequent incoming messages for that key in the state and not > > process any > > >>>>>> further messages for that key, until a retry succeeds or a human > > sends a > > >>>>>> 'skip' command message. > > >>>>>> > > >>>>>> diagram: > > >>>>>> > > > https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek > > >>>>>> mermaid (in case mermaid.live goes down in the future): > > >>>>>> graph LR > > >>>>>> incoming[incoming events] --> job(Flink statefun job) > > >>>>>> commands[commands] -->|skip| job > > >>>>>> job --> |sequentially per key| remote(remote function) > > >>>>>> remote --> |on failure, delayed message to retry| remote > > >>>>>> remote --> |async puts/gets with side effects| other(other > > >>>>>> systems) > > >>>>>> > > >>>>>> Having the processing happen outside of Flink is nice-to-have from > > an > > >>>>>> independent scalability point of view, but is not strictly > required. > > >>>>>> > > >>>>>> So long story short - no cyclic messaging, but also no way I can > > >>>>>> think of to use existing native Flink operators like async i/o > > (which when > > >>>>>> I last checked a few years back didn't have access to keyed state) > > >>>>>> > > >>>>>> > > >>>>>> P.S. Please note that there is already a pull request that has > > >>>>>> something to do wtih Flink 1.15, albeit without a description or a > > jira: > > >>>>>> https://github.com/apache/flink-statefun/pull/314 > > >>>>>> > > >>>>>> > > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren < > ga...@cvillewarrens.com > > > > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Hi Gordon (and others), > > >>>>>>> > > >>>>>>> I'm also using this project for stateful messaging, including > > >>>>>>> messaging > > >>>>>>> among functions. > > >>>>>>> > > >>>>>>> I've contributed a small amount of code in the past and have also > > >>>>>>> enabled > > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able to > > help > > >>>>>>> out > > >>>>>>> here. > > >>>>>>> > > >>>>>>> Thanks, > > >>>>>>> Galen > > >>>>>>> > > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler < > > >>>>>>> kkrugler_li...@transpac.com> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>> > Hi Gordon, > > >>>>>>> > > > >>>>>>> > We’re using it for stateful messaging, and also calling remote > > >>>>>>> > Python-based functions. > > >>>>>>> > > > >>>>>>> > So yes, also very interested in what is going to happen with > the > > >>>>>>> this > > >>>>>>> > subproject in the future. > > >>>>>>> > > > >>>>>>> > — Ken > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > Begin forwarded message: > > >>>>>>> > > > > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> > > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards > > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT > > >>>>>>> > > To: dev@flink.apache.org > > >>>>>>> > > Reply-To: dev@flink.apache.org > > >>>>>>> > > > > >>>>>>> > > Hi Filip, > > >>>>>>> > > > > >>>>>>> > > Thanks for bringing this up. > > >>>>>>> > > > > >>>>>>> > > The hard truth is that committers who were previously active > on > > >>>>>>> the > > >>>>>>> > > StateFun subproject, including myself, all currently have > other > > >>>>>>> focuses. > > >>>>>>> > > Indeed, we may need to discuss with the community on how to > > >>>>>>> proceed if > > >>>>>>> > > there seems to be no continued committer coverage. > > >>>>>>> > > > > >>>>>>> > > If it's just a matter of upgrading the supported Flink > version, > > >>>>>>> I'm still > > >>>>>>> > > familiar enough with the subproject to probably be able to > > drive > > >>>>>>> this (or > > >>>>>>> > > if your team is up to it, I can assist you on that). > > >>>>>>> > > > > >>>>>>> > > For the long-term, as a data point I'm curious to see how > many > > >>>>>>> users are > > >>>>>>> > > using StateFun in production today, and how you're using it? > > >>>>>>> > > > > >>>>>>> > > - Do your applications have arbitrary / cyclic / > > bi-directional > > >>>>>>> > > messaging between individual functions? > > >>>>>>> > > - Or are you utilizing StateFun simply to allow your > stateful > > >>>>>>> functions > > >>>>>>> > > to run remotely as separate processes? > > >>>>>>> > > > > >>>>>>> > > If the majority is only the latter category, there might be a > > >>>>>>> case to > > >>>>>>> > > support remote functions natively in Flink (which has been a > > >>>>>>> discussion > > >>>>>>> > in > > >>>>>>> > > the past). > > >>>>>>> > > > > >>>>>>> > > Thanks, > > >>>>>>> > > Gordon > > >>>>>>> > > > > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki < > > >>>>>>> filip.karni...@gmail.com > > >>>>>>> > > > > >>>>>>> > > wrote: > > >>>>>>> > > > > >>>>>>> > >> Hi, I noticed that the development on stateful functions has > > >>>>>>> come to a > > >>>>>>> > bit > > >>>>>>> > >> of a halt, with a pull request to update statefun to use > Flink > > >>>>>>> 1.15 > > >>>>>>> > being > > >>>>>>> > >> in the `open` state since May 2022. > > >>>>>>> > >> > > >>>>>>> > >> What do we think is the future of this sub-project? > > >>>>>>> > >> > > >>>>>>> > >> The background to this question is that my team is on a > shared > > >>>>>>> Flink > > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I > need > > to > > >>>>>>> > re-write > > >>>>>>> > >> all our code as a native Flink job (rather than a remote > > >>>>>>> stateful > > >>>>>>> > function) > > >>>>>>> > >> then I need to get started right away. > > >>>>>>> > >> > > >>>>>>> > >> Many thanks, > > >>>>>>> > >> Fil > > >>>>>>> > >> > > >>>>>>> > > > >>>>>>> > -------------------------- > > >>>>>>> > Ken Krugler > > >>>>>>> > http://www.scaleunlimited.com > > >>>>>>> > Custom big data solutions > > >>>>>>> > Flink, Pinot, Solr, Elasticsearch > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > >>>>>> > > >
