Btw, I'll assume that we're using this thread to gather consensus for code-freezing for 3.3.x series of StateFun. I know there hasn't been much activity on the repo, so this is just a formality really :)
>From the commit history, it looks like we're mainly including the below major changes and bug fixes for 3.3.x: - Flink upgrade to 1.15.2 - https://issues.apache.org/jira/browse/FLINK-26340 - https://issues.apache.org/jira/browse/FLINK-25866 - https://issues.apache.org/jira/browse/FLINK-25936 - https://issues.apache.org/jira/browse/FLINK-25933 I'll wait for 24 hours before cutting the release branch for 3.3.x, unless anyone raises any objections before that. Thanks, Gordon On Tue, Nov 1, 2022 at 10:09 AM Galen Warren <ga...@cvillewarrens.com> wrote: > Thanks Gordon and Filip, I appreciate your help on this one. > > On Tue, Nov 1, 2022 at 1:07 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> PR for upgrading to Flink 1.15.2 has been merged. Thanks for the efforts, >> Galen and Filip! >> >> We should be ready to kick off a new release for StateFun with the Flink >> version upgrade. >> I'll cut off a release branch now on apache/flink-statefun for >> release-3.3.x to move things forward. >> @Galen, @Filip if you want to, after the release branch is cut, you could >> probably upgrade the master branch to Flink 1.16.x as well. >> >> Afterwards we should decide who is available to drive the actual release >> process for 3.3.0. >> There's quite a few steps that would require committer write access. >> Unless someone else is up for this earlier, I'll have some availability >> towards the end of next week to help drive this. >> >> Thanks, >> Gordon >> >> On Mon, Oct 31, 2022 at 12:17 PM Galen Warren <ga...@cvillewarrens.com> >> wrote: >> >> > Yes, that makes sense. >> > >> > PR is here: [FLINK-29814][statefun] Change supported Flink version to >> > 1.15.2 by galenwarren · Pull Request #319 · apache/flink-statefun >> > (github.com) <https://github.com/apache/flink-statefun/pull/319>. >> > >> > On Mon, Oct 31, 2022 at 11:35 AM Till Rohrmann <trohrm...@apache.org> >> > wrote: >> > >> > > I think there might still be value in supporting 1.15 since not >> everyone >> > > upgrades Flink very fast. Hopefully, for Statefun the diff between >> Flink >> > > 1.15 and 1.16 boils down to changing the Flink dependencies. >> > > >> > > Cheers, >> > > Till >> > > >> > > On Mon, Oct 31, 2022 at 2:06 PM Galen Warren <ga...@cvillewarrens.com >> > >> > > wrote: >> > > >> > >> Sure thing. One question -- Flink 1.16 was just released a few days >> ago. >> > >> Should I support 1.15, or just go straight to 1.16? >> > >> >> > >> On Mon, Oct 31, 2022 at 8:49 AM Till Rohrmann <trohrm...@apache.org> >> > >> wrote: >> > >> >> > >>> Hi folks, >> > >>> >> > >>> if you can open a PR for supporting Flink 1.15 Galen, then this >> would >> > be >> > >>> awesome. I've assigned you to this ticket. The next thing after >> merging >> > >>> this PR would be creating a new StateFun release. Once we have >> merged >> > the >> > >>> PR, let's check who can help with it the fastest. >> > >>> >> > >>> Cheers, >> > >>> Till >> > >>> >> > >>> On Mon, Oct 31, 2022 at 1:10 PM Galen Warren < >> ga...@cvillewarrens.com> >> > >>> wrote: >> > >>> >> > >>>> Yes, I could do that. >> > >>>> >> > >>>> On Mon, Oct 31, 2022 at 7:48 AM Filip Karnicki < >> > >>>> filip.karni...@gmail.com> wrote: >> > >>>> >> > >>>>> Hi All >> > >>>>> >> > >>>>> So what's the play here? >> > >>>>> >> > >>>>> Galen, what do you think about taking this on? Perhaps ++Till >> would >> > >>>>> assign this jira to you (with your permission) given he's helped >> me >> > out >> > >>>>> with statefun work before >> > >>>>> https://issues.apache.org/jira/browse/FLINK-29814 >> > >>>>> >> > >>>>> I can try to move to move statefun to flink 1.16 when it's out >> > >>>>> >> > >>>>> >> > >>>>> Kind regards >> > >>>>> Fil >> > >>>>> >> > >>>>> On Thu, 27 Oct 2022 at 10:02, Filip Karnicki < >> > filip.karni...@gmail.com> >> > >>>>> wrote: >> > >>>>> >> > >>>>>> Hi All >> > >>>>>> >> > >>>>>> Our use case is that we need to process elements for the same key >> > >>>>>> sequentially, and this processing involves async operations. >> > >>>>>> >> > >>>>>> If any part of the processing fails, we store the offending and >> all >> > >>>>>> subsequent incoming messages for that key in the state and not >> > process any >> > >>>>>> further messages for that key, until a retry succeeds or a human >> > sends a >> > >>>>>> 'skip' command message. >> > >>>>>> >> > >>>>>> diagram: >> > >>>>>> >> > >> https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek >> > >>>>>> mermaid (in case mermaid.live goes down in the future): >> > >>>>>> graph LR >> > >>>>>> incoming[incoming events] --> job(Flink statefun job) >> > >>>>>> commands[commands] -->|skip| job >> > >>>>>> job --> |sequentially per key| remote(remote function) >> > >>>>>> remote --> |on failure, delayed message to retry| remote >> > >>>>>> remote --> |async puts/gets with side effects| other(other >> > >>>>>> systems) >> > >>>>>> >> > >>>>>> Having the processing happen outside of Flink is nice-to-have >> from >> > an >> > >>>>>> independent scalability point of view, but is not strictly >> required. >> > >>>>>> >> > >>>>>> So long story short - no cyclic messaging, but also no way I can >> > >>>>>> think of to use existing native Flink operators like async i/o >> > (which when >> > >>>>>> I last checked a few years back didn't have access to keyed >> state) >> > >>>>>> >> > >>>>>> >> > >>>>>> P.S. Please note that there is already a pull request that has >> > >>>>>> something to do wtih Flink 1.15, albeit without a description or >> a >> > jira: >> > >>>>>> https://github.com/apache/flink-statefun/pull/314 >> > >>>>>> >> > >>>>>> >> > >>>>>> On Wed, 26 Oct 2022 at 19:54, Galen Warren < >> ga...@cvillewarrens.com >> > > >> > >>>>>> wrote: >> > >>>>>> >> > >>>>>>> Hi Gordon (and others), >> > >>>>>>> >> > >>>>>>> I'm also using this project for stateful messaging, including >> > >>>>>>> messaging >> > >>>>>>> among functions. >> > >>>>>>> >> > >>>>>>> I've contributed a small amount of code in the past and have >> also >> > >>>>>>> enabled >> > >>>>>>> Flink 1.15 compatibility in a local fork, so I might be able to >> > help >> > >>>>>>> out >> > >>>>>>> here. >> > >>>>>>> >> > >>>>>>> Thanks, >> > >>>>>>> Galen >> > >>>>>>> >> > >>>>>>> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler < >> > >>>>>>> kkrugler_li...@transpac.com> >> > >>>>>>> wrote: >> > >>>>>>> >> > >>>>>>> > Hi Gordon, >> > >>>>>>> > >> > >>>>>>> > We’re using it for stateful messaging, and also calling remote >> > >>>>>>> > Python-based functions. >> > >>>>>>> > >> > >>>>>>> > So yes, also very interested in what is going to happen with >> the >> > >>>>>>> this >> > >>>>>>> > subproject in the future. >> > >>>>>>> > >> > >>>>>>> > — Ken >> > >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > > Begin forwarded message: >> > >>>>>>> > > >> > >>>>>>> > > From: "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> >> > >>>>>>> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards >> > >>>>>>> > > Date: October 26, 2022 at 10:25:26 AM PDT >> > >>>>>>> > > To: dev@flink.apache.org >> > >>>>>>> > > Reply-To: dev@flink.apache.org >> > >>>>>>> > > >> > >>>>>>> > > Hi Filip, >> > >>>>>>> > > >> > >>>>>>> > > Thanks for bringing this up. >> > >>>>>>> > > >> > >>>>>>> > > The hard truth is that committers who were previously >> active on >> > >>>>>>> the >> > >>>>>>> > > StateFun subproject, including myself, all currently have >> other >> > >>>>>>> focuses. >> > >>>>>>> > > Indeed, we may need to discuss with the community on how to >> > >>>>>>> proceed if >> > >>>>>>> > > there seems to be no continued committer coverage. >> > >>>>>>> > > >> > >>>>>>> > > If it's just a matter of upgrading the supported Flink >> version, >> > >>>>>>> I'm still >> > >>>>>>> > > familiar enough with the subproject to probably be able to >> > drive >> > >>>>>>> this (or >> > >>>>>>> > > if your team is up to it, I can assist you on that). >> > >>>>>>> > > >> > >>>>>>> > > For the long-term, as a data point I'm curious to see how >> many >> > >>>>>>> users are >> > >>>>>>> > > using StateFun in production today, and how you're using it? >> > >>>>>>> > > >> > >>>>>>> > > - Do your applications have arbitrary / cyclic / >> > bi-directional >> > >>>>>>> > > messaging between individual functions? >> > >>>>>>> > > - Or are you utilizing StateFun simply to allow your >> stateful >> > >>>>>>> functions >> > >>>>>>> > > to run remotely as separate processes? >> > >>>>>>> > > >> > >>>>>>> > > If the majority is only the latter category, there might be >> a >> > >>>>>>> case to >> > >>>>>>> > > support remote functions natively in Flink (which has been a >> > >>>>>>> discussion >> > >>>>>>> > in >> > >>>>>>> > > the past). >> > >>>>>>> > > >> > >>>>>>> > > Thanks, >> > >>>>>>> > > Gordon >> > >>>>>>> > > >> > >>>>>>> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki < >> > >>>>>>> filip.karni...@gmail.com >> > >>>>>>> > > >> > >>>>>>> > > wrote: >> > >>>>>>> > > >> > >>>>>>> > >> Hi, I noticed that the development on stateful functions >> has >> > >>>>>>> come to a >> > >>>>>>> > bit >> > >>>>>>> > >> of a halt, with a pull request to update statefun to use >> Flink >> > >>>>>>> 1.15 >> > >>>>>>> > being >> > >>>>>>> > >> in the `open` state since May 2022. >> > >>>>>>> > >> >> > >>>>>>> > >> What do we think is the future of this sub-project? >> > >>>>>>> > >> >> > >>>>>>> > >> The background to this question is that my team is on a >> shared >> > >>>>>>> Flink >> > >>>>>>> > >> cluster which will soon be upgrading to Flink 1.15. If I >> need >> > to >> > >>>>>>> > re-write >> > >>>>>>> > >> all our code as a native Flink job (rather than a remote >> > >>>>>>> stateful >> > >>>>>>> > function) >> > >>>>>>> > >> then I need to get started right away. >> > >>>>>>> > >> >> > >>>>>>> > >> Many thanks, >> > >>>>>>> > >> Fil >> > >>>>>>> > >> >> > >>>>>>> > >> > >>>>>>> > -------------------------- >> > >>>>>>> > Ken Krugler >> > >>>>>>> > http://www.scaleunlimited.com >> > >>>>>>> > Custom big data solutions >> > >>>>>>> > Flink, Pinot, Solr, Elasticsearch >> > >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > >>>>>>> > >> > >>>>>>> >> > >>>>>> >> > >> >
