Hi All So what's the play here?
Galen, what do you think about taking this on? Perhaps ++Till would assign this jira to you (with your permission) given he's helped me out with statefun work before https://issues.apache.org/jira/browse/FLINK-29814 I can try to move to move statefun to flink 1.16 when it's out Kind regards Fil On Thu, 27 Oct 2022 at 10:02, Filip Karnicki <filip.karni...@gmail.com> wrote: > Hi All > > Our use case is that we need to process elements for the same key > sequentially, and this processing involves async operations. > > If any part of the processing fails, we store the offending and all > subsequent incoming messages for that key in the state and not process any > further messages for that key, until a retry succeeds or a human sends a > 'skip' command message. > > diagram: > https://mermaid.live/edit#pako:eNplkL1uwzAMhF-F0JQADrp76FR06tSOcQfWom3V-nFFqoUR591L20mXaqAOxHd3AC-mTZZMbfqM0wAvr00EfS62KbjYn-8C6Jui8DucTo_wmT4Oz97FEVhQqCtxXR13q_IBo-XzXWyehUc3LSu2Uyq2qIXpq2iyQ-9nmCjDSPMCmUISOuwfaEErLsVbw2272VOEDp0vmSqw5HEmC4GYsSeQpKjkv7j_buQ5tjAV4YeehOHHyQDsLAF1HbXCCyQZKB-2CTyzUOCjqUygHNBZPdxljW2MAoEaU6u0mMfGNPGqHBZJb1piasmFKlMmqxd7cqj3Dqbu0DNdfwHTGoek > mermaid (in case mermaid.live goes down in the future): > graph LR > incoming[incoming events] --> job(Flink statefun job) > commands[commands] -->|skip| job > job --> |sequentially per key| remote(remote function) > remote --> |on failure, delayed message to retry| remote > remote --> |async puts/gets with side effects| other(other systems) > > Having the processing happen outside of Flink is nice-to-have from an > independent scalability point of view, but is not strictly required. > > So long story short - no cyclic messaging, but also no way I can think of > to use existing native Flink operators like async i/o (which when I last > checked a few years back didn't have access to keyed state) > > > P.S. Please note that there is already a pull request that has something > to do wtih Flink 1.15, albeit without a description or a jira: > https://github.com/apache/flink-statefun/pull/314 > > > On Wed, 26 Oct 2022 at 19:54, Galen Warren <ga...@cvillewarrens.com> > wrote: > >> Hi Gordon (and others), >> >> I'm also using this project for stateful messaging, including messaging >> among functions. >> >> I've contributed a small amount of code in the past and have also enabled >> Flink 1.15 compatibility in a local fork, so I might be able to help out >> here. >> >> Thanks, >> Galen >> >> On Wed, Oct 26, 2022 at 1:34 PM Ken Krugler <kkrugler_li...@transpac.com> >> wrote: >> >> > Hi Gordon, >> > >> > We’re using it for stateful messaging, and also calling remote >> > Python-based functions. >> > >> > So yes, also very interested in what is going to happen with the this >> > subproject in the future. >> > >> > — Ken >> > >> > >> > >> > > Begin forwarded message: >> > > >> > > From: "Tzu-Li (Gordon) Tai" <tzuli...@apache.org> >> > > Subject: Re: Stateful Functions with Flink 1.15 and onwards >> > > Date: October 26, 2022 at 10:25:26 AM PDT >> > > To: dev@flink.apache.org >> > > Reply-To: dev@flink.apache.org >> > > >> > > Hi Filip, >> > > >> > > Thanks for bringing this up. >> > > >> > > The hard truth is that committers who were previously active on the >> > > StateFun subproject, including myself, all currently have other >> focuses. >> > > Indeed, we may need to discuss with the community on how to proceed if >> > > there seems to be no continued committer coverage. >> > > >> > > If it's just a matter of upgrading the supported Flink version, I'm >> still >> > > familiar enough with the subproject to probably be able to drive this >> (or >> > > if your team is up to it, I can assist you on that). >> > > >> > > For the long-term, as a data point I'm curious to see how many users >> are >> > > using StateFun in production today, and how you're using it? >> > > >> > > - Do your applications have arbitrary / cyclic / bi-directional >> > > messaging between individual functions? >> > > - Or are you utilizing StateFun simply to allow your stateful >> functions >> > > to run remotely as separate processes? >> > > >> > > If the majority is only the latter category, there might be a case to >> > > support remote functions natively in Flink (which has been a >> discussion >> > in >> > > the past). >> > > >> > > Thanks, >> > > Gordon >> > > >> > > On Wed, Oct 26, 2022 at 3:30 AM Filip Karnicki < >> filip.karni...@gmail.com >> > > >> > > wrote: >> > > >> > >> Hi, I noticed that the development on stateful functions has come to >> a >> > bit >> > >> of a halt, with a pull request to update statefun to use Flink 1.15 >> > being >> > >> in the `open` state since May 2022. >> > >> >> > >> What do we think is the future of this sub-project? >> > >> >> > >> The background to this question is that my team is on a shared Flink >> > >> cluster which will soon be upgrading to Flink 1.15. If I need to >> > re-write >> > >> all our code as a native Flink job (rather than a remote stateful >> > function) >> > >> then I need to get started right away. >> > >> >> > >> Many thanks, >> > >> Fil >> > >> >> > >> > -------------------------- >> > Ken Krugler >> > http://www.scaleunlimited.com >> > Custom big data solutions >> > Flink, Pinot, Solr, Elasticsearch >> > >> > >> > >> > >> >
