Hi Barry, I've forward your email to the user mailing list as it is more suitable here :-)
Your question definitely makes sense, and let me try to provide you with some pointers: 1. The architecture that you've outlined has many advantages and is desirable if you can afford that. Some of them are - clean separation of concerns - better resource isolation. - different SLO and fault domains (failure/slowness in your Python function, doesn't trigger a failure/back-pressure in your ETL) - you can use event time watermarks for your ETL (statefun only works with processing time) 2. If you would still prefer to merge the two then you can checkout the DataStream integration API [1] Although it has some rough edges in respect with working with remote functions in particular. Good luck, Igal. [1] https://nightlies.apache.org/flink/flink-statefun-docs-release-3.1/docs/sdk/flink-datastream/ On Thu, Sep 2, 2021 at 1:07 PM Barry Higgins <barry.p.higgi...@gmail.com> wrote: > Hi, > > I have set up a remote stateful function in python which I’ve deployed > on an AWS EC2 box. I am interacting with this from a separate statefun > docker container running 2 flink-statefun images with roles master and > worker (on a separate EC2 instance). The ingress and egress points for > this function are Kafka. > > I then have a separate Java application using Flink, deployed on a > Ververica cluster. From this application I am communicating with the > statefun function by adding a sink/source pointing at the > ingress/egress above. > > I have a couple of questions on this setup. > > I am unsure if there is a better way to communicate with the function > from the Flink application > I am wondering if there is anyway that I can use the existing deployed > application to maintain the state of my remote function, meaning that > I can discard the statefun master/worker elements? > Failing that, do I just need to create a new Flink application, > translate the equivalent of the module.yml that is passed to the > existing master/worker to Java, add the dependencies and deploy that > jar? > > I hope that makes sense? > Kindest Regards, > > Barry >
