Hi Francesco, No, that architecture is not possible. I'm not sure if you've used Flink's DataStream API but embedded functions under the hood are very much like lightweight process functions. If you have a single DataStream application with two process functions you cannot scale their workers independently because they are sharing the same workers. The solution, in that case, is to separate them into separate jobs which communicate through a message bus such as Kafka.
What you are describing much more aligns with remote function deployments. The functions are deployed as stateless services and can scale independently. You could for instance put functions behind a load balancer or use a FaaS. The statefun runtime and workers now only manage message passing and state storage. Even if one function becomes heavy its compute can be trivially scaled. Note that while there is only a remote python sdk at the moment, the runtime only sees a single http endpoint which could be implemented in any language. In future releases the community hopes to quickly add more language sdks such as Java, NodeJs, Rust, and Golang. Seth On Thu, Jun 11, 2020 at 1:30 AM slinkydeveloper <francescogu...@gmail.com> wrote: > Hi Igal, thanks for your help. > If I understood correctly, the flink deployments (not the functions) needs > to use the same image right? Which means that the flink master and all > workers still needs to use the same image which includes the module.yaml > and > the jar with embedded modules of the full project, right? > I was looking for something different: scale the workers independently, > together with the functions. I'm trying to experiment something here: > https://github.com/slinkydeveloper/playing-with-statefun > <https://github.com/slinkydeveloper/playing-with-statefun> > > In this project I'm trying to deploy separately: > > * ingress > * egress > * "mapper" function > * "greeter" function > > They're all "embedded functions" and I wish to deploy each piece separately > in a separate deployment. > In the next iteration of the experiment I wish to create "remote functions" > deploying them in the same pod of the workers, so the worker talks to the > image using localhost. > > Hence my question: Is it possible to deploy one worker per function/group > of > functions and compose my application of multiple heterogeneous worker > images? If not, does it make sense to do it or it's just no sense for > statefun architecture? > > FG > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
