Hello Flink Folks,

I have been spending some time looking and reading about Flink Stateful
Functions and evaluating its usage for different types of streaming
processing jobs.
I know it is meant for an Event Driven System but I am evaluating it for
other use cases like simple stateful stream processing with drop in
functions. The idea is to allow product teams to bring their business logic
and be able to write data pipelines in less time and require less knowledge
compared to using and deploying something like the DataStream API (I am
aware of the other APIs like Table, Flink SQL and DataStream interop).

Below are my questions. I want to make sure I understand StateFuns and the
core functionality - and potentially some of the future roadmap.

1. KeyBy: From the looks of it, ingress / egress can be scaled, but how
does the scaling of the functions work? If those functions have state, do
they all have access to all state or is there any keying available for a
distributed system (similar to the DataStream API). I have read a large
portion of the code and I don't see anything that could resemble this
feature but maybe it is being added or I missed it?
2. Source vs SourceFunction: Is there any plan in the roadmap of Stateful
Functions to support the source API? I have been writing custom IO modules
for internal sources / sinks and think it would be awesome if (for the most
part) we could drag and drop Sources as embedded modules with simple
wrappers.
3. Does the community know of any companies using StateFuns at scale?

Thanks! If you need any clarifications I would be happy to oblige.

Ryan van Huuksloot
Data Developer | Data Platform Engineering | Streaming Capabilities
[image: Shopify]
<https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email>

Reply via email to