Re: Using Stateful Functions within a Flink pipeline

Hi Igal, Thanks for your responses. Regarding "having a pre-processing job that does whatever transformations necessary with the DataStream and outputs to Kafka / Kinesis, and then having a separate StateFun deployment that consumes from that transformed Kafka / Kinesis topic." I was wondering how

Re: Using Stateful Functions within a Flink pipeline

Hi Max, Sorry for not being clearer earlier, by splitting the pipeline I mean: having a pre-processing job that does whatever transformations necessary with the DataStream and outputs to Kafka / Kinesis, and then having a separate StateFun deployment that consumes from that transformed Kafka / Kin

Re: Using Stateful Functions within a Flink pipeline

Dear Igal, Can you elaborate more on your proposed solution of splitting the pipeline? If possible, providing some skeleton pseudocode would be awesome! Thanks in advance. Best, Max -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Using Stateful Functions within a Flink pipeline

@Igal, this sounds more comprehensive (better) than just opening DataStreams: "basically exposing the core Flink job that is the heart of stateful functions. " Great! -- [image: MotaWord] Oytun Tez M O T A W O R D | CTO & Co-Founder oy...@motaword.com

Re: Using Stateful Functions within a Flink pipeline

Hi Annemarie, There are plans to make stateful functions more easily embeddable within a Flink Job, perhaps skipping ingress/egress routing abstracting all together and basically exposing the core Flink job that is the heart of stateful functions. Although these plans are not concrete yet I believe

Re: Using Stateful Functions within a Flink pipeline

Hi Annemarie, Unfortunately this is not possible at the moment, but DataStream as in/egress is in the plans as much as I know. -- [image: MotaWord] Oytun Tez M O T A W O R D | CTO & Co-Founder oy...@motaword.com On Wed, Apr 22, 2020 at 10:26 AM Annemar

Using Stateful Functions within a Flink pipeline

I was wondering if it is possible to use a Stateful Function within a Flink pipeline. I know they work with different API's, so I was wondering if it is possible to have a DataStream as ingress for a Stateful Function. Some context: I'm working on a streaming graph analytics system, and want to sa