Dan Pettersson created FLINK-16319:
--------------------------------------
Summary: Pubsub-/Broadcast implementation
Key: FLINK-16319
URL: https://issues.apache.org/jira/browse/FLINK-16319
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Dan Pettersson
Hi everyone,
I have a use case where the id of the functions are brokerId + instrumentId
that receives trades and orders. The instrument has state changes (Open,
halted, closed etc) that almost all the functions are interested in. Some
functions only wants for example the Close message whereas other functions
wants all state changes for the specific instrument.
I've built a statefun pubsub module that exposes two interfaces, Subscriber and
Publisher, with these two methods:
default void subscribe(Context context, Subscription... subscriptions)
default void publish(Context context, PublishMessage publishMessage)
Behind the interfaces is a hidden StatefulPubSubFunction that keeps track of
which partition the subscriber is located in and to which topic it listens to.
Code is located under
[https://github.com/danp11/flink-statefun/tree/master/statefun-pubsub] if
anyone is interested.
This code is a "classic pub sub" pattern and I think that this kind of
functionality would be a great addition to Stateful functions. I create this
Jira to see if there is an interest to discuss how a optimal pubsub-/broadcast
solution would look like in SF? Igal has previously mentioned that Broadcast
could be a good fit for this kind of flow.
At the moment I don't know the internals of SF and-/or Flink good enough to
come up with a proposal myself unfortunately.
I know you are very busy at the moment (Its impressive how much you have
produced only the last couple of weeks!:-) but if someone, on a high level, has
any ideas on where and how a pub sub pattern could be implemented I'd really
appreciate it. In the future I hope we can come up with a proposal together as
I need your help here. If you think that a pubsub-/broadcast solution would
make SF better that is :-)
Hope to hear your thoughts on this!
Thanks,
/Dan
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
