Hi Rob,
Maybe I am quite late to the party, but I think it might be worth having
a look at the Stateful functions API[1] as well. Especially your latest
approach reminds me a bit about the architecture of the Stateful
functions. There you can have arbitrary routing to functions. You can
also deleg
Hi Rob,
your approach looks good, but I haven't used iterations in streams yet. If
it works for you, it can't be bad anyways.
If it is indeed working as expected, I'd recommend checking out broadcasts
to distribute the rules [1]. This pattern will allow you to dynamically add
new rules via a spec
Thank you for the excellent clarifications.
I couldn't quite figure out how to map the above to my domain.
Nevertheless i have a working demo that performs the following pseudo code:
Let's say that each "channel" has slightly different stream requirements
and we can look up the list of operations
Hi Rob,
1. When you start a flink application, you actually just execute a Java
main called the driver. This driver submits a job graph to the job manager,
which executes the job. Since the driver is an ordinary Java program that
uses the Flink API, you can compose the job graph in any way you wan
Very helpful thank you Arvid.
I've been reading up but I'm not sure I grasp all of that just yet. Please
may I ask for clarification?
1. Could I summarise correctly that I may build a list of functions from an
SQL call which can then be looped over?
This looping sounds appealing and you are righ
Hi Rob,
In the past I used a mixture of configuration and template queries to
achieve a similar goal (I had only up to 150 of these jobs per
application). My approach was not completely dynamic as you have described
but rather to compose a big query from a configuration during the start of
the app
Hi All,
It'd be great to consider stream processing as a platform for our upcoming
projects. Flink seems to be the closeted match.
However we have numerous stream processing workloads and would want to be
able to scale up to 1000's different streams; each quite similar in
structure/sequence but
