I just had to drop in and say that this may be a good fit for basic machine learning in the form of simple decision trees <https://en.wikipedia.org/wiki/Decision_tree>?
You could train simple decision tree classifier using examples of your messages and their associated output message types. Then you would create and deploy a worker that predicts the message type to send based on the fields in the messages it reads. It would then send that message type. Something like scikit-learn's DecisionTreeClassifier <https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier> may work, or there is sure to be a decision tree library in the JVM world that would work with Streams. Just a thought, Russell Jurney @rjurney <http://twitter.com/rjurney> russell.jur...@gmail.com LI <http://linkedin.com/in/russelljurney> FB <http://facebook.com/jurney> datasyndrome.com On Mon, Mar 11, 2019 at 11:41 AM Daniel Hinojosa < dhinoj...@evolutionnext.com> wrote: > I would go with Kafka Streams. Kafka Streams is just a library. So you can > make a simple or complex application. You can include a rules engine or > machine learning model as a dependency as well as a kafka dependency and > based on your message send to being different topics. Run a number of your > application equal to your partitions on different machines and let the > magic happen. > > On Mon, Mar 11, 2019, 10:28 AM Greenhorn Techie <greenhorntec...@gmail.com > > > wrote: > > > Hi, > > > > Our Kafka message has 10-15 fields. On the consumer side, we need to take > > some action based on a combination of fields. For example, if Field A = X > > and Field B = Y, then push the message to Topic T1. However, if Field C > = P > > or Field D = Q, then push the message into Topic T2. Likewise, we have > many > > such rules depending on which certain action has to be taken. > > > > Wondering whether there is standard pattern to implement such business > > rules logic in any of the Kafka ecosystem components? For Kafka Streams? > > Even otherwise, would be glad to understand how companies implement such > > decision trees in practice on streaming data. > > > > Thanks > > >
