If the SQL data is all (or mostly all) needed to join against the data from Kafka, then I might try a regular join.
Otherwise it sounds like you want to use an AsyncFunction to do ad hoc queries (in parallel) against your SQL DB. https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/asyncio.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/asyncio.html> — Ken > On Apr 15, 2018, at 12:15 PM, miki haiat <miko5...@gmail.com> wrote: > > Hi, > > I have a case of meta data enrichment and im wondering if my approach is the > correct way . > input stream from kafka. > MD in msSQL . > map to new pojo > I need to extract a key from the kafka stream and use it to select some > values from the sql table . > > SO i thought to use the table SQL api in order to select the table MD > then convert the kafka stream to table and join the data by the stream key . > > At the end i need to map the joined data to a new POJO and send it to > elesticserch . > > Any suggestions or different ways to solve this use case ? > > thanks, > Miki > > > -------------------------- Ken Krugler http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr
