Re: Handling HTTP Requests for Keyed Streams

2021-08-17 Thread JING ZHANG
Hi Rion, Your solution is good. It seems that you need enrich a stream with data queries from external Http request. There is another solution for reference, just like the mechanism of lookup join in Flink SQL. Lookup Join in Flink SQL supports two modes: Async mode and Sync mode. For each input d

Re: Handling HTTP Requests for Keyed Streams

2021-08-17 Thread Rion Williams
Hi Caizhi, I don’t mind the request being synchronous (or not using the Async I/O connectors). Assuming I go down that route would this be the appropriate way to handle this? Specifically creating an HttpClient and storing the result in state and on a keyed stream if the state was empty? It ma

Re: Handling HTTP Requests for Keyed Streams

2021-08-16 Thread Caizhi Weng
Hi! As you mentioned that the configuration fetching is very infrequent, why don't you use a blocking approach to send HTTP requests and receive responses? This seems like a more reasonable solution to me. Rion Williams 于2021年8月17日周二 上午4:00写道: > Hi all, > > I've been exploring a few different o

Handling HTTP Requests for Keyed Streams

2021-08-16 Thread Rion Williams
Hi all, I've been exploring a few different options for storing tenant-specific configurations within Flink state based on the messages I have flowing through my job. Initially I had considered creating a source that would periodically poll an HTTP API and connect that stream to my original event