RE: In-memory cache

2017-10-02 Thread Marchant, Hayden
, October 02, 2017 2:24 PM To: Marchant, Hayden [ICG-IT] Cc: user@flink.apache.org Subject: Re: In-memory cache How about connecting two streams of data, one from the reference data and one from the main data (I assume using key streams as you mention QueryableState) and keep state locally within

Re: In-memory cache

2017-10-02 Thread Stavros Kontopoulos
How about connecting two streams of data, one from the reference data and one from the main data (I assume using key streams as you mention QueryableState) and keep state locally within the operator. The idea is to have a local sub-copy of the reference data within the operator that is updated from

In-memory cache

2017-10-02 Thread Marchant, Hayden
We have an operator in our streaming application that needs to access 'reference data' that is updated by another Flink streaming application. This reference data has about ~10,000 entries and has a small footprint. This reference data needs to be updated ~ every 100 ms. The required latency for