Nice idea. Actually we are looking at connect for other parts of our solution 
in which the latency is less critical.

A few considerations of not using ‘connect’ in this case were:


1.       To isolate the two streams from each other to reduce complexity, 
simplify debugging etc…. – since we are newbies at Flink I was thinking that it 
is beneficial to keep the stream as simple as possible, and if need be, we can 
interface between them to ‘exchange data’

2.       The reference data, even though quite small, is updated every 100ms. 
Since we would need this reference data on each ‘consuming’ operator instance, 
we would be essentially nearly double the amount of tuples coming through the 
operator. Since low-latency is  key here, this was a concern, the assumption 
being that the two sides of the ‘connect’ share the same resources – whereas 
using a background thread to update a ‘map’ would not be competing with the 
incoming tuples)

I realize that structurally, connect is a neater solution.

If I can be convinced that my above concerns are unfounded, I’ll be happy to 
try that direction.

Thanks
Hayden

From: Stavros Kontopoulos [mailto:st.kontopou...@gmail.com]
Sent: Monday, October 02, 2017 2:24 PM
To: Marchant, Hayden [ICG-IT]
Cc: user@flink.apache.org
Subject: Re: In-memory cache

How about connecting two streams of data, one from the reference data and one 
from the main data (I assume using key streams as you mention QueryableState) 
and keep state locally within the operator.
The idea is to have a local sub-copy of the reference data within the operator 
that is updated from the source of the reference data. Reference data are still 
updated
externally from that low latency flink app. Here is a relevant question: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Accessing-state-in-connected-streams-td8727.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dflink-2Duser-2Dmailing-2Dlist-2Darchive.2336050.n4.nabble.com_Accessing-2Dstate-2Din-2Dconnected-2Dstreams-2Dtd8727.html&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=g-5xYRH8L3aCnCNTROw5LrsB5gbTayWjXSm6Nil9x0c&m=eLl5xx2Dc8nmmad2mz2k0aQ53NeI_Fb2V-qeRn-7CVQ&s=05n6tNvhZgLQ4o_N9tpkh8jhM1RcyCB_MIVcZILECtI&e=>.
 Would that help?

Stavros



On Mon, Oct 2, 2017 at 1:46 PM, Marchant, Hayden 
<hayden.march...@citi.com<mailto:hayden.march...@citi.com>> wrote:
We have an operator in our streaming application that needs to access 
'reference data' that is updated by another Flink streaming application. This 
reference data has about ~10,000 entries and has a small footprint. This 
reference data needs to be updated ~ every 100 ms. The required latency for  
this application is extremely low ( a couple of milliseconds), and we are 
therefore cautious of paying cost of I/O to access the reference data remotely. 
We are currently examining 3 different options for accessing this reference 
data:

1. Expose the reference data as QueryableState and access it directly from the 
'client' streaming operator using the QueryableState API
2. same as #1, but create an In-memory Java cache of the reference data within 
the operator that is asynchronously updated at a scheduled frequency using the 
QueryableState API
3. Output the reference data to Redis, and create an in-memory java cache of 
the reference data within the operator that is asynchronously updated at a 
scheduled frequency using Redis API.

My understanding is that one of the cons of using Queryable state, is that if 
the Flink application that generates the reference data is unavailable, the 
Queryable state will not exist - is that correct?

If we were to use an asynchronously scheduled 'read' from the distributed 
cache, where should it be done? I was thinking of using 
ScheduledExecutorService from within the open method of the Flink operator.

What is the best way to get this done?

Regards,
Hayden Marchant

Reply via email to