Is Kafka suitable for high volume, real-time sessions ?

[lrnop]

Hi All,





I wonder if I could get some feedback on a proposed design involving Kafka for 
high volume, real-time sessions ?


See the diagram below for the proposed scheme.


Some additional information:


There are a number of different messages streams (>10), with different 
throughput and latency constraints.


The vast majority of the streams involve request/response pairs. Invalid or 
late responses will cause failure of the session.


Some streams require very high throughput (e.g. > 1k operations per second) and 
low latency, with others the throughput and latency constraints are less strict.


In the diagram just 1 stream type is shown – C.a (== Client type ‘a’).


The message path is:


1.Different instances of C.a processes write to a kafka queue dedicated to that 
particular stream. (C.a.1, C.a.2 etc)


2.This queue is then processed by a routing layer to separate out requests 
based on a particular ‘key’ (e.g. subscriber identity).


3.The routing layer then directs the requests to a further kafka queue 
dedicated for requests for that particular key.


So the traffic is segmented e.g. Segment x referred to in the diagram.


4.This queue is then read by 1 or more consumer that apply the relevant 
service-level behaviour for the request.


5.These consumers place responses to the request onto a dedicated response 
queue for that stream.


6.This queue is then read by the routing layer, and placed in a response queue 
for consumption by the


Client instance that originated the original request (e.g. C.a.1).









Additional information:





a)The client instance producer/consumers (C.a.1 etc) may be connected to 
network components that generate large, unpredictable, bursts of messages.


b)If responses are delayed too long the network components will timeout, in 
some cases leading to large bursts of retries. (Timeouts typically 3..10 
seconds)


c)The vast majority (> 99.99999 %) of the messages queued will not be looked at 
after a fixed number of seconds (~ 10).


d)The queues will not have many different types of consumer (1..3).


e)As a general rule each segment will have about 1k requests per second. 
Second-to-second figures will vary significantly from the average.


f)Obviously any latency in the different interfaces 
(producer->broker,broker->consumer) will have an effect on the request/response 
round-trip times.


g)Some streams will be session-based, with multiple messages per-session. All 
messages are essential to the service.


h) Timeouts in client nodes are service and revenue affecting.





The basic question is – is Kafka a suitable technology for this case?





Thanks in advance!





lrnop