All,
Ahhhg! So sorry - the diagram referenced in the original post (a png) got stripped (Of course). Find below an attempt to reproduce the diagram in plain ascii. The png is available by request if anyone wants to look at it. To summarise the question again: 1. We have a potential case for Kafka involving a mixture of services with variable latency and throughput requirements, over 1k/sec in some cases. 2. A design has been proposed essentially using kafka as a message carrier for session requests and responses. 3. The design involves a routing layer for distribution of requests to segments based on some key in the messages. (And directing responses back to the producer of the original request) 4. Network elements will timeout if there are significant delays in delivering responses to requests - in some cases leading to very large bursts of retry attempts. 5. The messages in the queues will not generally be looked at more than once. So our question is whether Kafka seems a suitable choice for this kind of system? Many thanks, lrnop q1 = Kafka queue for requests, stream C.a q2 = Kafka queue for responses for requests, stream C.a q3 = Kafka queue for requests, stream C.a, segment x. q4 = Kafka queue for responses for requests, stream C.a, segment x C.a.1 = Producer for q1, consumer for q2 C.a.2 = Producer for q1, consumer for q2 C.a.n = Producer for q1, consumer for q2 RL = Routing layer to route service requests for all streams to correct segment. Will consist of 1 or more consumers for q1, producers for q2 SL.C.a.x = Service layer specific for stream C.a, segment x. Will consist of 1 or more consumers for q3, producers for q4 (C.a.1) (C.a.2) (C.a.n) (C.a.1) (C.a.2) (C.a.n) | | | | | | \______|_____/ \______|_____/ | | <q1> <q2> | | (RL) (RL) | | <q3> <q4> | | \___->(SL.C.a.x)->____/ -------- Original Message -------- Subject: Is Kafka suitable for high volume, real-time sessions ? Time (UTC): August 18 2015 3:05 pm From: lr...@protonmail.com To: users@kafka.apache.org CC: Hi All, I wonder if I could get some feedback on a proposed design involving Kafka for high volume, real-time sessions ? See the diagram below for the proposed scheme. Some additional information: There are a number of different messages streams (>10), with different throughput and latency constraints. The vast majority of the streams involve request/response pairs. Invalid or late responses will cause failure of the session. Some streams require very high throughput (e.g. > 1k operations per second) and low latency, with others the throughput and latency constraints are less strict. In the diagram just 1 stream type is shown â C.a (== Client type âaâ). The message path is: 1.Different instances of C.a processes write to a kafka queue dedicated to that particular stream. (C.a.1, C.a.2 etc) 2.This queue is then processed by a routing layer to separate out requests based on a particular âkeyâ (e.g. subscriber identity). 3.The routing layer then directs the requests to a further kafka queue dedicated for requests for that particular key. So the traffic is segmented e.g. Segment x referred to in the diagram. 4.This queue is then read by 1 or more consumer that apply the relevant service-level behaviour for the request. 5.These consumers place responses to the request onto a dedicated response queue for that stream. 6.This queue is then read by the routing layer, and placed in a response queue for consumption by the Client instance that originated the original request (e.g. C.a.1). Additional information: a)The client instance producer/consumers (C.a.1 etc) may be connected to network components that generate large, unpredictable, bursts of messages. b)If responses are delayed too long the network components will timeout, in some cases leading to large bursts of retries. (Timeouts typically 3..10 seconds) c)The vast majority (> 99.99999 %) of the messages queued will not be looked at after a fixed number of seconds (~ 10). d)The queues will not have many different types of consumer (1..3). e)As a general rule each segment will have about 1k requests per second. Second-to-second figures will vary significantly from the average. f)Obviously any latency in the different interfaces (producer->broker,broker->consumer) will have an effect on the request/response round-trip times. g)Some streams will be session-based, with multiple messages per-session. All messages are essential to the service. h) Timeouts in client nodes are service and revenue affecting. The basic question is â is Kafka a suitable technology for this case? Thanks in advance! lrnop
