Re: Capacity planning for Kafka Streams

[Mahendra Kariya]

Hi Eno,

Please find my answers inline.


We are in the process of documenting capacity planning for streams, stay
> tuned.
>

This would be great! Looking forward to it.

Could you send some more info on your problem? What Kafka version are you
> using?
>

We are using Kafka 0.10.0.0.


> Are the VMs on the same or different hosts?
>

The VMs are on Google Cloud. Two of them are in asia-east1-a and one is in
asia-east1-c. All three are n1-standard-4 Ubuntu instances.


> Also what exactly do you mean by “the lag keeps fluctuating”, what metric
> are you looking at?
>

We are looking at Kafka Manager for the time being. By fluctuating, I mean
the lag is few thousands at one time, we refresh it the next second, it is
in few lakhs, and again refresh it and it is few thousands. I understand
this may not be very accurate. We will soon have more accurate data once we
start pushing the consumer lag metric to Datadog.

But on a separate note, the difference between lags on different partitions
is way too high. I have attached a tab separated file herewith which shows
the consumer lag (from Kafka Manager) for the first the 50 partitions. As
is clear, the lag on partition 2 is 530 while the lag on partition 18 is
23K. Note that the same VM is pulling data from both the partitions.

Partition	LogSize	Consumer Offset	Lag	Consumer Instance Owner
0	129679486	129678929	557	aggregator_1-1-StreamThread-2-consumer-aaa:/10.x.y.125
1	129729186	129725163	4023	aggregator_1-1-StreamThread-5-consumer-bbb:/10.x.y.123
2	129602164	129601634	530	aggregator_1-1-StreamThread-1-consumer-ccc:/10.x.y.124
3	130425020	130424459	561	aggregator_1-1-StreamThread-8-consumer-ddd:/10.x.y.125
4	131598849	131598265	584	aggregator_1-1-StreamThread-4-consumer-eee:/10.x.y.124
5	130755630	130755081	549	aggregator_1-1-StreamThread-5-consumer-fff:/10.x.y.125
6	128054693	128054104	589	aggregator_1-1-StreamThread-1-consumer-ggg:/10.x.y.125
7	131354755	131354171	584	aggregator_1-1-StreamThread-6-consumer-hhh:/10.x.y.125
8	129797007	129796461	546	aggregator_1-1-StreamThread-10-consumer-iii:/10.x.y.125
9	129087167	129086580	587	aggregator_1-1-StreamThread-5-consumer-jjj:/10.x.y.124
10	128365714	128365102	612	aggregator_1-1-StreamThread-3-consumer-kkk:/10.x.y.125
11	132069213	132068751	462	aggregator_1-1-StreamThread-4-consumer-lll:/10.x.y.123
12	130412684	130412095	589	aggregator_1-1-StreamThread-3-consumer-mmm:/10.x.y.124
13	132199972	132199443	529	aggregator_1-1-StreamThread-2-consumer-nnn:/10.x.y.124
14	129481438	129480844	594	aggregator_1-1-StreamThread-7-consumer-ooo:/10.x.y.124
15	130105939	130105341	598	aggregator_1-1-StreamThread-2-consumer-aaa:/10.x.y.125
16	131973373	131972814	559	aggregator_1-1-StreamThread-8-consumer-ddd:/10.x.y.125
17	130084032	130082875	1157	aggregator_1-1-StreamThread-8-consumer-ppp:/10.x.y.123
18	130220737	130197576	23161	aggregator_1-1-StreamThread-6-consumer-qqq:/10.x.y.124
19	129962307	129961682	625	aggregator_1-1-StreamThread-8-consumer-rrr:/10.x.y.124
20	131692478	131691927	551	aggregator_1-1-StreamThread-1-consumer-ggg:/10.x.y.125
21	129638851	129638323	528	aggregator_1-1-StreamThread-6-consumer-hhh:/10.x.y.125
22	130918801	130918241	560	aggregator_1-1-StreamThread-4-consumer-eee:/10.x.y.124
23	131338127	131337572	555	aggregator_1-1-StreamThread-10-consumer-iii:/10.x.y.125
24	130390257	130389704	553	aggregator_1-1-StreamThread-3-consumer-kkk:/10.x.y.125
25	131383888	131383468	420	aggregator_1-1-StreamThread-9-consumer-sss:/10.x.y.123
26	127540041	127539487	554	aggregator_1-1-StreamThread-3-consumer-mmm:/10.x.y.124
27	130159933	130159385	548	aggregator_1-1-StreamThread-2-consumer-aaa:/10.x.y.125
28	129010992	129010397	595	aggregator_1-1-StreamThread-8-consumer-ddd:/10.x.y.125
29	128511277	128510723	554	aggregator_1-1-StreamThread-7-consumer-ooo:/10.x.y.124
30	131262395	131261954	441	aggregator_1-1-StreamThread-9-consumer-sss:/10.x.y.125
31	129162405	129161881	524	aggregator_1-1-StreamThread-9-consumer-ttt:/10.x.y.124
32	129693104	129692591	513	aggregator_1-1-StreamThread-1-consumer-ggg:/10.x.y.125
33	131763991	131763433	558	aggregator_1-1-StreamThread-10-consumer-iii:/10.x.y.125
34	130954567	130931326	23241	aggregator_1-1-StreamThread-6-consumer-qqq:/10.x.y.124
35	132248088	132247595	493	aggregator_1-1-StreamThread-3-consumer-kkk:/10.x.y.125
36	129893133	129892019	1114	aggregator_1-1-StreamThread-8-consumer-ppp:/10.x.y.123
37	130264671	130264251	420	aggregator_1-1-StreamThread-7-consumer-uuu:/10.x.y.125
38	131077963	131077526	437	aggregator_1-1-StreamThread-3-consumer-vvv:/10.x.y.123
39	132299025	132298434	591	aggregator_1-1-StreamThread-5-consumer-jjj:/10.x.y.124
40	130839786	130839203	583	aggregator_1-1-StreamThread-4-consumer-www:/10.x.y.125
41	130605454	130605019	435	aggregator_1-1-StreamThread-2-consumer-xxx:/10.x.y.123
42	128768310	128767757	553	aggregator_1-1-StreamThread-2-consumer-aaa:/10.x.y.125
43	130863269	130862681	588	aggregator_1-1-StreamThread-8-consumer-ddd:/10.x.y.125
44	128897764	128897205	559	aggregator_1-1-StreamThread-2-consumer-nnn:/10.x.y.124
45	129363920	129363346	574	aggregator_1-1-StreamThread-7-consumer-ooo:/10.x.y.124
46	130108718	130108198	520	aggregator_1-1-StreamThread-5-consumer-fff:/10.x.y.125
47	130231647	130231107	540	aggregator_1-1-StreamThread-10-consumer-yyy:/10.x.y.124
48	130138443	130137894	549	aggregator_1-1-StreamThread-9-consumer-ttt:/10.x.y.124
49	130573085	130549851	23234	aggregator_1-1-StreamThread-6-consumer-qqq:/10.x.y.124