I think this is expected behaviour. You will have to tune up IOPS
accordingly.
When you restart your brokers it tries to read metadata for all available
topics and partitions (including all files) and that is where your read
IOPS will shoot up. And if your cluster is busy one then it will try to
Hi Soumyajit,
It is possible that due to the broker restart, you benefit from less
I/O merges than under steady state. Intuitively, that would come from
a shift from sequential workload with one more dispersed in nature. It
is likely your broker generates more disk read than before the
restart, es
We are using mainly ephemeral instances like i3en as our pattern is more
fit for it.
On Tue, Apr 7, 2020 at 10:40 AM Soumyajit Sahu
wrote:
> @Suman, thanks for confirming. I will dig more then. The instances are
> dedicated to running Kafka, and so is the mounted volume.
>
> @Seva, thanks for th
@Suman, thanks for confirming. I will dig more then. The instances are
dedicated to running Kafka, and so is the mounted volume.
@Seva, thanks for the insight. I guess if nothing works, then we will move
from st1 to gp2 volumes.
On Tue, Apr 7, 2020 at 12:28 AM Suman B N wrote:
> We have used st
We have used st1 volumes and we never saw any issue.
Yes, we are using m-series. Even t-series worked for us :D
During those spikes, do you observe any background operations going on?
Check server logs, controller logs.
On Tue, Apr 7, 2020 at 12:49 PM Seva Feldman wrote:
> ST1 EBS fit only for
ST1 EBS fit only for sequential rights and reads. Once you have many
partitions on EBS it will be mostly random.
Interesting to monitor random vs sequential...
We tested kafka on ST1 with 1xx partitions on each EBS and it was
constantly lagging.
BR
On Tue, Apr 7, 2020 at 10:06 AM Soumyajit Sahu
Our typical IOPS stays at ~10K write ops/min, but it goes to 37K write
ops/min (which is where AWS throttles).
The spike in write ops isn't accompanied by any spike in write throughput
or produce requests (except for the first few minutes of catch up). The
write ops spike stays up (persistently for
We too have a similar setup but we never observed any such spikes.
Are you sure your disk IOPS is good enough? Check if that is throttling.
After a broker restarts, there might be more traffic as well because of
followers trying to catch up with the leader.
-Suman
On Tue, Apr 7, 2020 at 11:59 A
Are you using log compaction?
On Tue, Apr 7, 2020 at 6:29 PM Soumyajit Sahu
wrote:
> We are running Kafka on AWS EC2 instances (m5.2xlarge) with mounted EBS st1
> volume (one on each machine).
> Occasionally, we have noticed that the write ops/second goes through the
> roof and we get throttled
We are running Kafka on AWS EC2 instances (m5.2xlarge) with mounted EBS st1
volume (one on each machine).
Occasionally, we have noticed that the write ops/second goes through the
roof and we get throttled by AWS while the data throughput wouldn't have
changed much. As far as our observation goes, i
10 matches
Mail list logo
