Hi Roman,
Thank you for the reply.
Yes, I am aware that backpressure can be the result of many factors and
yes this is an oversimplification of something very complex, please bare
with me. Lets assume that this has been taken into account and has
lowered the threshold for when this status permanently comes into
effect, i.e. HIGH.
Example: The system is running along perfectly fine under normal
conditions, accessing external sources, and processing at an average of
100,000 messages/sec. Lets assume the maximum capacity is around 130,000
message/sec before back pressure starts propagating messages back up the
stream. Therefore, utilization is at 0.76 (100K/130K). Great, but at
present we dont know that 130,000 is the limit.
For this example or for any job, is there a way of finding this maximum
capacity (and hence the utilization) without pushing the system to its
limit based on the current throughput? Possibly by measuring (as you
say) the saturation of certain buffers (looking into this now, however,
i am not too familiar with flink internals)? It doesn't have to be
extremely precise. Any hints would be greatly appreciated.
Regards,
M.
On 25.02.20 13:34, Khachatryan Roman wrote:
Hi Morgan,
Regarding backpressure, it can be caused by a number of factors, e.g.
writing to an external system or slow input partitions.
However, if you know that a particular resource is a bottleneck then
it makes sense to monitor its saturation.
It can be done by using Flink metrics. Please see the documentation
for more details:
https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html
Regards,
Roman
On Tue, Feb 25, 2020 at 12:33 PM Morgan Geldenhuys
<morgan.geldenh...@tu-berlin.de
<mailto:morgan.geldenh...@tu-berlin.de>> wrote:
Hello community,
I am fairly new to Flink and have a question concerning
utilization. I
was hoping someone could help.
Knowing that backpressure is essentially the point at which
utilization
has reached 100% for any particular streaming pipeline and means that
the application cannot "keep up" with the messages coming into the
system.
I was wondering, assuming a fairly stable input throughput, is
there a
way of determining the average utilization as a percentage? Can we
determine how much more capacity each operator has before
backpressure
kicks in from metrics alone, i.e. 60% of capacity for example?
Knowing
that the maximum throughput of the DSP application is dictated by the
slowest part of the pipeline, we would need to identify the slowest
operator and then average horizontally.
The only method that I can see of determining the point at which the
system cannot keep up any longer is by scaling the input throughput
slowly until the backpressure HIGH alarm is shown and hence the
number
of messages/sec is known.
Yes I know this is a gross oversimplification and there are many many
factors that need to be taken into account when dealing with
backpressure, but it would be nice to have a general indicator, a
rough
estimate is fine.
Thank you in advance.
Regards,
M.
