I missed adding the mailing list in my previous email.
-- Forwarded message -
From: Matthias Pohl
Date: Tue, Oct 27, 2020 at 12:39 PM
Subject: Re: Flink memory usage monitoring
To: Rajesh Payyappilly Jose
Hi Rajesh,
thanks for reaching out to us. We worked on providing metrics
Classification: Internal
Hi,
Environment - Flink 1.11 on K8s
Is there a way to monitor the usage of managed memory, off-heap memory and
network memory?
-Rajesh
::DISCLAIMER::
The contents of this e-mail and any attachment(s) are confidential and intended
for t
I’ve used the following simple script to capture Flink metrics by running:
python -u ./statsd_server.py 9020 > statsd_server.log
>>> flink-conf.yaml
metrics.reporters: statsd_reporter
metrics.reporter.statsd_reporter.class:
org.apache.flink.metrics.statsd.StatsDReporter
metrics.reporter.
Can you use wget (curl will work as well)? You can find the taskmanagers
with wget -O - http://localhost:8081/taskmanagers
and wget -O - http://localhost:8081/taskmanagers/request> to see detailed jvm
memory stats. localhost:8081 is in my example the jobmanager.
On 04.11.2017 16:19, AndreaKinn
Anyway, If I understood how system metrics works (the results seems to be
showed in browser) I can't use it because my cluster is accessible only with
terminal via ssh
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
I have used sysstat linux tool.
On the node the only one application running is Flink. The outcomes measured
with metric system could be different?
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Hi,
How did you measure the memory usage ?
JVM processes tend to occupy the maximum memory allocated to them,
regardless of whether those memory are actively in used or not. To
correctly measure the memory usage, you should use Flink's metric system[1]
Regards,
Kien
[1]
https://ci.apache.
Hi,
I would like to share some considerations about Flink memory consumption.
I have a cluster composed of three nodes: 1 used both as JM and TM and other
2 TM.
I ran two identical applications (in different moments) on it. The only
difference is that on the second one I doubled every operators, e
ano Bortoli
> *Cc:* Newport, Billy [Tech]; Fabian Hueske; user@flink.apache.org
>
> *Subject:* Re: Flink memory usage
>
>
>
> Hi Billy,
>
>
>
> if you didn't split the different data sets up into different slot sharing
> groups, then your maximum parallelism is
ilto:trohrm...@apache.org]
> Sent: Thursday, April 20, 2017 11:52 AM
> To: Stefano Bortoli
> Cc: Newport, Billy [Tech]; Fabian Hueske; user@flink.apache.org
>
>
> Subject: Re: Flink memory usage
>
>
>
> Hi Billy,
>
>
>
> if you didn't split the different
AM
To: Stefano Bortoli
Cc: Newport, Billy [Tech]; Fabian Hueske; user@flink.apache.org
Subject: Re: Flink memory usage
Hi Billy,
if you didn't split the different data sets up into different slot sharing
groups, then your maximum parallelism is 40. Thus, it should be enough to
assign 40^2
y [mailto:billy.newp...@gs.com]
> *Sent:* Thursday, April 20, 2017 4:46 PM
> *To:* Stefano Bortoli ; 'Fabian Hueske' <
> fhue...@gmail.com>
>
> *Cc:* 'user@flink.apache.org'
> *Subject:* RE: Flink memory usage
>
>
>
> Your reuse idea kind of impli
, Billy [mailto:billy.newp...@gs.com]
Sent: Thursday, April 20, 2017 4:46 PM
To: Stefano Bortoli ; 'Fabian Hueske'
Cc: 'user@flink.apache.org'
Subject: RE: Flink memory usage
Your reuse idea kind of implies that it’s a GC generation rate issue, i.e. it’s
not collecting fast en
, Billy [Tech]; 'Fabian Hueske'
Cc: 'user@flink.apache.org'
Subject: RE: Flink memory usage
Hi Billy,
The only suggestion I can give is to check very well in your code for useless
variable allocations, and foster reuse as much as possible. Don’t create a new
collection at an
link.apache.org'
Subject: RE: Flink memory usage
I don’t think our function are memory heavy they typically are cogroups and
merge the records on the left with the records on the right.
We’re currently requiring 720GB of heap to do our processing which frankly
appears ridiculous to us. Could
We’re running this config now which is not really justifiable for what we’re
doing.
20 nodes 2 slots, 40 parallelism 36GB mem = 720GB of heap…
Thanks
From: Fabian Hueske [mailto:fhue...@gmail.com]
Sent: Wednesday, April 19, 2017 10:52 AM
To: Newport, Billy [Tech]
Cc: user@flink.apache.org
Subje
Hi Billy,
Flink's internal operators are implemented to not allocate heap space
proportional to the size of the input data.
Whenever Flink needs to hold data in memory (e.g., for sorting or building
a hash table) the data is serialized into managed memory. If all memory is
in use, Flink starts spi
How does Flink use memory? We're seeing cases when running a job on larger
datasets where it throws OOM exceptions during the job. We're using the Dataset
API. Shouldn't flink be streaming from disk to disk? We workaround by using
fewer slots but it seems unintuitive that I need to change these
18 matches
Mail list logo
