>
> Regards,
> Juha
> --
> *From:* Yun Tang
> *Sent:* Tuesday, September 15, 2020 8:06 AM
> *To:* Juha Mynttinen ; Stephan Ewen <
> se...@apache.org>
> *Cc:* user@flink.apache.org
> *Subject:* Re: Performance issue associated with managed RocksDB memory
: Yun Tang ; user@flink.apache.org
Subject: Re: Performance issue associated with managed RocksDB memory
Hey
I've fixed the code
(https://github.com/juha-mynttinen-king/flink/commits/arena_block_sanity_check
[github.com]<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
@flink.apache.org
Subject: Re: Performance issue associated with managed RocksDB memory
Hey
I've fixed the code
(https://github.com/juha-mynttinen-king/flink/commits/arena_block_sanity_check)
slightly. Now it WARNs if there is the memory configuration issue. Also, I
think there was a b
Subject: Re: Performance issue associated with managed RocksDB memory
Hey Juha!
I agree that we cannot reasonably expect from the majority of users to
understand block sizes, area sizes, etc to get their application running.
So the default should be "inform when there is a problem and s
orted in user mailing list, and I
> think it's worth to give some hints in Flink documentations.
>
> When talking about your idea to sanity check the arena size, I think a
> warning should be enough as Flink seems never throw exception directly when
> the performance cou
ms never throw exception directly when the
performance could be poor.
Best
Yun Tang
From: Juha Mynttinen
Sent: Tuesday, September 8, 2020 20:56
To: Yun Tang ; user@flink.apache.org
Subject: Re: Performance issue associated with managed RocksDB memory
Hey Yun,
Thanks for the detaile
Tuesday, September 8, 2020 20:56
> *To:* Yun Tang ; user@flink.apache.org <
> user@flink.apache.org>
> *Subject:* Re: Performance issue associated with managed RocksDB memory
>
> Hey Yun,
>
> Thanks for the detailed answer. It clarified how things work. Especially
> what is the rol
From: Juha Mynttinen
Sent: Tuesday, September 8, 2020 20:56
To: Yun Tang ; user@flink.apache.org
Subject: Re: Performance issue associated with managed RocksDB memory
Hey Yun,
Thanks for the detailed answer. It clarified how things work. Especially what
is the role of RocksDB arena, and arena
ecrease parallelism (if possible), 3) increase managed memory"
Regards,
Juha
From: Yun Tang
Sent: Friday, August 28, 2020 6:58 AM
To: Juha Mynttinen ; user@flink.apache.org
Subject: Re: Performance issue associated with managed RocksDB memory
Hi Juha
Thank
15:56
To: user@flink.apache.org
Subject: Re: Performance issue associated with managed RocksDB memory
The issue can be reproduced by using a certain combinations of the value of
RocksDBOptions.WRITE_BUFFER_RATIO (default 0.5) and the Flink job
parallelism.
Examples that break:
* Parallelism 1 and WRIT
The issue can be reproduced by using a certain combinations of the value of
RocksDBOptions.WRITE_BUFFER_RATIO (default 0.5) and the Flink job
parallelism.
Examples that break:
* Parallelism 1 and WRITE_BUFFER_RATIO 0.1
* Parallelism 5 and the default WRITE_BUFFER_RATIO 0.5
Examples that work:
* P
To clarify, that my questions were all against the very original issue
instead of the WordCount job. The timers come from the window operator you
mentioned as the source of the original issue:
===
bq. If I create a Flink job that has a single "heavy" operator
Hi Juha,
> I can also submit the more complex test with the bigger operator and and a
> window operator. There's just gonna be more code to read. Can I attach a
> file here or how should I submit a larger chuck of code?
You can just attach the file with the code.
> 2. I'm not sure what would / s
Andrey,
A small clarification. The tweaked WordCount I posted earlier doesn't
illustrate the issue I originally explained, i.e. the one where there's a
bigger operator and a smallest possible windows operator. Instead, the
modified WordCount illustrates the degraded performance of a very simple
Fl
PM
To: Andrey Zagrebin
Cc: Juha Mynttinen ; Yun Tang ; user
Subject: Re: Performance issue associated with managed RocksDB memory
Thanks for the ping Andrey.
Hi Juha,
Thanks for reporting the issue. I'd like to check the below things before
further digging into it:
1. Could you let us
Thanks for the ping Andrey.
Hi Juha,
Thanks for reporting the issue. I'd like to check the below things before
further digging into it:
1. Could you let us know your configurations (especially memory related
ones) when running the tests?
2. Did you watch the memory consumption before / after tu
Hi Juha,
Thanks for sharing the testing program to expose the problem.
This indeed looks suboptimal if X does not leave space for the window
operator.
I am adding Yu and Yun who might have a better idea about what could be
improved about sharing the RocksDB memory among operators.
Best,
Andrey
O
Hey,
Here's a simple test. It's basically the WordCount example from Flink, but
using RocksDB as the state backend and having a stateful operator. The
javadocs explain how to use it.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See
18 matches
Mail list logo
