Thanks, Roman~
Best,
Xintong
On Thu, Nov 17, 2022 at 10:56 PM Khachatryan Roman <
khachatryan.ro...@gmail.com> wrote:
> I agree, the current calculation logic is already complicated.
> I just think that not using managed memory complicates the memory model
> even further.
>
> But as I mentio
I agree, the current calculation logic is already complicated.
I just think that not using managed memory complicates the memory model
even further.
But as I mentioned earlier, both approaches have their pros and cons, so
I'll update the proposal to use unmanaged memory.
Thanks!
Regards,
Roman
Agreed that the documentation regarding managed memory could be improved.
Just to clarify, breaking isolation is just one of the concerns. I think my
biggest concern is the extra complexity in managed memory calculations.
I've been reached out by many users, online or offline, asking about the
man
> I think not being able to isolate all kinds of memory does not mean we
> should give up the isolation on all kinds of memory. And I believe
"managed
> memory is isolated and others are not" is much easier for the users to
> understand compared to "part of the managed memory is isolated and others
Concerning isolation, I think ideally we want everything to be isolated
between jobs running in the same cluster (i.e., slots in the same TM).
Unfortunately, this is impractical.
- Heap / Off-heap memory are directly allocated / deallocated through JVM /
OS. Flink does not have a good way to cap th
Thanks for your reply Xingong Song,
Could you please elaborate on the following:
> The proposed changes break several good properties that we designed for
> managed memory.
> 1. It's isolated across slots
Just to clarify, any way to manage the memory efficiently while capping its
usage
will break
I like the idea of sharing RocksDB memory across slots. However, I'm quite
concerned by the current proposed approach.
The proposed changes break several good properties that we designed for
managed memory.
1. It's isolated across slots
2. It should never be wasted (unless there's nothing in the j
Hi John, Yun,
Thank you for your feedback
@John
> It seems like operators would either choose isolation for the cluster’s
jobs
> or they would want to share the memory between jobs.
> I’m not sure I see the motivation to reserve only part of the memory for
sharing
> and allowing jobs to choose w
From: John Roesler
Sent: Thursday, November 10, 2022 10:15
To: dev@flink.apache.org
Subject: Re: [DISCUSS] Allow sharing (RocksDB) memory between slots
Hi Roman,
Thanks for the proposal! This will make scheduling a lot more flexible for our
use case.
Just a quick follow-up question about
Hi Roman,
Thanks for the proposal! This will make scheduling a lot more flexible for our
use case.
Just a quick follow-up question about the number of new configs we’re adding
here. It seems like the proposed configs provide a lot of flexibility, but at
the expense of added complexity.
It see
Hi Yanfei,
Thanks, good questions
> 1. Is shared-memory only for the state backend? If both
> "taskmanager.memory.managed.shared-fraction: >0" and
> "state.backend.rocksdb.memory.managed: false" are set at the same time,
> will the shared-memory be wasted?
Yes, shared memory is only for the state
Hi Roman,
Thanks for the proposal, this allows State Backend to make better use of
memory.
After reading the ticket, I'm curious about some points:
1. Is shared-memory only for the state backend? If both
"taskmanager.memory.managed.shared-fraction: >0" and
"state.backend.rocksdb.memory.managed: f
Hi everyone,
I'd like to discuss sharing RocksDB memory across slots as proposed in
FLINK-29928 [1].
Since 1.10 / FLINK-7289 [2], it is possible to:
- share these objects among RocksDB instances of the same slot
- bound the total memory usage by all RocksDB instances of a TM
However, the memory
13 matches
Mail list logo
