PYTHON = 100, which describes the
> situation when one slot has both consumers of managed python and state
> backend.
>
> Best
> Yun Tang
>
> From: Jark Wu
> Sent: Wednesday, January 6, 2021 13:51
> To: dev
> Subject: Re: [DISCUSS] Allow
operators to use managed memory
Thanks all for the discussion.
I have created an issue FLINK-20860 [1] to support this.
In conclusion, we will extend the configuration
`taskmanager.memory.managed.consumer-weights` to have 2 more consumer
kinds: OPERATOR and STATE_BACKEND, the available consumer
Thanks all for the discussion.
I have created an issue FLINK-20860 [1] to support this.
In conclusion, we will extend the configuration
`taskmanager.memory.managed.consumer-weights` to have 2 more consumer
kinds: OPERATOR and STATE_BACKEND, the available consumer kinds will be :
* `OPERATOR` for
>
> Would the default weight for OPERATOR and STATE_BACKEND be the same value?
>
I would say yes, to align with previous behaviors.
Thank you~
Xintong Song
On Tue, Jan 5, 2021 at 5:51 PM Till Rohrmann wrote:
> +1 for Jark's and Xintong's proposal.
>
> Would the default weight for OPERATOR
+1 for Jark's and Xintong's proposal.
Would the default weight for OPERATOR and STATE_BACKEND be the same value?
Cheers,
Till
On Tue, Jan 5, 2021 at 6:39 AM Jingsong Li wrote:
> +1 for allowing streaming operators to use managed memory.
>
> The memory use of streams requires some hierarchy, an
+1 for allowing streaming operators to use managed memory.
The memory use of streams requires some hierarchy, and the bottom layer is
undoubtedly the current StateBackend.
Let the stream operators freely use the managed memory, which will make the
memory management model to be unified and give the
+1 to Xingtong's proposal!
Best,
Jark
On Tue, 5 Jan 2021 at 12:13, Xintong Song wrote:
> +1 for allowing streaming operators to use managed memory.
>
> As for the consumer names, I'm afraid using `DATAPROC` for both streaming
> ops and state backends will not work. Currently, RocksDB state back
+1 for allowing streaming operators to use managed memory.
As for the consumer names, I'm afraid using `DATAPROC` for both streaming
ops and state backends will not work. Currently, RocksDB state backend uses
a shared piece of memory for all the states within that slot. It's not the
operator's dec
Hi Aljoscha,
I think we may need to divide `DATAPROC` into `OPERATOR` and
`STATE_BACKEND`, because they have different scope (slot vs. operator).
But @Xintong Song may have more insights on it.
Best,
Jark
On Mon, 4 Jan 2021 at 20:44, Aljoscha Krettek wrote:
> I agree, we should allow streami
I agree, we should allow streaming operators to use managed memory for
other use cases.
Do you think we need an additional "consumer" setting or that they would
just use `DATAPROC` and decide by themselves what to use the memory for?
Best,
Aljoscha
On 2020/12/22 17:14, Jark Wu wrote:
Hi all
Hi all,
I found that currently the managed memory can only be used in 3 workloads
[1]:
- state backends for streaming jobs
- sorting, hash tables for batch jobs
- python UDFs
And the configuration option `taskmanager.memory.managed.consumer-weights`
only allows values: PYTHON and DATAPROC (state
11 matches
Mail list logo
