Zakelly Lan created FLINK-36838:
---
Summary: [state] Join background threads when ForSt state backend
quit
Key: FLINK-36838
URL: https://issues.apache.org/jira/browse/FLINK-36838
Project: Flink
Rui Fan created FLINK-35738:
---
Summary: Release Testing: Verify FLINK-26050 Too many small sst
files in rocksdb state backend when using time window created in ascending order
Key: FLINK-35738
URL: https
Rui Fan created FLINK-35606:
---
Summary: Release Testing Instructions: Verify FLINK-26050 Too many
small sst files in rocksdb state backend when using time window created in
ascending order
Key: FLINK-35606
URL: https
Yanfei Lei created FLINK-34667:
--
Summary: Changelog state backend support local rescaling
Key: FLINK-34667
URL: https://issues.apache.org/jira/browse/FLINK-34667
Project: Flink
Issue Type: Bug
6:45
收件人: dev@flink.apache.org<mailto:dev@flink.apache.org>
主题: Re: Anyone encounters the non-heap memory increasing issue when using
RocksDB as state backend and has better solutions?
Hi Sue Alen,
AFAIK, Flink encountered memory leak issues with RocksDB block cache. The
root cause i
/browse/FLINK-18712
[2] https://github.com/apache/flink-docker/pull/43
Best,
Zakelly
On Fri, Dec 29, 2023 at 3:35 PM Sue Alen wrote:
> Does anyone encounter the non-heap memory increasing issue when using
> RocksDB as state backend with memory controlling enabled? It has plagued us
> for
Does anyone encounter the non-heap memory increasing issue when using RocksDB
as state backend with memory controlling enabled? It has plagued us for long
time. Our flink jobs are deployed under Kubernetes, and they would be killed
because of OOM by the increasing non-heap memory by Kubernetes
Rui Fan created FLINK-33935:
---
Summary: Improve the default value doc and logic for some state
backend and checkpoint related options
Key: FLINK-33935
URL: https://issues.apache.org/jira/browse/FLINK-33935
Junrui Li created FLINK-33669:
-
Summary: Update the documentation for RestartStrategy, Checkpoint
Storage, and State Backend.
Key: FLINK-33669
URL: https://issues.apache.org/jira/browse/FLINK-33669
Dawid Wysakowicz created FLINK-33599:
Summary: Run restore tests with RocksDB state backend
Key: FLINK-33599
URL: https://issues.apache.org/jira/browse/FLINK-33599
Project: Flink
Issue
Yuchi Duan created FLINK-33437:
--
Summary: Flink 1.17 sink commited legacy Committable state, but it
was not removed from state backend
Key: FLINK-33437
URL: https://issues.apache.org/jira/browse/FLINK-33437
Ran Tao created FLINK-32485:
---
Summary: Flink State Backend Changelog should support build
test-jar
Key: FLINK-32485
URL: https://issues.apache.org/jira/browse/FLINK-32485
Project: Flink
Issue
ZhouXianming created FLINK-32112:
Summary: Fixed State Backend sample config in zh-doc
Key: FLINK-32112
URL: https://issues.apache.org/jira/browse/FLINK-32112
Project: Flink
Issue Type
Yue Shang created FLINK-28562:
-
Summary: Rocksdb state backend is too slow when using
AggregateFunction
Key: FLINK-28562
URL: https://issues.apache.org/jira/browse/FLINK-28562
Project: Flink
Juntao Hu created FLINK-27657:
-
Summary: Implement remote operator state backend in PyFlink
Key: FLINK-27657
URL: https://issues.apache.org/jira/browse/FLINK-27657
Project: Flink
Issue Type: Sub
Yuan Mei created FLINK-27214:
Summary: Build Failed on state backend benchmark
Key: FLINK-27214
URL: https://issues.apache.org/jira/browse/FLINK-27214
Project: Flink
Issue Type: Bug
Zakelly Lan created FLINK-26998:
---
Summary: Remove log-related configuration from predefined options
in RocksDB state backend
Key: FLINK-26998
URL: https://issues.apache.org/jira/browse/FLINK-26998
Roman Khachatryan created FLINK-26963:
-
Summary: Allow heap state backend creation customization
Key: FLINK-26963
URL: https://issues.apache.org/jira/browse/FLINK-26963
Project: Flink
Yun Tang created FLINK-26575:
Summary: Improve the info message when restoring keyed state
backend
Key: FLINK-26575
URL: https://issues.apache.org/jira/browse/FLINK-26575
Project: Flink
Issue
shen created FLINK-26050:
Summary: Too Many small sst files in rocksdb state backend when
using processing time window
Key: FLINK-26050
URL: https://issues.apache.org/jira/browse/FLINK-26050
Project: Flink
Galen Warren created FLINK-25873:
Summary: Using RocksDB state backend causes
java.lang.ClassNotFoundException:
org.apache.flink.runtime.state.CheckpointStreamFactory$CheckpointStateOutputStream
Key: FLINK-25873
Yun Tang created FLINK-25872:
Summary: Restoring from incremental checkpoint with changelog
state-backend cannot work well with snapshot CLAIM mode
Key: FLINK-25872
URL: https://issues.apache.org/jira/browse/FLINK
Roman Khachatryan created FLINK-25850:
-
Summary: Consider notifying nested state backend about checkpoint
abortion
Key: FLINK-25850
URL: https://issues.apache.org/jira/browse/FLINK-25850
Project
Yun Tang created FLINK-25816:
Summary: Changelog keyed state backend would come across NPE
during notification
Key: FLINK-25816
URL: https://issues.apache.org/jira/browse/FLINK-25816
Project: Flink
Yun Tang created FLINK-25423:
Summary: Enable loading state backend via configuration in state
processor api
Key: FLINK-25423
URL: https://issues.apache.org/jira/browse/FLINK-25423
Project: Flink
Dawid Wysakowicz created FLINK-25322:
Summary: Support no-claim mode in changelog state backend
Key: FLINK-25322
URL: https://issues.apache.org/jira/browse/FLINK-25322
Project: Flink
刘方奇 created FLINK-25163:
---
Summary: Add more options for rocksdb state backend to make
configuration more flexible
Key: FLINK-25163
URL: https://issues.apache.org/jira/browse/FLINK-25163
Project: Flink
Yun Tang created FLINK-24783:
Summary: Improve state backend monitoring experience and usability
Key: FLINK-24783
URL: https://issues.apache.org/jira/browse/FLINK-24783
Project: Flink
Issue Type
shiwuliang created FLINK-24314:
--
Summary: Always use memory state backend with RocksDB
Key: FLINK-24314
URL: https://issues.apache.org/jira/browse/FLINK-24314
Project: Flink
Issue Type: Bug
information
when specify the type for the values in the state. Type Information could
generates serializer later. Please see more information in [1].
Snapshot for all meta information about one state in a state backend would
be stored in `StateMetaInfoSnapshot` which would be used in
serializationProxy
Hello,
I have been trying to understand how serialization works for the Flink Heap
Keyed state backend when a checkpoint happens. I want to know how Flink knows
which type of serializer to use to serialize the values for different states
during the snapshot.
I have debugged both RocksDB and
Seth Wiesman created FLINK-23728:
Summary: Support new state backend factory stack in State
Processor API
Key: FLINK-23728
URL: https://issues.apache.org/jira/browse/FLINK-23728
Project: Flink
Mason Chen created FLINK-23519:
--
Summary: Aggregate State Backend Latency by State Level
Key: FLINK-23519
URL: https://issues.apache.org/jira/browse/FLINK-23519
Project: Flink
Issue Type
yang89520 created FLINK-23440:
-
Summary: The "State. Backend" of "Flink-CONF.YAML" is inconsistent
with the documentation
Key: FLINK-23440
URL: https://issues.apache.org/jira/browse/FLINK-23440
Piotr Nowojski created FLINK-23398:
--
Summary: State backend benchmarks are failing
Key: FLINK-23398
URL: https://issues.apache.org/jira/browse/FLINK-23398
Project: Flink
Issue Type: Bug
Stephan Ewen created FLINK-22749:
Summary: Apply a robust default State Backend Configuration
Key: FLINK-22749
URL: https://issues.apache.org/jira/browse/FLINK-22749
Project: Flink
Issue
Hi Jun Qin,
Do you have any example OptionsFactory for Bloom Filter.
I did experiment and change Options from FLASH_SSD_OPTIMIZED to
SPINNING_DISK_OPTIMIZED_HIGH_MEM.
This gives me 2x better performance on NVME disk.
I think the reason is that I'm doing a lot of reads and
SPINNING_DISK_OPTIMIZED_H
Yun Tang created FLINK-22369:
Summary: RocksDB state backend might occur ClassNotFoundException
when deserializing on TM side
Key: FLINK-22369
URL: https://issues.apache.org/jira/browse/FLINK-22369
Yun Tang created FLINK-22146:
Summary: Migrate StateBackend related Chinese docs to latest state
backend
Key: FLINK-22146
URL: https://issues.apache.org/jira/browse/FLINK-22146
Project: Flink
Roman Khachatryan created FLINK-21648:
-
Summary: FLIP-151: Incremental snapshots for heap-based state
backend
Key: FLINK-21648
URL: https://issues.apache.org/jira/browse/FLINK-21648
Project
Hi everyone,
The voting time for FLIP-151: Incremental snapshots for heap-based
state backend [1] has passed. I'm closing the vote now.
There were 5 +1 votes, 4 of which are binding:
- Piotr Nowojski (binding)
- Zhijiang (binding)
- David Anderson (non-binding)
- Yu Li (binding)
-
for the FLIP.
> > >
> > > Please vote +1 to approve the FLIP, or -1 with a comment. The vote will
> > be
> > > open at least until Wednesday, Mar 3rd.
> > >
> > > [1] https://s.apache.org/flip-151-discussion
> > > [2]
> > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+Incremental+snapshots+for+heap-based+state+backend
> > >
> > > Regards,
> > > Roman
> > >
> >
gt; > [1] https://s.apache.org/flip-151-discussion
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+Incremental+snapshots+for+heap-based+state+backend
> >
> > Regards,
> > Roman
> >
>
comment. The vote will be
> open at least until Wednesday, Mar 3rd.
>
> [1] https://s.apache.org/flip-151-discussion
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+Incremental+snapshots+for+heap-based+state+backend
>
> Regards,
> Roman
>
+1 (binding) from my side, wish this feature going well!
Best,
Zhijiang
--
From:Piotr Nowojski
Send Time:2021年3月2日(星期二) 00:11
To:dev ; roman
Subject:Re: [VOTE] FLIP-151: Incremental snapshots for heap-based state backend
Thanks
art a formal vote for the FLIP.
>
> Please vote +1 to approve the FLIP, or -1 with a comment. The vote will be
> open at least until Wednesday, Mar 3rd.
>
> [1] https://s.apache.org/flip-151-discussion
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+I
p-151-discussion
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+Incremental+snapshots+for+heap-based+state+backend
Regards,
Roman
Yuan Mei created FLINK-21448:
Summary: Test Changelog State backend
Key: FLINK-21448
URL: https://issues.apache.org/jira/browse/FLINK-21448
Project: Flink
Issue Type: Sub-task
Qian Chao created FLINK-21437:
-
Summary: Memory leak when using filesystem state backend on
Alibaba Cloud OSS
Key: FLINK-21437
URL: https://issues.apache.org/jira/browse/FLINK-21437
Project: Flink
>> > other than common maintenance and small bug fixes. If we expect that
>> this
>> > requires a lot of follow-ups, then we end up splitting our work between
>> > this FLIP and FLIP-158, which seems a bit inefficient.
>> > What other committers would be
olved to ensure the community can
> > maintain this?
> >
> >
> > The design looks fine, in general, with one question:
> >
> > When persisting changes, you persist all changes that have a newer
> version
> > than the latest one confirmed by the JM.
> >
community can
> maintain this?
>
>
> The design looks fine, in general, with one question:
>
> When persisting changes, you persist all changes that have a newer version
> than the latest one confirmed by the JM.
>
> Can you explain why it is like that exactly? Alternatively,
e, in general, with one question:
When persisting changes, you persist all changes that have a newer version
than the latest one confirmed by the JM.
Can you explain why it is like that exactly? Alternatively, you could keep
the latest checkpoint ID for which the state backend persisted the
un Tang
>
> From: Till Rohrmann
> Sent: Monday, February 8, 2021 17:06
> To: dev
> Subject: Re: Activate bloom filter in RocksDB State Backend via Flink
> configuration
>
> Hi Jun,
>
> Making things easier to use and configure
Jun Qin created FLINK-21336:
---
Summary: Activate bloom filter in RocksDB State Backend via Flink
configuration
Key: FLINK-21336
URL: https://issues.apache.org/jira/browse/FLINK-21336
Project: Flink
Tang
From: Till Rohrmann
Sent: Monday, February 8, 2021 17:06
To: dev
Subject: Re: Activate bloom filter in RocksDB State Backend via Flink
configuration
Hi Jun,
Making things easier to use and configure is a good idea. Hence, +1 for
this proposal. Maybe create a JIRA ticket fo
Hi,
>
> Activating bloom filter in the RocksDB state backend improves read
> performance. Currently activating bloom filter can only be done by
> implementing a custom ConfigurableRocksDBOptionsFactory. I think we should
> provide an option to activate bloom filter via Flink configuration. W
Hi,
Activating bloom filter in the RocksDB state backend improves read performance.
Currently activating bloom filter can only be done by implementing a custom
ConfigurableRocksDBOptionsFactory. I think we should provide an option to
activate bloom filter via Flink configuration. What do you
Dawid Wysakowicz created FLINK-20517:
Summary: Disallow operators with mixed key/non-keyed inputs with
BATCH state backend
Key: FLINK-20517
URL: https://issues.apache.org/jira/browse/FLINK-20517
g the versions on this event - this allows
> you to let go of old snapshots and saves you from writing a log of
> antimatter entries. Maybe the ideas are still useful to you.
>
> Best,
> Stefan
>
> On 2020/11/04 01:54:25, Khachatryan Roman wrote:
> > Hi devs,>
> >
>
,
Stefan
On 2020/11/04 01:54:25, Khachatryan Roman wrote:
> Hi devs,>
>
> I'd like to start a discussion of FLIP-151: Incremental snapshots for>
> heap-based state backend [1]>
>
> Heap backend, while being limited state sizes fitting into memory, also has
Hi devs,
I'd like to start a discussion of FLIP-151: Incremental snapshots for
heap-based state backend [1]
Heap backend, while being limited state sizes fitting into memory, also has
some advantages compared to RocksDB backend:
1. Serialization once per checkpoint, not per state modific
Roman Khachatryan created FLINK-18786:
-
Summary: Performance regresssion on 2020.07.29 in some state
backend benchmarks
Key: FLINK-18786
URL: https://issues.apache.org/jira/browse/FLINK-18786
Yun Tang created FLINK-18373:
Summary: Drop performance unit tests in RocksDB state backend
module
Key: FLINK-18373
URL: https://issues.apache.org/jira/browse/FLINK-18373
Project: Flink
Issue
Yun Tang created FLINK-18060:
Summary: Add notice for heap-based state backend of state access
reference problem
Key: FLINK-18060
URL: https://issues.apache.org/jira/browse/FLINK-18060
Project: Flink
Yun Tang created FLINK-16876:
Summary: Make TtlTimeProvider configurable when creating keyed
state backend
Key: FLINK-16876
URL: https://issues.apache.org/jira/browse/FLINK-16876
Project: Flink
Yu Li created FLINK-14922:
-
Summary: Add performance e2e test suite for state backend
Key: FLINK-14922
URL: https://issues.apache.org/jira/browse/FLINK-14922
Project: Flink
Issue Type: Sub-task
Kezhu Wang created FLINK-14793:
--
Summary: Add SPI interface to discovery state backend and
establish well-known names
Key: FLINK-14793
URL: https://issues.apache.org/jira/browse/FLINK-14793
Project
Kezhu Wang created FLINK-14790:
--
Summary: Custom state backend in job graph composition without
depending on concrete state backend implementation
Key: FLINK-14790
URL: https://issues.apache.org/jira/browse/FLINK
Yun Tang created FLINK-14495:
Summary: Add documentation for memory control of RocksDB state
backend with writebuffer manager
Key: FLINK-14495
URL: https://issues.apache.org/jira/browse/FLINK-14495
Yu Li created FLINK-14492:
-
Summary: Modify state backend to report reserved memory to
MemoryManager
Key: FLINK-14492
URL: https://issues.apache.org/jira/browse/FLINK-14492
Project: Flink
Issue
Yu Li created FLINK-14489:
-
Summary: Check and make sure state backend fits into TaskExecutor
memory configuration
Key: FLINK-14489
URL: https://issues.apache.org/jira/browse/FLINK-14489
Project: Flink
gt; >
> > > On Fri, Aug 16, 2019 at 4:06 AM Congxian Qiu
> > > wrote:
> > >
> > > > Big +1 for this feature.
> > > >
> > > > This FLIP can help improves at least the following two scenarios:
> > > > - Tempora
ers,
> > Till
> >
> > On Fri, Aug 16, 2019 at 4:06 AM Congxian Qiu
> > wrote:
> >
> > > Big +1 for this feature.
> > >
> > > This FLIP can help improves at least the following two scenarios:
> > > - Temporary data peak when usi
, 2019 if we have sufficient
> > > votes.
> > > > Thanks.
> > > >
> > > > [1] https://s.apache.org/cq358
> > > > [2]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > >
> >
>
e burden.
>
> Cheers,
> Till
>
> On Fri, Aug 16, 2019 at 4:06 AM Congxian Qiu
> wrote:
>
> > Big +1 for this feature.
> >
> > This FLIP can help improves at least the following two scenarios:
> > - Temporary data peak when using Heap StateBackend
sday August 20, 2019 if we have sufficient
> > votes.
> > > Thanks.
> > >
> > > [1] https://s.apache.org/cq358
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
> > >
> > > Best Regards,
> > > Yu
> > >
> >
>
t; >
> > This vote will be open for at least 72 hours. Unless objection I will try
> > to close it by end of Tuesday August 20, 2019 if we have sufficient
> votes.
> > Thanks.
> >
> > [1] https://s.apache.org/cq358
> > [2]
> >
> >
> https
y end of Tuesday August 20, 2019 if we have sufficient votes.
> Thanks.
>
> [1] https://s.apache.org/cq358
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
>
> Best Regards,
> Yu
>
https://s.apache.org/cq358
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
Best Regards,
Yu
Fri, Aug 16, 2019 at 4:06 AM Congxian Qiu wrote:
> Big +1 for this feature.
>
> This FLIP can help improves at least the following two scenarios:
> - Temporary data peak when using Heap StateBackend
> - Heap State Backend has better performance than RocksDBStateBackend,
> espec
Big +1 for this feature.
This FLIP can help improves at least the following two scenarios:
- Temporary data peak when using Heap StateBackend
- Heap State Backend has better performance than RocksDBStateBackend,
especially on SATA disk. there are some guys ever told me that they
increased the
ordon) Tai
wrote:
> +1 to start a VOTE for this FLIP.
>
> Given the properties of this new state backend and that it will exist as a
> new module without touching the original heap backend, I don't see a harm
> in including this.
> Regarding design of the feature, I've alr
+1 to start a VOTE for this FLIP.
Given the properties of this new state backend and that it will exist as a
new module without touching the original heap backend, I don't see a harm
in including this.
Regarding design of the feature, I've already mentioned my comments in the
original
make me feel fear about the OOM errors.
Look forward for more powerful enrichment to state-backend, and help Flink to
achieve better performance together.
Best
Yun Tang
From: Stephan Ewen
Sent: Thursday, August 15, 2019 23:07
To: dev
Subject: Re: [DISCUSS] FLIP-50
here it is:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
>
> Thanks!
>
> Best Regards,
> Yu
>
>
> On Tue, 13 Aug 2019 at 18:06, Yu Li wrote:
>
> > Hi All,
> >
> > We ever held a discussion about this
Sorry for forgetting to give the link of the FLIP, here it is:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
Thanks!
Best Regards,
Yu
On Tue, 13 Aug 2019 at 18:06, Yu Li wrote:
> Hi All,
>
> We ever held a discussion about thi
Hi All,
We ever held a discussion about this feature before [1] but now opening
another thread because after a second thought introducing a new backend
instead of modifying the existing heap backend is a better option to
prevent causing any regression or surprise to existing in-production usage.
A
Maximilian Michels created FLINK-12653:
--
Summary: Keyed state backend fails to restore during rescaling
Key: FLINK-12653
URL: https://issues.apache.org/jira/browse/FLINK-12653
Project: Flink
Fabian Hueske created FLINK-12486:
-
Summary: RocksDB State Backend docs don't explain that operator
state is kept on heap
Key: FLINK-12486
URL: https://issues.apache.org/jira/browse/FLINK-
Yu Li created FLINK-12066:
-
Summary: Remove StateSerializerProvider field in keyed state
backend
Key: FLINK-12066
URL: https://issues.apache.org/jira/browse/FLINK-12066
Project: Flink
Issue Type
Elias Levy created FLINK-11517:
--
Summary: Inefficient window state access when using RocksDB state
backend
Key: FLINK-11517
URL: https://issues.apache.org/jira/browse/FLINK-11517
Project: Flink
local snapshots can be done cheaply and asynchronously.
>>
>> I heard folks tried to build mysqlbackend(deprecated), remote rocksdb as
>> service backend(hard to scale and performance bottleneck) , Cassandra(hard
>> to snapshot). All of which shares same trait on lack of loca
d to build mysqlbackend(deprecated), remote rocksdb as
> service backend(hard to scale and performance bottleneck) , Cassandra(hard
> to snapshot). All of which shares same trait on lack of local
> parallelizable snapshot semantic.
>
> Hope this helps!
> Chen
>
> On Thu, Dec
AM miki haiat wrote:
> Did try to use rocksdb[1] as state backend?
>
>
> 1.
>
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html#the-rocksdbstatebackend
>
>
> On Thu, 27 Dec 2018, 18:17 Naveen Kumar .invalid
> wrote:
>
> &g
Did try to use rocksdb[1] as state backend?
1.
https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html#the-rocksdbstatebackend
On Thu, 27 Dec 2018, 18:17 Naveen Kumar Hi,
>
> I am exploring if we can plugin hbase as state backend in Flink. We have
>
successfully used rocksdb state backend with incremental snapshots
with several terabytes of state in production for years.
The only main advantage I see for hbase and similar kvstores as
statebackend is the instant recovery you get but even in that case you
probably want an implementation that combines
Hi,
I am exploring if we can plugin hbase as state backend in Flink. We have
need for streaming jobs with large window states, high throughput and
reliability.
I wanted to know if implementing Flink backend in Hbase or other
distributed KV store is possible. Any documentation or pointers will be
Sayat Satybaldiyev created FLINK-11171:
--
Summary: Unexpected timestamp deserialization failure in RocksDB
state backend
Key: FLINK-11171
URL: https://issues.apache.org/jira/browse/FLINK-11171
cailiuyang created FLINK-11138:
--
Summary: RocksDB's wal seems to be useless in
rocksdb-state-backend that reduce the disk's overhead
Key: FLINK-11138
URL: https://issues.apache.org/jira/browse/F
1 - 100 of 168 matches
Mail list logo
