Hi Maybe FLIP-49[1] "Unified Memory Configuration for TaskExecutors" can give some information here
[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-49-Unified-Memory-Configuration-for-TaskExecutors-td31436.html Best, Congxian Cam Mach <cammac...@gmail.com> 于2019年8月9日周五 上午4:59写道： > Hi Biao, Yun and Ning. > > Thanks for your response and pointers. Those are very helpful! > > So far, we have tried with some of those parameters (WriterBufferManager, > write_buffer_size, write_buffer_count, ...), but still continuously having > issues with memory. > Here are our cluster configurations: > > - 1 Job Controller (32 GB RAM and 8 cores) > - 10 Task Managers: (32 GB RAM, 8 cores CPU, and 300GB SSD configured > for RocksDB, and we set 10GB heap for each) > - Running under Kuberntes > > We have a pipeline that read/transfer 500 million records (around 1kb > each), and write to our sink. Our total data is around 1.2 Terabytes. Our > pipeline configurations are as follows: > > - 13 operators - some of them (around 6) are stateful > - Parallelism: 60 > - Task slots: 6 > > We have run several tests and observed that memory just keep growing while > our TM's CPU stay around 10 - 15% usage. We are now just focusing limiting > memory usage from Flink and RocksDB, so Kubernetes won't kill it. > > Any recommendations or advices are greatly appreciated! > > Thanks, > > > > > On Thu, Aug 8, 2019 at 6:57 AM Yun Tang <myas...@live.com> wrote: > >> Hi Cam >> >> I think FLINK-7289 [1] might offer you some insights to control RocksDB >> memory, especially the idea using write buffer manager [2] to control the >> total write buffer memory. If you do not have too many sst files, write >> buffer memory usage would consume much more space than index and filter >> usage. Since Flink would use per state per column family, and the write >> buffer number increase when more column families created. >> >> >> [1] https://issues.apache.org/jira/browse/FLINK-7289 >> [2] https://github.com/dataArtisans/frocksdb/pull/4 >> >> Best >> Yun Tang >> >> >> ------------------------------ >> *From:* Cam Mach <cammac...@gmail.com> >> *Sent:* Thursday, August 8, 2019 21:39 >> *To:* Biao Liu <mmyy1...@gmail.com> >> *Cc:* miki haiat <miko5...@gmail.com>; user <user@flink.apache.org> >> *Subject:* Re: Capping RocksDb memory usage >> >> Thanks for your response, Biao. >> >> >> >> On Wed, Aug 7, 2019 at 11:41 PM Biao Liu <mmyy1...@gmail.com> wrote: >> >> Hi Cam, >> >> AFAIK, that's not an easy thing. Actually it's more like a Rocksdb issue. >> There is a document explaining the memory usage of Rocksdb [1]. It might be >> helpful. >> >> You could define your own option to tune Rocksdb through >> "state.backend.rocksdb.options-factory" [2]. However I would suggest not to >> do this unless you are fully experienced of Rocksdb. IMO it's quite >> complicated. >> >> Meanwhile I can share a bit experience of this. We have tried to put the >> cache and filter into block cache before. It's useful to control the memory >> usage. But the performance might be affected at the same time. Anyway you >> could try and tune it. Good luck! >> >> 1. https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB >> 2. >> https://ci.apache.org/projects/flink/flink-docs-master/ops/state/large_state_tuning.html#tuning-rocksdb >> >> Thanks, >> Biao /'bɪ.aʊ/ >> >> >> >> On Thu, Aug 8, 2019 at 11:44 AM Cam Mach <cammac...@gmail.com> wrote: >> >> Yes, that is correct. >> Cam Mach >> Software Engineer >> E-mail: cammac...@gmail.com >> Tel: 206 972 2768 >> >> >> >> On Wed, Aug 7, 2019 at 8:33 PM Biao Liu <mmyy1...@gmail.com> wrote: >> >> Hi Cam, >> >> Do you mean you want to limit the memory usage of RocksDB state backend? >> >> Thanks, >> Biao /'bɪ.aʊ/ >> >> >> >> On Thu, Aug 8, 2019 at 2:23 AM miki haiat <miko5...@gmail.com> wrote: >> >> I think using metrics exporter is the easiest way >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#rocksdb >> >> >> On Wed, Aug 7, 2019, 20:28 Cam Mach <cammac...@gmail.com> wrote: >> >> Hello everyone, >> >> What is the most easy and efficiently way to cap RocksDb's memory usage? >> >> Thanks, >> Cam >> >>
