Re: Capping RocksDb memory usage

t;>> On Thu, Aug 8, 2019 at 6:57 AM Yun Tang wrote: >>>> >>>>> Hi Cam >>>>> >>>>> I think FLINK-7289 [1] might offer you some insights to control >>>>> RocksDB memory, especially the idea using write buffer manager [2] to >

Re: Capping RocksDb memory usage

er manager [2] to control the >>>> total write buffer memory. If you do not have too many sst files, write >>>> buffer memory usage would consume much more space than index and filter >>>> usage. Since Flink would use per state per column family, and the write >&g

Re: Capping RocksDb memory usage

s, write >>> buffer memory usage would consume much more space than index and filter >>> usage. Since Flink would use per state per column family, and the write >>> buffer number increase when more column families created. >>> >>> >>> [1] https://is

Re: Capping RocksDb memory usage

hen more column families created. >> >> >> [1] https://issues.apache.org/jira/browse/FLINK-7289 >> [2] https://github.com/dataArtisans/frocksdb/pull/4 >> >> Best >> Yun Tang >> >> >> -- >> *From:* Cam Mach >> *Sent:* Thursday, August 8, 2019

Re: Capping RocksDb memory usage

m:* Cam Mach > *Sent:* Thursday, August 8, 2019 21:39 > *To:* Biao Liu > *Cc:* miki haiat ; user > *Subject:* Re: Capping RocksDb memory usage > > Thanks for your response, Biao. > > > > On Wed, Aug 7, 2019 at 11:41 PM Biao Liu wrote: > > Hi Cam, > > A

Re: Capping RocksDb memory usage

: Cam Mach Sent: Thursday, August 8, 2019 21:39 To: Biao Liu Cc: miki haiat ; user Subject: Re: Capping RocksDb memory usage Thanks for your response, Biao. On Wed, Aug 7, 2019 at 11:41 PM Biao Liu mailto:mmyy1...@gmail.com>> wrote: Hi Cam, AFAIK, that's not an easy thing. Actuall

Re: Capping RocksDb memory usage

Hi Cam, This blog post has some pointers in tuning RocksDB memory usage that might be of help. https://klaviyo.tech/flinkperf-c7bd28acc67 Ning On Wed, Aug 7, 2019 at 1:28 PM Cam Mach wrote: > > Hello everyone, > > What is the most easy and efficiently way to cap RocksDb's memory usage? > > Tha

Re: Capping RocksDb memory usage

Thanks for your response, Biao. On Wed, Aug 7, 2019 at 11:41 PM Biao Liu wrote: > Hi Cam, > > AFAIK, that's not an easy thing. Actually it's more like a Rocksdb issue. > There is a document explaining the memory usage of Rocksdb [1]. It might be > helpful. > > You could define your own option

Re: Capping RocksDb memory usage

Hi Cam, AFAIK, that's not an easy thing. Actually it's more like a Rocksdb issue. There is a document explaining the memory usage of Rocksdb [1]. It might be helpful. You could define your own option to tune Rocksdb through "state.backend.rocksdb.options-factory" [2]. However I would suggest not

Re: Capping RocksDb memory usage

Yes, that is correct. Cam Mach Software Engineer E-mail: cammac...@gmail.com Tel: 206 972 2768 On Wed, Aug 7, 2019 at 8:33 PM Biao Liu wrote: > Hi Cam, > > Do you mean you want to limit the memory usage of RocksDB state backend? > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Thu, Aug 8, 2019 at 2:23 A

Re: Capping RocksDb memory usage

Hi Cam, Do you mean you want to limit the memory usage of RocksDB state backend? Thanks, Biao /'bɪ.aʊ/ On Thu, Aug 8, 2019 at 2:23 AM miki haiat wrote: > I think using metrics exporter is the easiest way > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html

Re: Capping RocksDb memory usage

I think using metrics exporter is the easiest way [1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#rocksdb On Wed, Aug 7, 2019, 20:28 Cam Mach wrote: > Hello everyone, > > What is the most easy and efficiently way to cap RocksDb's memory usage? > > Thanks, > C

Capping RocksDb memory usage

Hello everyone, What is the most easy and efficiently way to cap RocksDb's memory usage? Thanks, Cam