Re: Configuring RocksDB SST file size

Hi, Kishore, Could you open a JIRA for this small SST files issue? It is good to track it s.t. we won't forget this one. Thanks! -Yi On Thu, Dec 17, 2015 at 4:16 AM, Kishore N C wrote: > Hi Tao, > > > I am not sure what do you mean by ulimit issues > > When so many small SST files are created

Re: Configuring RocksDB SST file size

Hi Tao, > I am not sure what do you mean by ulimit issues When so many small SST files are created, I run into limits on maximum open files (ulimit -n). I dug into RocksDB's (plethora of) options today and identified the option that causes the 2.3 MB sizes: target_file_size_base

Re: Configuring RocksDB SST file size

Hi Kishore, I am not sure what do you mean by ulimit issues, could you help to explain a little bit? And I am not sure if user could control the size of SST files as each SST file corresponds with one sorted run. My understanding is that the SST file size could depends on how many memtables get f

Configuring RocksDB SST file size

Hi, During a catch-up job that might require reprocessing of 100s of millions of records, I wanted to tweak RocksDB configuration to ensure that it's optimized for bulk writes. According to the documentation here