RocksDB can not be configured to spill to another filesystem or object store. It is designed as an embedded database, and each task manager needs to have sufficient disk space for its state on the host disk. You might be tempted to use network attached storage for the working state, but that's usually a bad idea (for performance reasons).
Regards, David On Fri, Dec 11, 2020 at 10:59 PM Rex Fenley <r...@remind101.com> wrote: > Hi, > > I'm wondering if the RocksDB state storage has any way of automatically > spilling into S3 (assuming that's what's set for checkpoints) when the host > disk starts filling up too much? If not, is the expectation that the host > disk must always carry all the relevant data in RocksDB, meaning we're > bound by the size of host storage? > > Thanks! > > -- > > Rex Fenley | Software Engineer - Mobile and Backend > > > Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> | > FOLLOW US <https://twitter.com/remindhq> | LIKE US > <https://www.facebook.com/remindhq> >
