> On 01 Dec 2015, at 18:34, Stephan Ewen <[email protected]> wrote: > > Hi! > > If you want to run with checkpoints (fault tolerance), you need to specify a > place to store the checkpoints to. > > By default, it is the master's memory (or zookeeper in HA), so we put a limit > on the size of the size of the state there.
Regarding the ZooKeeper in HA part: we don’t store the actual state in ZooKeeper, but a pointer to the state (e.g. a pointer to the files, which in turn store the actual state). So you don’t have to worry about ZooKeeper being flooded with your large data when you run with HA. – Ufuk PS: Nice use case, indeed! :)
