Denis, Several clarifying questions: 1. Do you have an idea why metadata registration takes so long? So poor disks? So many data to write? A contention with disk writes by other subsystems? 2. Do we need a persistent metadata for in-memory caches? Or is it so accidentally?
Generally, I think that it is possible to move metadata saving operations out of discovery thread without loosing required consistency/integrity. As Alex mentioned using metastore looks like a better solution. Do we really need a fast fix here? (Are we talking about fast fix?) ср, 14 авг. 2019 г. в 11:45, Zhenya Stanilovsky <arzamas...@mail.ru.invalid>: > > Alexey, but in this case customer need to be informed, that whole (for > example 1 node) cluster crash (power off) could lead to partial data > unavailability. > And may be further index corruption. > 1. Why your meta takes a substantial size? may be context leaking ? > 2. Could meta be compressed ? > > > >Среда, 14 августа 2019, 11:22 +03:00 от Alexei Scherbakov > ><alexey.scherbak...@gmail.com>: > > > >Denis Mekhanikov, > > > >Currently metadata are fsync'ed on write. This might be the case of > >slow-downs in case of metadata burst writes. > >I think removing fsync could help to mitigate performance issues with > >current implementation until proper solution will be implemented: moving > >metadata to metastore. > > > > > >вт, 13 авг. 2019 г. в 17:09, Denis Mekhanikov < dmekhani...@gmail.com >: > > > >> I would also like to mention, that marshaller mappings are written to disk > >> even if persistence is disabled. > >> So, this issue affects purely in-memory clusters as well. > >> > >> Denis > >> > >> > On 13 Aug 2019, at 17:06, Denis Mekhanikov < dmekhani...@gmail.com > > >> wrote: > >> > > >> > Hi! > >> > > >> > When persistence is enabled, binary metadata is written to disk upon > >> registration. Currently it happens in the discovery thread, which makes > >> processing of related messages very slow. > >> > There are cases, when a lot of nodes and slow disks can make every > >> binary type be registered for several minutes. Plus it blocks processing of > >> other messages. > >> > > >> > I propose starting a separate thread that will be responsible for > >> writing binary metadata to disk. So, binary type registration will be > >> considered finished before information about it will is written to disks on > >> all nodes. > >> > > >> > The main concern here is data consistency in cases when a node > >> acknowledges type registration and then fails before writing the metadata > >> to disk. > >> > I see two parts of this issue: > >> > Nodes will have different metadata after restarting. > >> > If we write some data into a persisted cache and shut down nodes faster > >> than a new binary type is written to disk, then after a restart we won’t > >> have a binary type to work with. > >> > > >> > The first case is similar to a situation, when one node fails, and after > >> that a new type is registered in the cluster. This issue is resolved by the > >> discovery data exchange. All nodes receive information about all binary > >> types in the initial discovery messages sent by other nodes. So, once you > >> restart a node, it will receive information, that it failed to finish > >> writing to disk, from other nodes. > >> > If all nodes shut down before finishing writing the metadata to disk, > >> then after a restart the type will be considered unregistered, so another > >> registration will be required. > >> > > >> > The second case is a bit more complicated. But it can be resolved by > >> making the discovery threads on every node create a future, that will be > >> completed when writing to disk is finished. So, every node will have such > >> future, that will reflect the current state of persisting the metadata to > >> disk. > >> > After that, if some operation needs this binary type, it will need to > >> wait on that future until flushing to disk is finished. > >> > This way discovery threads won’t be blocked, but other threads, that > >> actually need this type, will be. > >> > > >> > Please let me know what you think about that. > >> > > >> > Denis > >> > >> > > > >-- > > > >Best regards, > >Alexei Scherbakov > > > -- > Zhenya Stanilovsky -- Best regards, Ivan Pavlukhin
