Hi Andrei, thanks again. We have a use-case that's very high on writes/deletes, and the DB was apparently getting very fragmented. Using regular compact() with 80 fill rate has halved the size, and we've done away with the nightly "big compact" task. So for this use case it's working very well so far.
The need for a cache flush is due to the fact that we store many versions of the same data, most of them as small diffs against a previous version. We reconstitute these versions into Clojure persistent maps, which means there's a very high level of structural sharing between objects. So when adding two maps A & B to the cache, the used memory for those maps is almost certainly nothing like getMemory(A) + getMemory(B) due to (probable) structural sharing between them. We did try some heuristic workarounds, but were always off by so much that performance suffered badly (either flushing the cache when we didn't need to, or writing too often). So, we use an effectively infinite cache, flush overnight and let it re-fill. Not ideal, but don't have a better solution right now. Cheers, Matt. On Sunday, July 18, 2021 at 11:49:43 PM UTC+9:30 andrei...@gmail.com wrote: > Hi Matt, > IMHO, the best way to compact will be off-line one - > MVStoreTool.compact(), and it can take only seconds (your mileage may very, > of course). > If you can not afford 1 min off-line interruption, then you can try just > to let it run and do it's own maintenance in the background (asuming > autoCommitDelay > 0). > If I only knew some "best/better" way for on-line compaction, it would > probably be there already, as a background maintenance procedure. > I expect that the existing one will fit the bill, unless you update rate > is quite high. > BTW, flushing the cache looks like a futile exercise, indeed. > > On Thursday, July 15, 2021 at 7:36:05 PM UTC-4 matt...@gmail.com wrote: > >> Hello Andrei, >> >> thanks very much for your reply. >> >> Yes, I'm aware I'm on an old version: if it's not broken, don't fix it ;) >> Version 1.4.197 has been rock-solid for us for years, and I'm always loathe >> to change things for no reason. But you have given me a good reason, so >> I'll give the latest MVStore a try. >> >> Can you recommend the best way to 'manually' compact the database in the >> latest release? >> >> And just to be sure: could there be any data-loss issues from flushing >> the cache? >> >> Cheers, >> >> Matt. >> >> On Friday, July 16, 2021 at 4:45:12 AM UTC+9:30 andrei...@gmail.com >> wrote: >> >>> Hi Matt, >>> >>> If you are experiencing a problem, which looks and smells like a >>> cuncurrency issue, then there is definitely a good reason to suspect a >>> concurrency issue. 8-) >>> The real question here is: if you care enough about those problems, why >>> are you still on version 1.4.197. MVStore's concurrency / synchronization >>> was totally re-designed since then (and we are talking years here), for >>> example you will not even find MVStore.compactRewriteFully() method >>> anymore, but instead it might just do all that space management, so you >>> won't need that background operation at all. >>> In any case, I would not expect that someone will look at 1.4.197 issues >>> at this point. On the other hand, if you will find similar problem with >>> current trunk version, and will be able to reproduce it, I will be more >>> than happy to work on it. >>> >>> Cheers, >>> Andrei. >>> >>> On Thursday, July 15, 2021 at 3:39:40 AM UTC-4 matt...@gmail.com wrote: >>> >>>> Hello, >>>> >>>> I'm trying to track down a perplexing problem when using an MVStore, >>>> where it appears that a write using MVMap.put() is being dropped (H2 >>>> 1.4.197). It's only happened twice, but both of those times have been >>>> after >>>> a series of cache flushes, and where the .put() is done *concurrently* >>>> with >>>> a long-running call to compactRewriteFully() (which takes around 90s for >>>> this DB). We're not using rollback, or transactions or anything fancy, >>>> just >>>> raw put(), get(), commit(). >>>> >>>> My question: is there any reason to suspect that the cache flushes or, >>>> I think more likely, the concurrent compactRewriteFully() might somehow be >>>> causing the write to be dropped? >>>> >>>> If so, I'm open to compacting the DB in some other way, but >>>> compactRewriteFully() has been the most reliable at keeping the DB size >>>> stable, despite its overhead (it's currently being run from a background >>>> thread that runs once a day in the wee hours). >>>> >>>> Cheers, >>>> >>>> Matt. >>>> >>> -- You received this message because you are subscribed to the Google Groups "H2 Database" group. To unsubscribe from this group and stop receiving emails from it, send an email to h2-database+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/h2-database/8e6369db-58d6-454a-997a-ced73d655091n%40googlegroups.com.
