Sorry to keep replying to myself, but another question. After trialling the latest H2 (1.4.200) with our existing data, I saw heap usage shoot up from 150-200MB where it hovers with 1.4.147 to 840M and rising after a fairly short run time.
Now, I expect this is totally caused by my set up since (a) our DataType.getMemory () simply returns 1024, and our MVStore cache size is set to 32000 (~32GB). I chose this due to the above issues with object size estimation so that data would effectively never leave the cache, and we manage long-term this by flushing the cache every 24 hours. Since the dataset in this case fits in a 35MB .mv file (it is compressed with LZW though), the all-in-memory idea seemed reasonable, and indeed ran stably with 1.4.197. I can see from the source code for Page and MVMap that a lot of work has gone into this area, but after some time looking at it I'm still not sure how to proceed, other than playing with other numbers for getMemory () and/or cache size. Do you have any ideas why my hacky solution no longer works and/or suggestions on how I might approach making it work with 1.4.200? Cheers, Matt. On Monday, July 19, 2021 at 11:24:30 AM UTC+9:30 Matthew Phillips wrote: > Hi Andrei, > > thanks again. We have a use-case that's very high on writes/deletes, and > the DB was apparently getting very fragmented. Using regular compact() with > 80 fill rate has halved the size, and we've done away with the nightly "big > compact" task. So for this use case it's working very well so far. > > The need for a cache flush is due to the fact that we store many versions > of the same data, most of them as small diffs against a previous version. > We reconstitute these versions into Clojure persistent maps, which means > there's a very high level of structural sharing between objects. So when > adding two maps A & B to the cache, the used memory for those maps is > almost certainly nothing like getMemory(A) + getMemory(B) due to (probable) > structural sharing between them. > > We did try some heuristic workarounds, but were always off by so much that > performance suffered badly (either flushing the cache when we didn't need > to, or writing too often). So, we use an effectively infinite cache, flush > overnight and let it re-fill. Not ideal, but don't have a better solution > right now. > > Cheers, > > Matt. > On Sunday, July 18, 2021 at 11:49:43 PM UTC+9:30 andrei...@gmail.com > wrote: > >> Hi Matt, >> IMHO, the best way to compact will be off-line one - >> MVStoreTool.compact(), and it can take only seconds (your mileage may very, >> of course). >> If you can not afford 1 min off-line interruption, then you can try just >> to let it run and do it's own maintenance in the background (asuming >> autoCommitDelay > 0). >> If I only knew some "best/better" way for on-line compaction, it would >> probably be there already, as a background maintenance procedure. >> I expect that the existing one will fit the bill, unless you update rate >> is quite high. >> BTW, flushing the cache looks like a futile exercise, indeed. >> >> On Thursday, July 15, 2021 at 7:36:05 PM UTC-4 matt...@gmail.com wrote: >> >>> Hello Andrei, >>> >>> thanks very much for your reply. >>> >>> Yes, I'm aware I'm on an old version: if it's not broken, don't fix it >>> ;) Version 1.4.197 has been rock-solid for us for years, and I'm always >>> loathe to change things for no reason. But you have given me a good reason, >>> so I'll give the latest MVStore a try. >>> >>> Can you recommend the best way to 'manually' compact the database in the >>> latest release? >>> >>> And just to be sure: could there be any data-loss issues from flushing >>> the cache? >>> >>> Cheers, >>> >>> Matt. >>> >>> On Friday, July 16, 2021 at 4:45:12 AM UTC+9:30 andrei...@gmail.com >>> wrote: >>> >>>> Hi Matt, >>>> >>>> If you are experiencing a problem, which looks and smells like a >>>> cuncurrency issue, then there is definitely a good reason to suspect a >>>> concurrency issue. 8-) >>>> The real question here is: if you care enough about those problems, why >>>> are you still on version 1.4.197. MVStore's concurrency / synchronization >>>> was totally re-designed since then (and we are talking years here), for >>>> example you will not even find MVStore.compactRewriteFully() method >>>> anymore, but instead it might just do all that space management, so you >>>> won't need that background operation at all. >>>> In any case, I would not expect that someone will look at 1.4.197 >>>> issues at this point. On the other hand, if you will find similar problem >>>> with current trunk version, and will be able to reproduce it, I will be >>>> more than happy to work on it. >>>> >>>> Cheers, >>>> Andrei. >>>> >>>> On Thursday, July 15, 2021 at 3:39:40 AM UTC-4 matt...@gmail.com wrote: >>>> >>>>> Hello, >>>>> >>>>> I'm trying to track down a perplexing problem when using an MVStore, >>>>> where it appears that a write using MVMap.put() is being dropped (H2 >>>>> 1.4.197). It's only happened twice, but both of those times have been >>>>> after >>>>> a series of cache flushes, and where the .put() is done *concurrently* >>>>> with >>>>> a long-running call to compactRewriteFully() (which takes around 90s for >>>>> this DB). We're not using rollback, or transactions or anything fancy, >>>>> just >>>>> raw put(), get(), commit(). >>>>> >>>>> My question: is there any reason to suspect that the cache flushes or, >>>>> I think more likely, the concurrent compactRewriteFully() might somehow >>>>> be >>>>> causing the write to be dropped? >>>>> >>>>> If so, I'm open to compacting the DB in some other way, but >>>>> compactRewriteFully() has been the most reliable at keeping the DB size >>>>> stable, despite its overhead (it's currently being run from a background >>>>> thread that runs once a day in the wee hours). >>>>> >>>>> Cheers, >>>>> >>>>> Matt. >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "H2 Database" group. To unsubscribe from this group and stop receiving emails from it, send an email to h2-database+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/h2-database/5c591555-ce35-48c6-96ba-cd5e31ee003an%40googlegroups.com.
