As a follow-up, I managed to get everything to work as I wanted with forking but I got a major issue..
The golang runtime mutexes are invalidated by the forking, that means that anything in the runtime that needs a mutex is broken so garbage collection had to be disabled and additional memory allocation that required the heap to be enlarged was also broken. When a runtime operation was requiring a mutex, it would lock forever and since the process was single threaded after the fork, then having the main thread locked would make the process hang forever. Although this didn't happen in all cases (would succeed 9/10 times with memory pre-allocation), it isn't suitable for me and will certainly go beyond my knowledge of Golang when I hit those issues in production. So, instead of fighting against Golang and most likely lose the battle, I decided to go with the fine grain locking using the following logic: - RLock the map and grab a copy of its keys - Iterate through all the keys -- RLock the map to get the pointer of the struct associated to the current key and release it right after -- Check that the pointer is not pointing to nil (keys can disapear in my case) -- RLock the structure -- Serialize the structure -- Release the RLock on the structure - Write the serialized structures to the disk I'll be optimizing it as well to write to the disk while I'm serializing each individual struct to same some more memory but I found it to be effective and I already had part of the plumbing in place to make this lock strategy work I would have found it nice though to be able to make the Redis strategy work but I do realize now that the candies that come with Golang (GC + heap) have some limitations when working with very low level system calls (at least this is my understanding of it) Anyway, thanks to everybody for the input - Julien On Tuesday, 30 January 2018 07:36:26 UTC-5, Julien Semaan wrote: > > I believe copying the map containing structs and not pointers would make > it consume twice the amount of memory which isn't desirable in my case. > > - Julien > > On Tuesday, 30 January 2018 00:09:21 UTC-5, Tamás Gulácsi wrote: >> >> Do you really need pointers tonyour structs in the map? Because that >> causes the headache here: if no pointers, then simply rlock-ing the map, >> copying / serializing would be enough. >> >> Could even have two maps, each gets every modification, except when >> copying, when the secondary is serialized, thn updated with the accumulated >> changes. With a goroutine managing the secondary, executing commands >> (update, delete, marshal) from a buffered channel, this wouldn't uterfere >> with the main map. >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
