My main goal is to avoid complex granular locking inside the data structure since writes are very common in the map (whether its adding items or modifying the actual item). My idea was to implement MarshalJSON for my map and then while creating the JSON I'd be holding locks at the appropriate times. That is not my preferred solution though because of the locking.
As for forking... I believe forking on Linux is now very efficient due to copy-on-write so the cost should be minimal. I got some kind of basic prototype working but I need to figure out if it will stay memory and performance efficient when my parent starts writing to the map. Since its a map of pointers, I think the footprint will be minimal. When I get my prototype done, I'll likely post it here to get some input on whether or not it does make sense - Julien On Monday, 29 January 2018 10:39:35 UTC-5, matthe...@gmail.com wrote: > > Can you describe your performance needs in more detail? > > I missed the need for the struct values in the copy (obviously needed to > encode to JSON). Going back to locks for a moment, my thought now is that a > sync.RWMutex on this map would have an RLock taken by every struct > read/write and then the Lock would be taken for the map copy which would > return a map to copied structs instead of a map to pointers. Wouldn’t a > fork lock the entire process while a larger copy happens anyway? > > Where I’d start otherwise is adding additional data. Perhaps a second map > could help with performance here. > > Matt > > On Monday, January 29, 2018 at 6:44:17 AM UTC-6, Julien Semaan wrote: >> >> Hi, >> >> Thanks for all the input. >> >> Unfortunately, as Tamás and Jake pointed out, I can't simply grab a copy >> of the map that has the same pointers since I'll need to be locking the >> structure everytime I'm copying it. >> So that means Matt's idea wouldn't really work for my use-case >> >> I was really hoping to have something similar to the fork behavior where >> I could be working with a memory copy of the process in order to be able to >> persist everything without holding a single lock either on the map or on >> the actual structure. >> >> - Julien >> >> On Friday, 26 January 2018 15:44:46 UTC-5, Julien Semaan wrote: >>> >>> Hi, >>> >>> I'm currently tackling a head scratching problem (at least for me). >>> >>> First, I'll start by explaining my main goal: >>> I have an in memory 'map[string]*SomeStructPointer' that I want to JSON >>> encode and write to a file. >>> Now since I want to prevent concurrent access to the map, I need to >>> lock-down access to it during the JSON encoding. >>> I want to avoid that locking or have it held for the smallest amount of >>> time. >>> >>> Next, my idea: >>> Fork the process, leveraging the copy-on-write, JSON encode the good >>> stuff, write it to a file, then exit the child process. >>> During that time, I wouldn't need to lock-down access to the map since >>> the forked process would have its own copy of the memory >>> I pretty much wanted to do what Redis does, but in Golang, and for a >>> multi-threaded process which I now learned is not really possible (please >>> correct me if I'm wrong I'd be happy). >>> >>> So, with that failed idea behind me, I'm looking at how I could >>> accomplish this in Golang while being memory efficient. >>> >>> I'm opened to a lot of ideas, but I'd consider myself intermediate in >>> Golang and beginner in C (if the answer involves CGO) so I might need >>> detailed examples if we get in deep low level stuff. >>> >>> I'd also like to have something that is pure Golang code since I'll be >>> cross compiling this on multiple architectures using the go compiler. >>> >>> Obviously, if more details are needed to answer the question, I'd be >>> happy to provide them. >>> >>> Thanks in advance! >>> >>> - Julien >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.