Hi Ibe, gob is unsupported and not optimal in many ways.
I would invite you to try my serialization package. https://github.com/glycerine/greenpack You can serialize unexported fields if you wish to with the -unexported flag, though I generally don't. For the slow down, you might look at using non-quadratic approximate nearest neighbors methods like HNSW or faiss from facebook. Best, Jason On Sunday, June 22, 2025 at 1:21:52 AM UTC+2 Learned Byerror wrote: > All, > > I have an application that uses gonum/vptree and gonum/graph/simple. > Currently every run of the application instantiates the vptree from a > database, queries the vptree to find nearest neighbors, writes the data to > database table A, creates a weighted undirected graph from table A, > extracts k-clique communities from the graph and then generates SVGs for > each community. > > All of this works as expected and runs in about 7 minutes. The initial > version of the program ran in about 2 hours. I heavily used pprof to > identify optimization opportunities. I think the code is optimized as much > as is possible using the gonum packages. > > This program is run against a ever growing set of data. The current data > set has almost 3MM entities and grows by approximately 25K for each new > data set. As such, I expect the run time to continue increase. There are > two steps that take over 80% of the run time: vptree nearest neighbor > query, generation of the weighted undirected graph. I would like to be able > to save the state of the vptree and graph at the end of each run and use > that as input in the next run. > > Both gonum/vptree an gonum/graph/simple contain private fields. Neither > implement a GobEncode or GobDecode Interface. Consequently, I cannot use > encoding/gob. > > Please note, this is about saving and retrieving state in the same version > of the same program. It is not about transferring data from one program to > a different program. Neither of these packages have a dependency on > external state at run time. > > Are there alternative methods that I should consider? Approaches using > unsafe are acceptable to me. If something fails, I can alway recover by > making a full run as I am currently doing. > > The only alternative that I have come up with at this point if to make a > pull request and add the GobEncode/Decode functionality myself. While this > is an option, the effort required to do so is likely significant. > > Thank you in advance for your guidance! > > lbe > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/aa45c200-7f09-4727-89b2-b05cdfedcebbn%40googlegroups.com.
