I suspect this fact is going to be the most relevant thing to your investigation:
> Upgrading the sqlite3 driver which had one non-negligible change: adding SetFinalizer to all these objects. See https://go.dev/doc/gc-guide#Common_finalizer_issues for a variety of ways finalizers can cause leaks (on both the Go and C side). Go 1.24's cleanups (runtime.AddCleanup) might work be better, provided a deterministic execution order isn't required. (This point might be moot since it's in go-sqlite3, which sounds like it's something you don't have control over.) I have a patch that provides a finalizer/cleanup leak detector by setting GODEBUG=detectcleanupleaks=1, if you want to try it. It's https://go.dev/cl/634599. Happy to explain how to patch and build the Go toolchain if you're up for it. On Sunday, March 9, 2025 at 5:49:03 PM UTC-4 robert engels wrote: > Looks to me like you are reading a lot of rows under a lock, and never > releasing the lock, so the rows remain in memory. > > I don’t know the internals of the SQLite very well, but my understanding > is that it is not really a “driver” in the traditional sense that > communicates with a db - but rather it is the implementation as well. Since > it is the implementation, holding the lock seems reasonable to also hold > the rows. > > On Mar 9, 2025, at 2:59 PM, Gavra <gav...@gmail.com> wrote: > > Hi, > So we have been trying really hard to understand a major leak in our > product. > We are using this sqlite3 driver: https://github.com/mattn/go-sqlite3 > The pprof heap profile indicates the source is the call to > SQLiteRows.Columns func and two additional calls, one on SQLiteStmt and the > other on SQLiteConn. > According to the code > 1. SQLiteRows references SQLiteStmt which then references to SQLiteConn > 2. The SQLiteRows instance is the sole object holding a ref to the > allocation by Columns(). > This is a strong indication that refs to SQLiteRows are leaked. > We can confirm the leak is increasing over time and does not appear to > reflect a burst or large data. > We thought we forgot to close rows or somehow appened refs to rows etc but > we ruled it out completely since. Note the heap profile only shows SQLite > allocations, no app objects allocated. We verified that forgetting to call > SQLiteRows.Close leaks only memory in CGO which means it is not visible in > the heap profile. > So on one hand we are convinced someone is holding a reference to > SQLiteRows but on the other hand it is not our application and not the > SQLite driver. > > We tracked back our source code changes and noticed that correlate to the > appearance of this issue: > 1. Upgrading go: 1.22.6 to 1.22.9 > 2. Upgrading the sqlite3 driver which had one non-negligible change: > adding SetFinalizer to all these objects. > > This is a very weird thing to suggest, but we think this could be caused > by the go runtime, somehow. > I attached a screenshot of the heap profile, focused on the major leak > around the sqlite3 driver. > (our current plans is to use goref or dlv on a core dump to understand who > is holding these references but that could take a few more days). > > Thank you. > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts...@googlegroups.com. > To view this discussion visit > https://groups.google.com/d/msgid/golang-nuts/8a8b608b-e501-403a-b7a7-0d0bda657e4cn%40googlegroups.com > > <https://groups.google.com/d/msgid/golang-nuts/8a8b608b-e501-403a-b7a7-0d0bda657e4cn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > <Screenshot 2025-03-09 at 21.40.22.png> > > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/b5c236d6-1a90-4776-b33b-1234d0b58e02n%40googlegroups.com.
