Thanks for explaining it in detail. I used to have to chase large amounts of memory leaks and your discussion reminded me of that complex chase every two to three months which had to fit in our software release cycles.
Sent from my iPhone > On Dec 5, 2018, at 3:50 AM, Simon King <simon.k...@uni-jena.de> wrote: > > Hi Steven, > >> On 2018-12-05, Steven Craighead <steven.craigh...@gmail.com> wrote: >> How difficult is it to create a stack that can control the order of objects >> being created and destroyed so you prevent leaks? Can you add a new method >> on your base class that is inherited to all children to track this? > > One basic principle in the development of Sage is: "Do not reinvent the > wheel, but build the car." > > So, we use mainstream programming languages (Python for user > interface, Python and Cython for the Sage library), and we use > existing software packages, partly from the Python ecosystem, > partly stand-alone, such as Singular. These third-party packages can of > course be written in all kinds of languages, such as fortran, C++, Java, > etc. > > Therefore, memory leaks could have three causes: > > 1. A leak in a third-party package. In that case, we send a bug report to > upstream, possibly fixing it by a downstream patch until the bug is > fixed upstream. > 2. A bug in Python's cyclic garbage collection. I don't know if we ever > stumbled over such bug, but in principle it is a possible cause. > 3. Constructions in the Sage library that prevent Python's cyclic > garbage collection from properly working. > > This thread is about yet another instance of cause 3, or rather about a > strategy to generally prevent instances of cause 3. > > What prevents cyclic gc from working? > - If there is a reference cycle involving one instance with a __del__ > method, then Python would not apply garbage collection to that cycle. > If I recall correctly, there has been a Sage memory leak in the past where > the existence of a __del__ method was the underlying cause. > - For performance reason, Sage does a lot of caching, and the caching > involves objects that typically have lots of cross references. See, > for example, the coercion system. Note that coercion in Sage is not > the same as coercion in C. A coercion in Sage, in first approximation, > is a canonical morphism in a mathematical category. If caching is done > improperly, then there may be some external reference to a reference > cycle, which of course means that Python wouldn't (and shouldn't) > garbage collect that cycle. > > In order to prevent external strong references to a reference cycle, we > often use weak references for caching. > But that needs to be done with care: If there are two many weak > references, then stuff may be garbage collected although we wanted it to > be cached, and also following a weak references is slower than following > a strong reference; but using too few weak references creates a memory > leak. > > I believe it would be an extremely bad idea to try and implement our own > cyclic garbage collection, disabling Python's. > > Best regards, > Simon > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com. > To post to this group, send email to sage-devel@googlegroups.com. > Visit this group at https://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
