Here are my thoughts on the matter, but I'm not an expert on the
innerworkings of sage, so please forgive/tell me if this is already
done/impossible.
I propose limiting the size of the cache of parents and keep track of the
references of parents. Thus parents with the fewest references should be
replaced from the cache once we reach the maximum. Additionally if a parent
has no references, we allow the garbage collector to take the parent. To
get around referenced parents being spontaneously deleted, every time we
return a parent object, we have a lightweight bridge class which recreates
the parents if they've been deleted when called (which also gets notified
when the parent is deleted). Something like this:
class ParentBridge:
def __init__(self, parent_class, data):
self._parent_class = parent_class
self._data = data # arguments passed to the parent
self._parent = None
def parent(self):
if self._parent is None:
self._create_parent()
return self._parent
def _create_parent(self):
# Do stuff to create parent in the cache
self._parent = # the created parent
def _parent_deleted(self):
self._parent = None
We also return the same ParentBridge when the parent is stored in the
cache. This would basically be a slight modification of a weak reference
which recreates the target object if it is invalid. Another variant is we
implement some other type of cache replacement algorithm (
http://en.wikipedia.org/wiki/Cache_algorithms).
Alternatively we could just allow parents with no references are allowed to
be garbage collected. This will likely not break any doctests since
checking parent identity is usually in successive lines and the garbage
collector usually does not have time to collect anything within a few lines
when doctesting. We might also want to add a flag for (very) select
instances which says that it can never be collected.
In both of the above, there is at most 1 instance of a given parent at any
one time, so I do not foresee any problems (as long as we can reconstruct
the parent object and appropriate references if it's deleted).
Nevertheless, how we implement this must minimally change the interface,
and I suspect the first way I suggested may require substantial change...
Best,
Travis
On Saturday, November 3, 2012 12:58:11 PM UTC-7, Nils Bruin wrote:
>
> Presently, Sage has a significant memory leak issue: Uniqueness of
> parents is currently guaranteed by keeping them in memory
> permanently.This prevents many computational strategies that are
> otherwise perfectly legitimate, but require the construction of, for
> instance, many finite fields and/or polynomial rings. A lot of
> arithmetic geometric constructions fall in that category and current
> approaches to tackle noncommutative algebra do too. Every single time
> I have tried to use sage for some significant computation, this has
> prevented me from completing it.
>
> There has been work on resolving this issue by replacing permanent
> storage by weak caching, so that parents can actually get delete; see
> tickets #715 and #11521, for instance. The code on these tickets is by
> now one of the most carefully reviewed (future) parts of sage.
> However, time and again, issues elsewhere crop up because there is
> broken code elsewhere that never got exercised because parents never
> got deleted, even though they should.
>
> We have been in shape a couple of times now, where all noticeable
> issues were resolved. However, the merger of *other* tickets brought
> to light even different issues, resulting in pulling #715 and #11521.
>
> If we ever want sage to be usable on a reasonable scale, parents need
> to be deleted every now and again. The basic design allows them to.
> It's just that there is a lot code in sage that breaks when that
> actually happens. Apparently, the normal ticket review and merger
> process is not suitable for a change that's so fundamental to sage's
> infrastructure, because it favours small-scale and superficial patches
> (and hence keeps moving the goal posts for infrastructure changes).
> Any ideas how to get this done?
> For me this is a must-have to consider sage as viable platform and I
> suspect I am not the only one for which it is.
>
> Cheers,
>
> Nils
>
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To post to this group, send email to sage-devel@googlegroups.com.
To unsubscribe from this group, send email to
sage-devel+unsubscr...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel?hl=en.
