On Mon, Aug 29, 2011 at 2:56 PM, Maarten Derickx <m.derickx.stud...@gmail.com> wrote: > At the last sage-days there were some people working on making sage start up > faster. I think it would be nice to have a short piece in the documentation > about "best practices for importing" so that all sage developers have a > reference point on how to deal with these difficulties. > Before I can write down such a piece I would like to have heard the opinion > of several sage developers on this. In particular it would be nice to have > hear from all the people who have been thinking about these problems verry > hard on the last sage days what causes of slowdown they found and how to > work around these causes, the more general the case the better of course. > Also feel free to give your opinion if you weren't at the last sage days! > I will start with giving my opinion on one of the causes of slowdown. I > heard from William that during sage startup there is a huge amount of > filesystem acces going on, this causes a very bad startup time behaviour if > you startup sage from a slow filesystem. I heard him complaining that there > where a lot of useless checks if certain files existed before they where > loaded (a lot of looking in different places and finding nothing). I think > the reason for this is that we use way to many absolute imports. Importing > stuff from the top level as: > import sage.some_module.some_submodule > will search for stuff in a lot of different places (this is also what > William observed IIRC). > A quote from: > http://docs.python.org/tutorial/modules.html#intra-package-references > > The submodules often need to refer to each other. For example, > the surround module might use the echo module. In fact, such references are > so common that the import statement first looks in the containing package > before looking in the standard module search path. > > So it seems to me we should make more use of this. And also more use of the > explicit relative import statements described there. > > from . import echo > from .. import formats > from ..filters import equalizer > > I also have the feeling that a part of the slowness comes from our custom > all.py import construction and not using the `__all__` construction in > __init__.py as described on > http://docs.python.org/tutorial/modules.html#importing-from-a-package > > altough I have to explicitly test that to check wether this has something to > do with it. > To give you a feeling on how much none relative imports occur on the top > level of files and to proof that this is really an issue see the output of > the following commands in the $SAGE_ROOT/devel/sage directory > egrep -r "^from sage." ./ | nl > > gives 5312 absolute sage imports! > and: > egrep -r "^import sage." ./ | nl > another 893. Some of these are of course imported multiple times, so not all > of them cause trouble but it at least shows that this is really an issue.
I don't think avoiding absolute imports is the right solution to this problem at all. Much better is the code that Volker Braun wrote that caches import locations between Sage runs. Basically, his code uniformly solves the above problem once and for all without requiring any unnatural changes at all to the Sage library. > Another two things I think that should be avoided as much as possible are: > 1. Code execution in modules, a module should have mainly function and class > declarations, executing code makes the startup time longer. Explicit > examples of where this happend, and how a workaround was found are > appriciated. Maybe we should make a lazy decorator decorator (decorating a > function also causes code execution) so that the execution of decorators > which have an expensive initialization can be delayed (it depends on how > much time is spend in decoration right now if it's worth the effort). I remember trying to referee one purported example of this, which involved elliptic curve isogenies and big precomputed polynomials... > 2. using > from some_module import some_function, some_class > this should mainly be avoided because of problems with circular imports. > when doing imports you are not sure whether some_module has been initialized > yet so this could give problems. For the same reason the following should be > avoided: > import some_module > some_module.some_function() > since you are not sure wether some_module is initialized yet. Even if you > are sure that some_module will be initialized it will make the whole sage > liberary harder to maintain since removing "unused" import statements from > other files might change the way modules are initialized an give unexpected > import errors. The following however is ok to do: > from module import sub_module > Since this will never lead to circular import problems on its own (if > sub_module was not initialized yet the above statement will initialize > sub_module, however if you use it to import an explicit class of function it > might lead to problems read for example > http://effbot.org/zone/import-confusion.htm for a better explanation). > Ps. maybe we should also change our custom all.py initialization method and > use the __init__.py method. This will make it possible in the future to only > import certain sub modules of sage. It is unfortunately naive to think that replacing all.py's by __init__.py will magically "make it possible to only import certain sub modules of sage". > For example if I was bothered by the > startup time from sage, and I only want to do stuff with rings this would > make it possible to make an optional argument for sage such that > sage -import rings > would only load the rings submodule and it's dependancies. > Of course we could also do that right now with or all.py files, but it would > be good if we obeyed the python standards. (does anybody now why we use > all.py and not __init__.py for initialization?) > Ok. this became a longer post than I anticipated, but I still hope on > feedback from others :) It's great that you've raised this issue for discussion, and I hope this thread turns out to be long. Even after banging on startup time, etc., a lot last week, I personally do not know what the "best practices" should be for imports in Sage. There are pro's and con's with several things you suggest above. -- William -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
