http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Import_Statement_Overhead has some good information.
On Aug 29, 5:33 pm, Robert Bradshaw <rober...@math.washington.edu> wrote: > On Mon, Aug 29, 2011 at 4:21 PM, William Stein <wst...@gmail.com> wrote: > > On Mon, Aug 29, 2011 at 2:56 PM, Maarten Derickx > > <m.derickx.stud...@gmail.com> wrote: > >> At the last sage-days there were some people working on making sage start > >> up > >> faster. I think it would be nice to have a short piece in the documentation > >> about "best practices for importing" so that all sage developers have a > >> reference point on how to deal with these difficulties. > >> Before I can write down such a piece I would like to have heard the opinion > >> of several sage developers on this. In particular it would be nice to have > >> hear from all the people who have been thinking about these problems verry > >> hard on the last sage days what causes of slowdown they found and how to > >> work around these causes, the more general the case the better of course. > >> Also feel free to give your opinion if you weren't at the last sage days! > >> I will start with giving my opinion on one of the causes of slowdown. I > >> heard from William that during sage startup there is a huge amount of > >> filesystem acces going on, this causes a very bad startup time behaviour if > >> you startup sage from a slow filesystem. I heard him complaining that there > >> where a lot of useless checks if certain files existed before they where > >> loaded (a lot of looking in different places and finding nothing). I think > >> the reason for this is that we use way to many absolute imports. Importing > >> stuff from the top level as: > >> import sage.some_module.some_submodule > >> will search for stuff in a lot of different places (this is also what > >> William observed IIRC). > >> A quote from: > >>http://docs.python.org/tutorial/modules.html#intra-package-references > > >> The submodules often need to refer to each other. For example, > >> the surround module might use the echo module. In fact, such references are > >> so common that the import statement first looks in the containing package > >> before looking in the standard module search path. > > >> So it seems to me we should make more use of this. And also more use of the > >> explicit relative import statements described there. > > >> from . import echo > >> from .. import formats > >> from ..filters import equalizer > > >> I also have the feeling that a part of the slowness comes from our custom > >> all.py import construction and not using the `__all__` construction in > >> __init__.py as described on > >>http://docs.python.org/tutorial/modules.html#importing-from-a-package > > >> altough I have to explicitly test that to check wether this has something > >> to > >> do with it. > >> To give you a feeling on how much none relative imports occur on the top > >> level of files and to proof that this is really an issue see the output of > >> the following commands in the $SAGE_ROOT/devel/sage directory > >> egrep -r "^from sage." ./ | nl > > >> gives 5312 absolute sage imports! > >> and: > >> egrep -r "^import sage." ./ | nl > >> another 893. Some of these are of course imported multiple times, so not > >> all > >> of them cause trouble but it at least shows that this is really an issue. > > > I don't think avoiding absolute imports is the right solution to this > > problem at all. Much better is the code that Volker Braun wrote that > > caches import locations between Sage runs. Basically, his code > > uniformly solves the above problem once and for all without requiring > > any unnatural changes at all to the Sage library. > > +1. This might help some, but is a lot more work for a lot less > benefit than the "cached import locations" hack. Also I find absolute > imports easier to read in many cases. > > >> Another two things I think that should be avoided as much as possible are: > >> 1. Code execution in modules, a module should have mainly function and > >> class > >> declarations, executing code makes the startup time longer. Explicit > >> examples of where this happend, and how a workaround was found are > >> appriciated. > > +1 > > >> Maybe we should make a lazy decorator decorator (decorating a > >> function also causes code execution) so that the execution of decorators > >> which have an expensive initialization can be delayed (it depends on how > >> much time is spend in decoration right now if it's worth the effort). > > Decorators are usually quite cheap, so I don't think this is a concern > here. On that note, however, the cached_method/function decorator > should be pointed out as an alternative to pre-computing things in the > module global space. > > > > > > > > > > > I remember trying to referee one purported example of this, which > > involved elliptic curve isogenies and big precomputed polynomials... > > >> 2. using > >> from some_module import some_function, some_class > >> this should mainly be avoided because of problems with circular imports. > >> when doing imports you are not sure whether some_module has been > >> initialized > >> yet so this could give problems. For the same reason the following should > >> be > >> avoided: > >> import some_module > >> some_module.some_function() > >> since you are not sure wether some_module is initialized yet. Even if you > >> are sure that some_module will be initialized it will make the whole sage > >> liberary harder to maintain since removing "unused" import statements from > >> other files might change the way modules are initialized an give unexpected > >> import errors. The following however is ok to do: > >> from module import sub_module > >> Since this will never lead to circular import problems on its own (if > >> sub_module was not initialized yet the above statement will initialize > >> sub_module, however if you use it to import an explicit class of function > >> it > >> might lead to problems read for example > >>http://effbot.org/zone/import-confusion.htmfor a better explanation). > > There are pros and cons to this. In particular, if you have "import > sage.rings.all" and then in your code you use "sage.rings.all.ZZ" you > are forcing 4 lookups per access. It also makes when the module in > question is actually imported much harder to track down. Of course > there are valid usecases (e.g. when circular imports are necessary). > > >> Ps. maybe we should also change our custom all.py initialization method and > >> use the __init__.py method. This will make it possible in the future to > >> only > >> import certain sub modules of sage. > > > It is unfortunately naive to think that replacing all.py's by > > __init__.py will magically "make it possible to only import certain > > sub modules of sage". > > I've been looking into using __init__.py files to clean things up, but > not necessarily (at least not at this point) getting rid of the all.py > files. In particular, having __init__.py import * from all.py prevents > one from cherry-picking submodules without proper initialization of > their context, which is one of the reasons sage startup and changing > imports is so fragile. Though it wouldn't help with startup time per > se, it would I think make things much easier to follow and debug (e.g. > as part of a greater strategy to both simplify and streamline things). > > > > > > > > >> For example if I was bothered by the > >> startup time from sage, and I only want to do stuff with rings this would > >> make it possible to make an optional argument for sage such that > >> sage -import rings > >> would only load the rings submodule and it's dependancies. > >> Of course we could also do that right now with or all.py files, but it > >> would > >> be good if we obeyed the python standards. (does anybody now why we use > >> all.py and not __init__.py for initialization?) > >> Ok. this became a longer post than I anticipated, but I still hope on > >> feedback from others :) > > > It's great that you've raised this issue for discussion, and I hope > > this thread turns out to be long. > > > Even after banging on startup time, etc., a lot last week, I > > personally do not know what the "best practices" should be for imports > > in Sage. There are pro's and con's with several things you suggest > > above. > > > -- William > > > -- > > To post to this group, send an email to sage-devel@googlegroups.com > > To unsubscribe from this group, send an email to > > sage-devel+unsubscr...@googlegroups.com > > For more options, visit this group > > athttp://groups.google.com/group/sage-devel > > URL:http://www.sagemath.org -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
