On 04/23/2015 03:36 AM, Volker Braun wrote: > I wouldn't want another environment variable for that.
Well, easier for playing than editing the sources... ;-) A good default value should be chosen automatically. > If we have to, > chunking at 10 would be fine. I think it's ok for machines with 2 to 4 GB (saw ~0.5 +/-0.1 GB with one process); if one uses more cores anyway, there's not really a problem (thumb rule 2 GB RAM / core), i.e., up to "unlimited" everything would be ok (until we add the next few hundred modules). > Though ideally this would be dealt with on > the cython side, I think the Sage library is kind of special, but a 10-fold increase in memory usage (with a single process) can't really be what upstream wants or should tolerate. There's presumably a slight gain through caching a lot, but apparently too much is kept, so I'd consider it a bug. -leif > I made a post here: > https://groups.google.com/d/msg/cython-users/7UoMj84Y5os/6UAXi64MA88J > > > On Wednesday, April 22, 2015 at 6:04:41 PM UTC-4, leif wrote: > > On 04/21/2015 11:57 PM, Andrey Novoseltsev wrote: > > It is a single core single thread 64-bit CPU (Athlon 64M), > suggestions > > to use USB 3/PCIe SSD are funny but miss the point: for no > apparent gain > > systems that were capable of building Sage can't do it anymore (or > > require tweaking to reduce parallelism). I find it quite worrisome > since > > the situation may continue to worsen if not fixed. Unfortunately, I > > personally can't do much as I have no clue what is going on and > how to > > debug it. > > Tentative patch (tested with Sage 6.6): > > diff --git a/src/setup.py b/src/setup.py > index 82bc8dd..c642242 100644 > --- a/src/setup.py > +++ b/src/setup.py > @@ -553,14 +553,40 @@ def run_cythonize(): > force = False > > global ext_modules > - ext_modules = cythonize( > - ext_modules, > - nthreads=int(os.environ.get('SAGE_NUM_THREADS', 0)), > - build_dir='build/cythonized', > - force=force, > - compiler_directives={ > - 'profile': profile, > - }) > + > + nthreads = int(os.environ.get('SAGE_NUM_THREADS', 0)) > + nmodules = int(os.environ.get('SAGE_CYTHONIZE_AT_ONCE', 0)) # 0 > means all / unlimited > + build_dir = 'build/cythonized' > + compiler_directives = { > + 'profile': profile, > + } > + > + > + if not nmodules: > + # cythonize all modules at once > + ext_modules = cythonize( > + ext_modules, > + nthreads=nthreads, > + build_dir=build_dir, > + force=force, > + compiler_directives=compiler_directives > + ) > + else: > + # cythonize module list in chunks of nmodules modules > + chunk_size = nmodules > + cythonized_modules = [] > + > + for i in range(0, len(ext_modules), chunk_size): > + cythonized_modules.extend( > + cythonize( > + ext_modules[i:i+chunk_size], > + nthreads=nthreads, > + build_dir=build_dir, > + force=force, > + compiler_directives=compiler_directives > + ) > + ) > + ext_modules = cythonized_modules > > open(version_file, 'w').write(version_stamp) > > > > One can try this with e.g. > > env SAGE_CYTHONIZE_AT_ONCE=10 ./sage -ba-force > > or > > export SAGE_CYTHONIZE_AT_ONCE=25 > make > > Setting SAGE_NUM_THREADS is orthogonal to setting > SAGE_CYTHONIZE_AT_ONCE; I originally wanted to (ab)use the former to > use > some randomly chosen small chunk size if its value is zero, but within > Sage it's always positive and this way is more flexible. > > If we really want to merge such a work-around, the new environment > variable (better names appreciated) needs to be documented, and we > should probably add some magic to automatically cythonize in chunks if > RAM or total memory is low, probably also based on the number of > cores / > the setting of SAGE_NUM_THREADS. > > > Have fun, > > -leif -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
