On 04/21/2015 11:57 PM, Andrey Novoseltsev wrote:
> It is a single core single thread 64-bit CPU (Athlon 64M), suggestions
> to use USB 3/PCIe SSD are funny but miss the point: for no apparent gain
> systems that were capable of building Sage can't do it anymore (or
> require tweaking to reduce parallelism). I find it quite worrisome since
> the situation may continue to worsen if not fixed. Unfortunately, I
> personally can't do much as I have no clue what is going on and how to
> debug it.
Tentative patch (tested with Sage 6.6):
diff --git a/src/setup.py b/src/setup.py
index 82bc8dd..c642242 100644
--- a/src/setup.py
+++ b/src/setup.py
@@ -553,14 +553,40 @@ def run_cythonize():
force = False
global ext_modules
- ext_modules = cythonize(
- ext_modules,
- nthreads=int(os.environ.get('SAGE_NUM_THREADS', 0)),
- build_dir='build/cythonized',
- force=force,
- compiler_directives={
- 'profile': profile,
- })
+
+ nthreads = int(os.environ.get('SAGE_NUM_THREADS', 0))
+ nmodules = int(os.environ.get('SAGE_CYTHONIZE_AT_ONCE', 0)) # 0
means all / unlimited
+ build_dir = 'build/cythonized'
+ compiler_directives = {
+ 'profile': profile,
+ }
+
+
+ if not nmodules:
+ # cythonize all modules at once
+ ext_modules = cythonize(
+ ext_modules,
+ nthreads=nthreads,
+ build_dir=build_dir,
+ force=force,
+ compiler_directives=compiler_directives
+ )
+ else:
+ # cythonize module list in chunks of nmodules modules
+ chunk_size = nmodules
+ cythonized_modules = []
+
+ for i in range(0, len(ext_modules), chunk_size):
+ cythonized_modules.extend(
+ cythonize(
+ ext_modules[i:i+chunk_size],
+ nthreads=nthreads,
+ build_dir=build_dir,
+ force=force,
+ compiler_directives=compiler_directives
+ )
+ )
+ ext_modules = cythonized_modules
open(version_file, 'w').write(version_stamp)
One can try this with e.g.
env SAGE_CYTHONIZE_AT_ONCE=10 ./sage -ba-force
or
export SAGE_CYTHONIZE_AT_ONCE=25
make
Setting SAGE_NUM_THREADS is orthogonal to setting
SAGE_CYTHONIZE_AT_ONCE; I originally wanted to (ab)use the former to use
some randomly chosen small chunk size if its value is zero, but within
Sage it's always positive and this way is more flexible.
If we really want to merge such a work-around, the new environment
variable (better names appreciated) needs to be documented, and we
should probably add some magic to automatically cythonize in chunks if
RAM or total memory is low, probably also based on the number of cores /
the setting of SAGE_NUM_THREADS.
Have fun,
-leif
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
