I wouldn't want another environment variable for that. If we have to,
chunking at 10 would be fine. Though ideally this would be dealt with on
the cython side, I made a post here:
https://groups.google.com/d/msg/cython-users/7UoMj84Y5os/6UAXi64MA88J
On Wednesday, April 22, 2015 at 6:04:41 PM UTC-4, leif wrote:
>
> On 04/21/2015 11:57 PM, Andrey Novoseltsev wrote:
> > It is a single core single thread 64-bit CPU (Athlon 64M), suggestions
> > to use USB 3/PCIe SSD are funny but miss the point: for no apparent gain
> > systems that were capable of building Sage can't do it anymore (or
> > require tweaking to reduce parallelism). I find it quite worrisome since
> > the situation may continue to worsen if not fixed. Unfortunately, I
> > personally can't do much as I have no clue what is going on and how to
> > debug it.
>
> Tentative patch (tested with Sage 6.6):
>
> diff --git a/src/setup.py b/src/setup.py
> index 82bc8dd..c642242 100644
> --- a/src/setup.py
> +++ b/src/setup.py
> @@ -553,14 +553,40 @@ def run_cythonize():
> force = False
>
> global ext_modules
> - ext_modules = cythonize(
> - ext_modules,
> - nthreads=int(os.environ.get('SAGE_NUM_THREADS', 0)),
> - build_dir='build/cythonized',
> - force=force,
> - compiler_directives={
> - 'profile': profile,
> - })
> +
> + nthreads = int(os.environ.get('SAGE_NUM_THREADS', 0))
> + nmodules = int(os.environ.get('SAGE_CYTHONIZE_AT_ONCE', 0)) # 0
> means all / unlimited
> + build_dir = 'build/cythonized'
> + compiler_directives = {
> + 'profile': profile,
> + }
> +
> +
> + if not nmodules:
> + # cythonize all modules at once
> + ext_modules = cythonize(
> + ext_modules,
> + nthreads=nthreads,
> + build_dir=build_dir,
> + force=force,
> + compiler_directives=compiler_directives
> + )
> + else:
> + # cythonize module list in chunks of nmodules modules
> + chunk_size = nmodules
> + cythonized_modules = []
> +
> + for i in range(0, len(ext_modules), chunk_size):
> + cythonized_modules.extend(
> + cythonize(
> + ext_modules[i:i+chunk_size],
> + nthreads=nthreads,
> + build_dir=build_dir,
> + force=force,
> + compiler_directives=compiler_directives
> + )
> + )
> + ext_modules = cythonized_modules
>
> open(version_file, 'w').write(version_stamp)
>
>
>
> One can try this with e.g.
>
> env SAGE_CYTHONIZE_AT_ONCE=10 ./sage -ba-force
>
> or
>
> export SAGE_CYTHONIZE_AT_ONCE=25
> make
>
> Setting SAGE_NUM_THREADS is orthogonal to setting
> SAGE_CYTHONIZE_AT_ONCE; I originally wanted to (ab)use the former to use
> some randomly chosen small chunk size if its value is zero, but within
> Sage it's always positive and this way is more flexible.
>
> If we really want to merge such a work-around, the new environment
> variable (better names appreciated) needs to be documented, and we
> should probably add some magic to automatically cythonize in chunks if
> RAM or total memory is low, probably also based on the number of cores /
> the setting of SAGE_NUM_THREADS.
>
>
> Have fun,
>
> -leif
>
>
>
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
