Re: OpenBLAS and performance

2017-12-21 Thread Pjotr Prins 
On Thu, Dec 21, 2017 at 12:02:55AM +0100, Ricardo Wurmus wrote:
> 
> Pjotr Prins  writes:
> 
> > On Wed, Dec 20, 2017 at 09:00:46PM +0100, Ricardo Wurmus wrote:
> >> > I do think we need to default to a conservative openblas for general
> >> > use. Question is how we make it fly on dedicated hardware.
> >>
> >> Have you tried preloading the special library with LD_PRELOAD?
> >
> > It is not a question of what I can do. It is a question of how we give
> > other people the benefit of optimized libs in an elegant way.
> 
> I’m asking because preloading different BLAS libraries is a thing.  If
> this works then we can ask people to just pick their favourite BLAS
> library variant and preload it.  We don’t need to build all combinations
> of library variants and applications.

Ah, sorry, I misunderstood. Let me play with that a little. Do note
that it is a bit more complex than it looks. For example, often you
need a cblas API. This comes with gslclas and openblas built-in. So,
to use atlas you also need libgslcblas.so. With optimized openblas you
don't.

> > I think channels actually should make a difference if we don't cater
> > in default Guix for such use cases.
> 
> I think channels (and really: alternative build farms) do make sense for
> tuned builds.

+1. The use case is simply:

  guix channel pjotr-optimized-haswell
  guix package -i python-numpy

which would install my optimized edition.

Pj.

Re: Building from a local source code checkout

2017-12-21 Thread Pjotr Prins 
On Thu, Dec 21, 2017 at 07:38:53AM +0100, Konrad Hinsen wrote:
> Perhaps I should stop trying to understand computers ;-)

We are waiting for AI to start programming ;)

Pj.
--

Re: Building from a local source code checkout

2017-12-21 Thread Martin Castillo 
Hi,


> Oops, you are right! So
> 
>guix build python-activepapers 
> --with-source=$HOME/Development/python-activepapers/
> 
> actually works, but
> 
>guix build python-activepapers 
> --with-source=$HOME/Development/python-activepapers
> 
As one error message earlier in the thread stated, guix uses lstat to
get info about the stated file. lstat doesn't follow symlinks. You force
the dereference by appending the slash.

Martin
-- 
GPG: 7FDE 7190 2F73 2C50 236E  403D CC13 48F1 E644 08EC

Re: OpenBLAS and performance

2017-12-21 Thread Ludovic Courtès 
Pjotr Prins  skribis:

> On Wed, Dec 20, 2017 at 07:15:16PM +0100, Ricardo Wurmus wrote:

[...]

>> The FAQ document[1] says this:
>> 
>>   The environment variable which control the kernel selection is
>>   OPENBLAS_CORETYPE (see driver/others/dynamic.c) e.g. export
>>   OPENBLAS_CORETYPE=Haswell. And the function char*
>>   openblas_get_corename() returns the used target.
>> 
>> [1]: https://github.com/xianyi/OpenBLAS/wiki/Faq
>> 
>> Have you tried this and compared the performance?
>
> About 10x difference on 24+ cores for matrix multiplication (my
> version vs what comes with Guix).

Even when you use OPENBLAS_CORETYPE=haswell (lower-case?)?

That would be surprising: it’s the same code after all.  The only
difference should be what happens at load time.

Ludo’.

Re: OpenBLAS and performance

2017-12-21 Thread Ludovic Courtès 
Hello,

Dave Love  skribis:

> Fedora sensibly builds separately-named libraries for different flavours
> , but I'd
> argue also for threaded versions being available with the generic soname
> in librray sub-directories.  There's some discussion and measurements
> (apologies if I've referenced it before) at
> 

I like the idea of an ‘update-alternative’ kind of approach for
interchangeable implementations.

Unfortunately my understanding is that implementations aren’t entirely
interchangeable, especially for LAPACK (not sure about BLAS), because
BLIS, OpenBLAS, etc. implement slightly different subsets of netlib
LAPACK, AIUI.  Packages also often check for specific implementations in
their configure/CMakeLists.txt rather than just for “BLAS” or “LAPACK”.

FlexiBLAS, which Eric mentioned, looks interesting because it’s designed
specifically for that purpose.  Perhaps worth giving it a try.

Besides, it would be good to have a BLAS/LAPACK policy in Guix.  We
should at least agree (1) on default BLAS/LAPACK implementations, (2)
possibly on a naming scheme for variants based on a different
implementation.

For #1 we should probably favor implementations that support run-time
implementation selection such as OpenBLAS (or the coming BLIS release).

Thoughts?

Ludo’.

Re: emacs-25.3 aborted

2017-12-21 Thread Ludovic Courtès 
Hi Daniel,

Daniel Pimentel  skribis:

> I installed emacs with "--no-grafts" and emacs-exwm with this flag and
> it works.

Be aware that using --no-grafts means that the resulting Emacs lacks
many security fixes:

  https://www.gnu.org/software/guix/manual/html_node/Security-Updates.html

Thus it’s not a proper solution.

Ludo’.

Re: Building from a local source code checkout

2017-12-21 Thread Ludovic Courtès 
Hi,

Martin Castillo  skribis:

>> Oops, you are right! So
>> 
>>guix build python-activepapers 
>> --with-source=$HOME/Development/python-activepapers/
>> 
>> actually works, but
>> 
>>guix build python-activepapers 
>> --with-source=$HOME/Development/python-activepapers
>> 
> As one error message earlier in the thread stated, guix uses lstat to
> get info about the stated file. lstat doesn't follow symlinks. You force
> the dereference by appending the slash.

Indeed.

I agree the matching-name constraint can be annoying.  The alternative
would be allow the user to (optionally?) specify the name of the package
whose source is being changed, as in:

  guix build python-activepapers \
--with-source=python-activepapers=$HOME/Dev/AP

WDYT?

A bit heavyweight syntactically but maybe we have to live with it.

Ludo’.

Re: OpenBLAS and performance

2017-12-21 Thread Dave Love 
Ricardo Wurmus  writes:

> Hi Pjotr,
>
>> I was just stating that the default openblas package does not perform
>> well (it is single threaded, for one).
>
> Is it really single-threaded?  I remember having a couple of problems
> with OpenBLAS on our cluster when it is used with Numpy as both would
> spawn lots of threads.  The solution was to limit OpenBLAS to at most
> two threads.

Yes, it's symlinked from the libopenblasp variant, which is linked
against libpthread, and I'd expect such problems.

Anyhow, there's something badly wrong if it doesn't perform roughly
equivalently to MKL on SIMD other than AVX512.  If I recall correctly,
the DGEMM single-threaded performance/core for HPC-type Sandybridge is
in the high 20s GFLOPs, and roughly double that for avx2
({Has,broad}well).  I don't think the bad L2 cache value that currently
used for Haswell has much effect in that case, but does in other
benchmarks.  I'll supply a patch for that.

Another point about the OB package is that it excludes LAPACK for some
reason that doesn't seem to be recorded.  I think that should be
included, partly for convenience, and partly because it optimizes some
of LAPACK.

Re: OpenBLAS and performance

2017-12-21 Thread Dave Love 
Eric Bavier  writes:

> Related only to this specific case of BLAS libraries, and not to the
> general idea of optimized libraries:

> I recently discovered "FlexiBLAS" from the Max Planck Institute
> https://www.mpi-magdeburg.mpg.de/projects/flexiblas which I thought
> might be useful for Guix.

That's a new one on me; I'll see how it works.  (You'd hope you could do
it with weak symbols or other ELFin stuff, but I couldn't see how.)

> It lets one choose the desired BLAS backend
> at runtime via a configuration file or environment variables.

The Fedora package I referenced also does that, makes it easy to have
local defaults on heterogeneous clusters, and has been used in
production.  The same technique allows you to use proprietary BLAS if
necessary.

> In it's
> current state it needs a little configuration before use, but I think
> with a little work we could make picking a BLAS implementation as easy
> as, e.g.
>
>   guix package -i python-numpy openblas-haswellp

Really, you shouldn't need to do that.
  
By the way, there's hope for free ~MKL-equivalent L3 BLAS on avx512 from
some work that's promised in the new year.  (BLIS dgemm currently has
~70% of MKL performance.)

Re: Building from a local source code checkout

2017-12-21 Thread Martin Castillo 
> I agree the matching-name constraint can be annoying.  The alternative
> would be allow the user to (optionally?) specify the name of the package
> whose source is being changed, as in:
> 
>   guix build python-activepapers \
> --with-source=python-activepapers=$HOME/Dev/AP
> 
> WDYT?
> 
> A bit heavyweight syntactically but maybe we have to live with it.
> 
> Ludo’.
> 

Why can't guix use the argument package name argument?
 v-- this one
guix build python-activepapers --with-source=$HOME/Dev/AP

Martin
-- 
GPG: 7FDE 7190 2F73 2C50 236E  403D CC13 48F1 E644 08EC

Re: Building from a local source code checkout

2017-12-21 Thread Konrad Hinsen 
l...@gnu.org (Ludovic Courtès) writes:

> I agree the matching-name constraint can be annoying.  The alternative
> would be allow the user to (optionally?) specify the name of the package
> whose source is being changed, as in:
>
>   guix build python-activepapers \
> --with-source=python-activepapers=$HOME/Dev/AP
>
> WDYT?

Looks much clearer to me. But, as you say, this could well be optional
if we can live with an ambiguity in the rare case of filenames containg
an equal sign.

Konrad.

Re: emacs-25.3 aborted

2017-12-21 Thread Daniel Pimentel 

Hi Ludo,


Be aware that using --no-grafts means that the resulting Emacs lacks
many security fixes:

  
https://www.gnu.org/software/guix/manual/html_node/Security-Updates.html


Thus it’s not a proper solution.


Yes, I agree because it isn't better solution. I'm using it now, but 
I'll study more to fix and I'll send a patch.


Thanks,

Re: OpenBLAS and performance

2017-12-21 Thread Ricardo Wurmus 

Dave Love  writes:

> Another point about the OB package is that it excludes LAPACK for some
> reason that doesn't seem to be recorded.  I think that should be
> included, partly for convenience, and partly because it optimizes some
> of LAPACK.

That was me, I think.  I did this because I assumed that if users want
LAPACK they’d just install the lapack package.  If this turns out to be
a misguided idea because the OB LAPACK differs then I’m fine with
enabling LAPACK in the OB package. 

(I’m not very knowlegdeable about all of this.  I just happened to
package OpenBLAS first.)

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: website: showcase or list videos and slides, sources of all talks

2017-12-21 Thread sirgazil 

On 20/12/17 17:32, Ludovic Courtès wrote:

Hartmut Goebel  skribis:


Am 19.12.2017 um 17:15 schrieb ng0:

It should be here:

   https://www.gnu.org/software/guix/blog/tags/talks/
   https://www.gnu.org/software/guix/blog/tags/papers/

This doesn't work really well to discover them, like on
a submenu page in the head menu, leading to these pages
or dedicated (non-blog, non-markdown) pages.


+1 for linking them from a more prominent page on the main website.

Blogs are like a diary, but not good for documentation an important content.


sirgazil, WDYT?  :-)


Well, I think we could convert the current Blog item in the website menu 
into a Media item with the following subitems:


  Blog
  Papers
  Screenshots
  Talks

Just like screenshots and blog posts, some talks could be highlighted in 
the home page. These changes could make it easier for people to find 
them, I think.


Note that video preview/thumbnail images will have to be created manually.

Mark H Weaver says:

---
For now, an easy improvement would be to include links to the above URLs
from our Help page.  However, I also found the old list of talks and
papers to be easier to browse than the new method, and I wonder if it
could be restored.

For example, for those who prefer to see a video, it was easy to see at
a glance on the old site which talks included videos, whereas on the new
site one must apparently click on each individual talk to find out which
ones come with videos.  Ditto for slides.
---

We can design the new pages and update current pages so that people 
browsing the summaries of posts, talks, etc. know if the full content of 
a given item contains video, slides, etc.


With these changes, which I think I can start designing right now, I 
think it would not be necessary to list talks and papers in the Help 
section again. What do you think, Mark?

Re: OpenBLAS and performance

2017-12-21 Thread Ricardo Wurmus 

Hi Dave,

> I wrote:
>
>> If you do provide some sort of threaded version for Python, then as far
>> as I remember it must use pthreads, not OpenMP, though you want the
>> OpenMP version for other purposes, and I hadn't realized there wasn't
>> one currently.
>
> I was confused.  I see the only version of the library shipped is built
> with pthreads.  I think there should be serial, pthreads, and OpenMP
> versions, as for Fedora.

Do these library variants have the same binary interface, so that a user
could simply preload one of them to override the default variant we use
in the input graph of a given package?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net