Dear Ana,
it is easy to ask the question (and I've been asked several times), but
somewhat difficult to answer. To add to Graeme's excellent explanations:
- all developers of MX processing software have seriously considered to
implement their algorithms on GPUs, and have decided that the effort (which is
very significant) is not worth it, in terms of benefit for users and developers
(who should pay them for the effort? - after all, this would not result in a
highly cited publication!). We are aware of the fact that they are much faster
than CPUs for specific types of calculations that are most useful for images
where each pixel is treated in the same way - but these types of (potentially
highly parallel) calculations do not represent a large fraction of where a MX
data processing program spends its time, and even worse, the parallel and
serial parts of the calculation alternate in fast succession (XDS has on the
order of 10 parallel regions, none of which dominates the CPU time).
Ultimately, it is the serial fraction of a program that determines its
potential speed-up, due to Amdahl's law.
- MX data processing programs (at least XDS and DIALS) already exploit
parallelism by using multiple CPUs at the same time; the current version of XDS
can in principle use up to 99*99=9801 processors, and 60 machines each running
60 threads (see below) would process a 360° dataset composed of 0.1° frames
within seconds, if DELPHI=6.
- the recent Ryzen Threadripper 3XXX series CPUs have a significantly better
cost/performance ratio than other processor families. A TR 3970X workstation
can be bought for less than 5000€, and offers 64 threads. Graeme mentions AMD
Rome; this is the server variant. The data transfer would become the
bottleneck; to me, a cluster of workstations each equipped with two 10Gb ports
looks attractive.
Finally, I have the feeling that speed in data collection and processing is
over-rated. I get the impression that some (many?) people think they should
collect a data set as quickly as the machine permits. But they may not be aware
of the fact that the quality of the data is then not optimal. Going 10 times
slower, and reducing the transmission to 10%, gives a resonable safety margin.
Further questions arise - does every crap crystal have to be put into the beam?
And does every crap data set have to be processed? Do all of us really want and
need to collect from thousands of crystals every synchrotron day? Are all of us
really producing that many crystals? Who is? (you probably realize my lack of
imagination by now)
I know that people who build and run synchrotron beamlines have a different perspective,
concerning these questions, than their users. Some common sense, and a lot of discussion,
would benefit our community more than resorting to technological "solutions".
best wishes,
Kay
On Wed, 19 Feb 2020 08:08:40 +0000, Winter, Graeme (DLSLtd,RAL,LSCI)
<graeme.win...@diamond.ac.uk> wrote:
Dear Ana,
To follow up on the contributions from others, there are some particular
annoyances with MX processing which differentiate it from other “big data” or
imaging problems.
In tomographic reconstruction you have a big block of data which needs to (as a
simplistic approximation) be transformed by a bunch of trigonometric functions
to another big block of data. The shape of the calculation is the same
independent of the data itself, and overall this represents a massively
parallel computationally expensive problem, which makes it worth the cost of
getting the data in and out of the GPU (this is not cheap) - even in this case,
the parallelism of modern CPUs means that this is not a given. These folks are
usually the ones who are making a lot of noise about how awesome GPU boards
are, and for their use case this is absolutely true.
In MX we have a particularly annoying problem, as about half of the
calculations are nicely parallel (spot finding, peak integration) and are
memory bandwidth / CPU breadth limited and the other half (indexing,
refinement, scaling) are not very parallel CPU speed bound, so finding the best
CPU architecture is hard to start with. In terms of GPU, the data need to
typically pass through main memory three times - for spot finding you need to
look at every pixel, and integration typically needs to load full frames to
extract the profiles and then fit them (the shoebox regions can be cached
between these, but they still need to pass in and out of the CPU). Since moving
data in and out of memory is expensive and GPU memory is expensive this is a
problem. For reference, a typical Eiger 16M data set uncompressed needs about
half a terabyte of RAM (7,200 * 18 megapixels * 4 bytes) so in memory
processing presents real challenges. The image analysis calculations themselves
are typically rather light weight floating point work (e.g. summed area table
calculations) without a lot of trigonometry.
All this, combined with the annoying habit of using words like “if” and “for”
in the code (which kills GPU calculations dead) mean that even for spot finding
it’s not worth the effort of moving the data into a GPU - we DIALS folks looked
into this a couple of years back with a specialist from NVIDIA.
For what it’s worth we have spent some time looking at this here at Diamond,
where we have a certain interest in speedy processing of MX data and the
current (2020/02) best bang for buck appears to be AMD Rome.
We as a community have a challenge with keeping up with high data rate
beamlines at both synchrotrons and FELs - I feel it is important to keep an eye
on emerging technology and make best use of it (and share experiences of using
it!) but we should also keep in mind that the processing done in MX is actually
rather well defined and mathematical at its heart. It is very unlikely that
deep learning will help with the mathematical challenges we face [1] as we know
exactly the calculations we need to do (which are very well documented in the
literature, thank you to everyone who has written these up over the years) and
instead a clear focus on making the maths fast is needed.
Up to the point where someone comes up with a completely new way of looking at
the data, of course. I’m sure someone out there is looking at this :-)
On the topic of raspberry pi machines ;-) these are fun but I would hate to
look at the interconnect necessary to get enough boards to work together to
keep up with a single AMD Rome box…
best wishes Graeme
[1] with the possible exception of classifying individual found spots and other
niche areas
On 19 Feb 2020, at 07:04, Leonarski Filip Karol (PSI)
<filip.leonar...@psi.ch<mailto:filip.leonar...@psi.ch>> wrote:
Dear Ana,
To benefit from GPU architecture, over CPU, the algorithm needs to do quite
significant number crunching – i.e. do at least certain number of floating
point operations (FLOP) per one byte of data. It also needs to be highly
parallel, preferably without conditional (if/else) statements. Finally, there
is a variety of GPU architectures on the market and it is not exactly obvious
that code written for one GPU will be optimal on another one. So if the code is
based on a general purpose library, it will be easier to make sure that it runs
efficiently on all GPU hardware.
I believe combination of these factors makes a big difference between imaging
and MX.
Imaging processing is limited by FFT performance, which needs floating point
performance. Libraries for FFT on GPUs are standard and provided by hardware
vendors, so it is easy to implement.
On the other hand MX algorithms for image processing, at least the one I know
of, do only handful of FLOP per pixel and they will probably not benefit from
GPU processing significantly, even if ported to such architecture – which would
be also a non-negligible effort. So while it is not impossible to imagine
GPU-accelerated MX software and hopefully people are working on this, it is not
a low hanging fruit, like in case of GPU acceleration for imaging or cryo-EM.
On a side note if one could find a way to use machine learning for data
processing and implement data processing pipeline in Tensorflow, then GPUs
would pay off quickly.
Regarding Tim’s Raspberry Pi argument – it should be compared with Nvidia
Jetson price, which is more or less RPi with GPU, and it won’t be actually that
significant difference.
Best,
Filip
From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>> on behalf
of Ana Carolina de Mattos Zeri <ana.z...@lnls.br<mailto:ana.z...@lnls.br>>
Reply to: Ana Carolina de Mattos Zeri
<ana.z...@lnls.br<mailto:ana.z...@lnls.br>>
Date: Tuesday, 18 February 2020 at 20:58
To: "CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>"
<CCP4BB@JISCMAIL.AC.UK<mailto:CCP4BB@JISCMAIL.AC.UK>>
Subject: [ccp4bb] MX data processing with GPUs??
Dear all
we have asked this of a few people, but the question remains:
does any of you have experienced/tried using GPU based software to treat MX
data? for reducing or subsequent image analysis?
is it a lost battle?
how do you deal with the crescent amount of data we are facing, at Synchrotrons
and XFELs?
Here at the Manaca beamline at Sirius we will continue to support CPU based
software, but due to developments in the imaging beam lines, GPU machines are
looking very attractive.
many thanks in advance for your thoughts,
all the best
Ana
Ana Carolina Zeri, PhD
Manaca Beamline Coordinator (Macromolecular Micro and Nano Crystallography)
Brazilian Synchrotron Light Laboratory (LNLS)
Brazilian Center for Research in Energy and Materials (CNPEM)
Zip Code 13083-970, Campinas, Sao Paulo, Brazil.
(19) 3518-2498
www.lnls.br<http://www.lnls.br/>
ana.z...@lnls.br<mailto:ana.z...@lnls.br>
Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais
e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual
consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você
recebeu esta mensagem por engano, por favor avise o remetente e apague-a
imediatamente.
Disclaimer: This email and its attachments may contain confidential and/or
privileged information. Observe its content carefully and consider possible
querying to the sender before copying, disclosing or distributing it. If you
have received this email by mistake, please notify the sender and delete it
immediately.
________________________________
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
________________________________
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
--
This e-mail and any attachments may contain confidential, copyright and or
privileged material, and are for the use of the intended addressee only. If you
are not the intended addressee or an authorised recipient of the addressee
please notify us of receipt by returning the e-mail and do not use, copy,
retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not
necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments
are free from viruses and we cannot accept liability for any damage which you
may sustain as a result of software viruses which may be transmitted in or with
the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and
Wales with its registered office at Diamond House, Harwell Science and
Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCP4BB&A=1
