Hi,
a dedicated/fast stand-alone workstation will be in most cases a better
choice for running single jobs than running on multiple machines as the
network capabilities are a limiting factor once all nodes try to access
the disk with the data (unless locally stored on each node on an
ultra-fast M.2 SSD). Obviously, if a central computing service is
provided to you (for free), the situation may be a different one if time
does not play that big a role or, you plan to run multiple jobs in parallel.
You are correct to note that I/O operations are very important and often
one badly chosen/implemented "element(s)" slows down everything. A
well-balanced I/O speed between DDR memory, GPU (and its memory), CPU
and fast SSD will be needed for a stand-alone machine to be
useful/compatible.
My thoughts / two cents / feedback:
CPU: 1 CPU with 32 cores
GPU: 2 GPUs with 16-24 GB GDDR each
DDR: 256 GB (minimally 128 GB)
SSD: 1 TB M.2 PCIe 3.0 x 4 (for the dataset being processed)
2nd SSD: 1 TB standard SSD (not PCIe 3.0 x 4) for the linux/software etc.
Raid: 2 x 12TB HDD Disks (for general dataset storage)
Motherboard: maximum (all possible) number of PCIe lanes implemented and
with 10GB ethernet (for example to connect 2 dedicated machines with one
another)
- 32 core CPU will do fine unless budget plays no role or, you want to
outperform the GPUs: The 3990X will out-render an RTX 2060 Super. Extra
money is better spent on professional GPUs and 256 GB DDR memory
(instead of the minimum 128 GB)
- 2 GPUs with 16-24 GB GDDR each in order to optimize data shuffling: 2
GPU cards can be each run at PCIe 3.0 x16 speed instead of more cards at
x8 speed. More memory on the card is more efficient/important than the
total number of cards / cuda-cores. With 2 cards you will have less
problems with heat production in the case and no problem with the supply
of power. Many do not notice that because of heat, CPUs and especially
GPUs are throttled severely: Consumer RTX cards are not designed for the
purpose of running 4 of them on one motherboard / in one case! The hot
air is not "exported" out of the case by consumer RTX cards. Finally,
Nvidia still generously allows to install the cuda drivers on consumer
RTX cards but actually not in the environment of a central university
facility (read the small print)! Further, the cuda software profits from
having professional quadro cards installed such as the RTX 5000 or 6000,
which in addition do not get that hot!
- 256 GB for big datasets, minimally 128 GB but with the ability for
later upgrade -> octa-channel mainboard
- 1 TB M.2 PCIe 3.0 x 4 ultra-fast SSD disk for placing the dataset
under process as it will be read more than once during processing: Many
mainboards allow to install only one ultra-fast M.2 disk, for all other
software including Linux and Relion you will thus need a normal SSD. Buy
a Samsung 980 Pro for placing the data to be processed as these will
last longer but be prepared to replace it after some time because of the
heavy I/O burden....
- Motherboard with maximum possible number of PCIe lanes -> Threadripper
motherboard: No consumer motherboard supplies the maximum possible
number of lanes! X299 motherboards (Intel) and several proposedly
high-end Intel CPUs suffer from the same "problem". Beware, PCIe lanes
or other "ports" may become disabled once you plug-in further M.2 SSD
disks etc. Make really sure to study/scrutinize the small print in the
manual of the motherboard, it will pay off.
Cheers,
Jeroen
Am 25.03.21 um 13:29 schrieb Pierre-Damien Coureux:
Hi everyone
we are thinking in buying new nodes for our data processing with
relion. We still have technical questions that need to be addressed
and I couldn't find recent benchmarks for the answers:
- what's the best choice between buying 2 nodes with 4xGPU or 1 node
with 8xGPU with RTX 3080/3090 cards. In terms of speed calculation, is
1 node with 8xGPU still equivalent to 3 nodes with 4xGPU ?
- Comparing RTX 3080 (8704 cores, 10 Go GDDR, 2-Slot height) and RTX
3090 cards (10496 cores, 24 Go GDDR, 3-Slot height), would 4x3090 GPU
cards be more advantageous in one node if you can run relion with 2
mpi/card (because of the 24 Go RAM/card) instead of having one node
with 8x3080 GPU cards and run 1 mpi/card?
- for a node with 8xGPU with RTX 30XX cards, because of the speed of
the cards, would the RAID SSD drive Bus speed (1 Go/s) be enough to
"feed" all the GPU without slowing down the system ? Is extra
PCI-Express card with SSD at 3 Go/s a preferred choice for this kind
of node ?
Any feedback is welcome
Pierre-Damien Coureux
--
*Dr.math. et dis. nat.Jeroen R. Mesters*
Deputy, Lecturer, Program Coordinator /Infection Biology
/
<http://www.uni-luebeck.de/studium/studiengaenge/infection-biology/introduction.html>Visiting
Professorship (South Bohemian University) in Biophysics
Logo Uni Lübeck
*University of Lübeck*
Center for Structural and Cell Biology in Medicine
*Institute of Biochemistry*
Tel +49 451 3101 3105 (secretariate 3101)
Fax +49 451 3101 3104
jeroen.mest...@uni-luebeck.de <mailto:jeroen.mest...@uni-luebeck.de>
www.biochem.uni-luebeck.de <http://www.biochem.uni-luebeck.de>
*Ratzeburger Allee 160
23538 Lübeck, Schleswig-Holstein
Germany*
########################################################################
To unsubscribe from the CCP4BB list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/WA-JISC.exe?SUBED1=CCP4BB&A=1
This message was issued to members of www.jiscmail.ac.uk/CCP4BB, a mailing list
hosted by www.jiscmail.ac.uk, terms & conditions are available at
https://www.jiscmail.ac.uk/policyandsecurity/
