GPU compute server - gpu18

This machine is for GPU-accelerated scientifique computations of university members. Each GPU can be scheduled to jobs separately. SLURM is used as job scheduler. Operating system is Linux/Ubuntu. Connection and data transfer is possible via ssh (secure shell).

8GPUs inside DGX1

Access is possible after login using ssh to gpu18.urz.uni-magdeburg.de via intra-uni-network only. An account can be created on formless inquiry to the hpc-admins. Please add information about the project name and its duration (maximum estimated), how much connected GPUs (mostly we expect a single GPU), GPU-memory (max 32GB/GPU), main-Memory and Disk-Space is needed at maximum for each class of jobs you may have. This data will be used to adjust the slurm jobsystem to different needs for best usage i.e. adapting the job queues for different job sizes. Students need a permission (formless email) from its tutor which must be an employee of the university.
Please use other smaller GPU-systems if your jobs do not really need 32GB GPU memory to leave the GPUs for users which want that huge GPU-memory and have no other choices (p.e. t100 gpu-nodes). The GPU-memory and its speed is the most expensive part of that kind of GPU-machines. Also note, that the GPU machine though to have huge main memory (512GB), but if that must be distributed to 8 jobs on 8 GPUs only 64 GB memory is left for each users job, which is only a factor of 2 to the GPU-memory. That means you dont have much space to buffer deep-learning data to memory feeding the GPU. If you want more main memory, there must be other users which are satisfied with less than 64GB main memory. Also there is only 7TB SSD-storage for the 8 GPUs. We try to expand this clear bottleneck 2020/2021 by EDR-infiniband storage if we get money for it.
Please always be careful in allocating CPUs and memory to avoid conflicts with other jobs. Check scaling before using multiple GPUs for one job. Sharing a single GPU for multiple jobs is not enabled by default to avoid out-of-memory-conflicts for GPU memory (which may crash the GPU, please report your experiences to us).
Operating system is Linux Ubuntu. CPU-intense preparation of jobs should not be done on the GPU node. Please use the hpc18 system for it.
This machine is not suited for work with personal data. That means there is no encryption and failed disks will be replaced without any possibility to destroy data on this disk before giving it back to a third party due to hardware warranty terms.

History:

2019-01-30 installation + first GPU-test-users
2019-01-30 ssh-access through ssh gpu18.urz.uni-magdeburg.de (intranet only)
2019-03-20 slurm 15.08.7 installed (from ubuntu reposities)
2019-04-08 fixed typo in dns-nameservers (network was partly slow)
2019-06-12 server room power loss due to short (de: Kurzschluss)
2019-08-08 config central kerberos-passwords (university)
2019-08-12 slurm 17.11.12 installed (+dbd for single GPU allocation)
2019-09-22 single GPU-allocation by slurm, old: node allocation
2019-10-29 reboot to get dead nvidia6 running (slurm problem)
2020-01-14 install g++-5 g++-6 g++-7 g++-9 (old: g++-8)
2020-03-11 reboot (nfs4 partly hanging, some non-accessible files, ca. 40 zombies)
2020-03-20 nfs-server overload errors, nfs4 switched to nfs3 testwise
2020-10-05 07:20 system SSD 480GB defect, system down
2020-10-08 SSD replaced, system restore from system backup (2019-11)
2020-10-12 system up, users allowed, test phase (plz report errors)
2020-10-28 13:55 -crash- uncorrectable memory error

Problems: