Magnolia
MCSR offers access to the Magnolia supercomputer.
About Magnolia
Magnolia is an HPE Apollo 2000 cluster with 60 total compute nodes: 54 CPU-only nodes and 6 nodes with dual NVIDIA L40S GPUs. The nodes are interconnected with HDR100 Infiniband. It runs SUSE SLES 15 Linux, uses the Bright Cluster Management software, and uses SLURM for job management.
CPU Nodes
Each CPU node has dual Intel Xeon Gold 6342 processors, 512GB of RAM, and a 2TB SSD for fast local storage. Eight nodes have 1TB of RAM.
GPU Nodes
GPU nodes have two NVIDIA L40S accelerators each. They also have a single Intel Xeon Gold 6548Y+ 32 core processor, 512GB of RAM, and a 3.8TB SSD for local storage.
Software
Magnolia currently has Anaconda Python, R, MOLPRO, and Gaussian 16 installed. Example jobs, with Slurm scripts, are available in /usr/local/apps/example_jobs. To request software, please contact us via email.
Slurm
All past MCSR systems have used PBS for job management. Magnolia uses Slurm, a more modern job management system in wide use in the HPC world. Below are Slurm commands you’ll need to submit and manage jobs. Currently there are only two job queues (Slurm calls them partitions): defq for CPU-only jobs and gpuq for jobs that require GPUs.
Submit a job
sbatch name.slurm
Check on status of job
squeue -j jobid
Get detailed information on job
scontrol show job jobid
See all jobs
squeue
See queue status
sinfo
Cancel job
scancel jobid
Submit interactive job
srun -c 4 --mem 4g --time 1:00:00 --pty bash
GPUs on Magnolia
Magnolia has 6 GPU nodes with 2 NVIDIA L40S GPUs each. The GPU nodes are in the gpuq “partition” or queue. Jobs requiring a GPU should submit requests to this queue and request the appropriate number of GPUs using the Slurm –gpus option, or one of its variations.
The CUDA toolkit is available via the cuda12.6/toolkit module.
The following command starts an interactive job on a GPU node, requesting one GPU and half of the node’s CPUs and memory:srun -p gpuq --gpus 1 -c 16 --mem 250g --time 1:00:00 --pty bash