Maple
MCSR offers access to the Maple supercomputer.
About Maple
Maple is a Cray cluster with 140 compute nodes, a total of 5,454 CPU cores, and 45 NVIDIA GPUs, and 27.1TB of memory. This cluster has theoretical peak of 382 TFLOPS. It is interconnected with Infiniband. This cluster is partially funded by the National Science Foundation (CHE-1338056).
Specifications
The cluster originally consisted of 29 compute nodes, each with two 10 core Intel Xeon CPUs, 64GB of memory, and an NVIDIA Kepler K20 GPU. The cluster was upgraded in 2017, 2018, 2019, 2020, 2021, and 2022.
Maple has much newer CPUs than Sequoia and Catalpa and does not share their software stack. Also, while all of our systems run SUSE Linux Enterprise Server (SLES), Maple is our only Cray system and our only system to use Bright Cluster Manager, so its operating system and some software is setup differently.
All Maple nodes have local hard disks, accessible in /tmp. The original nodes (cn001-cn029) have 1TB disks and all newer nodes (cn030-cn140) have 2TB disks.
PBS
Maple has three overlapping PBS queues: workq, gpu, and biggpu. The biggpu queue contains the nodes with large memory GPUs (two each). This queue should only be used by jobs requiring the large memory GPUs. The gpu queue contains the 29 nodes with small memory GPUs. The workq queue is Maple’s default queue and contains all compute nodes with the exception of the biggpu nodes. This is the queue that should be used when no GPUs are required.
Requests for software installation can be made via email.
MPI on Maple
Maple has several flavors of MPI installed, but MVAPICH2 serves as our default. An example job can be found on Maple at /usr/local/apps/example_jobs/mpi_maple_example.
First copy the example job to your home directory:cp -r /usr/local/apps/example_jobs/mpi_maple_example/ ~
The example contains a simple MPI program written in C, a Makefile to compile the MPI program, and a PBS script to run it.
To compile the program, first load the MVAPICH module, then call make:module load intel/compiler/64/2019 mvapich2
make
To run the job, submit the PBS script:qsub hello.pbs
GPUs on Maple
The Maple cluster has two nodes with two NVIDIA Tesla P100 GPUs (12GB memory, midgpu queue) each, and six nodes with two NVIDIA Tesla V100S GPUs (32GB memory, biggpu queue) each. These GPUs can be used to substantially speed up certain types of programs, assuming software has been written and configured to make use of the GPUs.
CUDA
To load the CUDA Development Toolkit, run:module load cuda10.2/toolkit
This module will also need to be loaded inside PBS scripts for CUDA programs to run properly.
OpenCL
To compile OpenCL code, load the CUDA toolkit as described above, the use the -lOpenCL option when compiling.
Running GPU Jobs
To use a GPU on Maple, it must be requested, just like CPUs and memory. PBS has an ngpus resource for reserving GPUs. For instance, to reserve a single GPU, include#PBS -l ngpus=1
in your PBS script or on the command line when you start an interactive session of PBS (qsub -I).
Users must also submit their jobs to either the gpu, midgpu, or biggpu queue to get a node with a GPU. To use the newer, large memory GPUs, use the biggpu queue. There are twelve total, two GPUs each on six nodes. Submitting to the default queue with ngpus=1 will result in the job sitting in the queue indefinitely.
The nodes with the larger GPUs are a valuable resource. Do not use these if the smaller GPUs will suffice.