Repository navigation
xanadu gpu usage
This is a brief guide to the somewhat convoluted way you should use GPU resources on Xanadu.
The reason this is convoluted is because Xanadu runs an older version of SLURM, which does not manage GPU resources properly. When our new cluster, Mantis, comes online, you will not need to do this.
Currently, Xanadu has 10 nodes with GPUs. They all have NVIDIA A10s (one also has several NVIDIA Tesla M10s). They are all in the gpu partition (some are also in other partitions).
Ideally, you could simply write your batch script using some GPU-enabled software like Guppy and say sbatch guppy.sh and it would be sent off to a GPU-bearing node where it would happily chug away on a GPU. This is NOT always the case however.
Because SLURM is not tracking the GPUs, sometimes SLURM will assign your job to a node where the GPU is already occupied. This can be problematic. Obviously, if the two jobs are capable of fully using the GPU, they will run more slowly together, and sometimes we see out of memory errors.
So, the solution, for now, is to check whether a given node has the available resources you need (CPU, GPU and memory) and then either submit directly to that node, or exclude nodes that have GPU jobs running. This is not a perfect solution, but it's the best we can come up with for now.
So, how to check the status of each node?
First, see how much memory and CPU are unallocated for nodes in the GPU partition:
sinfo --format="%10P %6t %15C %10m %10e %15n %30E" -p gpuThis will print:
PARTITION STATE CPUS(A/I/O/T) MEMORY FREE_MEM HOSTNAMES REASON
gpu mix 27/9/0/36 257669 2550 xanadu-01 none
gpu mix 18/18/0/36 257669 33874 xanadu-02 none
gpu mix 32/4/0/36 257845 30971 xanadu-04 none
gpu mix 5/31/0/36 225612 35070 xanadu-05 none
gpu mix 28/8/0/36 515701 34642 xanadu-06 none
gpu mix 23/13/0/36 1031988 7691 xanadu-07 none
gpu mix 30/6/0/36 257669 124276 xanadu-08 none
gpu mix 1/47/0/48 386972 375820 xanadu-84 none
gpu alloc 36/0/0/36 257845 67005 xanadu-03 none
gpu idle 0/48/0/48 386972 372332 xanadu-85 none
The CPUS field lists the CPUs that are allocated (A), idle (I), other (O), and the total number on the node (T). The MEMORY field gives the total memory on the node in megabytes, and FREE_MEM field gives the memory that is unallocated by existing jobs. The HOSTNAMES field gives the name of each node.
Now you want to check whether the GPUs are occupied. The key command here is nvidia-smi. It will give you a GPU's current status. It must be executed on each node individually though.
You can request an interactive session on each of these nodes successively until you find an open one, or you can submit very tiny batch jobs in a bash loop to query all of them.
To request a specific node in an interactive session, use the -w flag:
srun -w xanadu-01 -p gpu --qos=general -c 1 --mem=1G --pty bashThen run nvidia-smi.
To query all of the nodes efficiently, you can write a loop that submits batch jobs (without a script) to each node and only runs nvidia-smi.
For example:
mkdir check_gpus
cd check_gpus
for NN in {01,02,03,04,05,06,07,08,84,85}
do sbatch -w xanadu-${NN} -c 1 --mem=1G -p gpu --qos=general -J node-${NN} -o xanadu-${NN}.out --wrap "nvidia-smi"
doneIf the jobs all run then you will have a list of files:
xanadu-01.out
xanadu-02.out
xanadu-04.out
xanadu-05.out
xanadu-06.out
xanadu-07.out
xanadu-08.out
xanadu-84.out
xanadu-85.out
Each will contain the status of that node's GPU. If it's unoccupied, it will look like this:
Thu Dec 7 09:31:47 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A10 Off | 00000000:03:00.0 Off | 0 |
| 0% 39C P0 52W / 150W | 0MiB / 22731MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
If it is occupied, it will look something like this:
Thu Dec 7 09:31:46 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A10 Off | 00000000:83:00.0 Off | 0 |
| 0% 84C P0 150W / 150W | 22408MiB / 22731MiB | 99% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2807806 C guppy_basecaller 1371MiB |
| 0 N/A N/A 4129570 C dorado 21035MiB |
+-----------------------------------------------------------------------------+
In the second one you can see that volatile GPU utilization is at 99% (meaning the processor is running full steam ahead) and the memory usage is also very high 22408MiB / 22731MiB. You don't want to submit a job to this node.
In the case that we see 4 nodes with unoccupied GPUs, let's say 01,02,03,04, and nodes 05,06,07,08 are occupied and 84, 85, are unsuitable for our application because they have older AMD processors, we can either submit our GPU job directly to one of these nodes (assuming partition, qos, cpu, memory etc are specified in the SLURM header in the script):
sbatch -w xanadu-01 guppy.shor we can exclude the nodes that we don't want our job to be submitted to:
sbatch --exclude xanadu-[05,06,07,08,84,85] guppy.sh