Introduction to AI Workloads on LUMI
🔵 Slurm
LUMI-G Node Resources
1/8 of Node (1 GPU)
| Total | Available | |
|---|---|---|
| Cores | 8 | 7 |
| RAM | 64 GB | 60 GB |
| VRAM | 64 GB | 64 GB |
Whole Node
| Total | Available | |
|---|---|---|
| Cores | 64 | 56 |
| RAM | 512 GB | 480 GB |
| VRAM | 512 GB | 512 GB |
Partitions
| Partition | Allocation |
|---|---|
| standard-g | Whole LUMI-G nodes |
| small-g | Fraction of LUMI-G node |
| standard | Whole LUMI-C nodes |
| small | Fraction of LUMI-C node |
https://docs.lumi-supercomputer.eu/runjobs/scheduled-jobs/partitions/
Resources Allocation and Job Scheduling
Interactive Session
Allocate single GPU for 30 minutes.
salloc --partition=small-g \
--account=project_46xxxxxxx \
--nodes=1 \
--gpus-per-node=1 \
--ntasks-per-node=1 \
--cpus-per-task=7 \
--mem-per-gpu=60G \
--time=00:30:00Then run the code.
srun echo "Hello world!"Interactive Session via Srun
Allocate single GPU and echo “Hello world!”.
srun --partition=small-g \
--account=project_46xxxxxxx \
--nodes=1 \
--gpus-per-node=1 \
--ntasks-per-node=1 \
--cpus-per-task=7 \
--mem-per-gpu=60G \
--time=00:05:00 \
echo "Hello world!"Batch Script
hello_world.sh
#!/bin/bash
#SBATCH --job-name=hello-world
#SBATCH --partition=small-g
#SBATCH --account=project_46xxxxxxx
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=7
#SBATCH --mem-per-gpu=60G
#SBATCH --time=00:05:00
#SBATCH --output=slurm-%j.log
srun echo "Hello world!"Schedule the job.
sbatch hello_world.shOther Slurm Commands
Show Availability of Nodes
sinfo -sAllocated/Idle/Other/Total
List My Jobs
squeue --meCancel Job
scancel <job_id>Get to Allocated Node
srun --overlap --pty --jobid=<job_id> bashThen you can for example watch the gpu load.
watch -n 1 rocm-smiWatch Logs of Job Scheduled via Sbatch
It’s not a slurm command but it’s useful.
tail -f slurm-<job_id>.logMultiple Nodes, GPUs and Tasks
Count of tasks = count of processes
Single Node, 2 GPUs, Single Task
#!/bin/bash
#SBATCH --job-name=hello-world
#SBATCH --partition=small-g
#SBATCH --account=project_46xxxxxxx
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=14
#SBATCH --gpus-per-node=2
#SBATCH --mem-per-gpu=60G
#SBATCH --time=00:05:00
#SBATCH --output=slurm-%j.log
srun echo "Hello world!"The file slurm-<job_id>.log will contain:
Hello world!Single Node, 2 GPUs, 2 Tasks
#!/bin/bash
#SBATCH --job-name=hello-world
#SBATCH --partition=small-g
#SBATCH --account=project_46xxxxxxx
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=7
#SBATCH --gpus-per-node=2
#SBATCH --mem-per-gpu=60G
#SBATCH --time=00:05:00
#SBATCH --output=slurm-%j.log
srun echo "Hello world!"The file slurm-<job_id>.log will contain:
Hello world!
Hello world!2 Nodes, 2 Tasks
#!/bin/bash
#SBATCH --job-name=hello-world
#SBATCH --partition=standard-g
#SBATCH --account=project_46xxxxxxx
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=56
#SBATCH --gpus-per-node=8
#SBATCH --mem-per-gpu=60G
#SBATCH --time=00:05:00
#SBATCH --output=slurm-%j.log
srun echo "Hello world!"The file slurm-<job_id>.log will contain:
Hello world!
Hello world!⚫ Storage
- /project/project_46xxxxxxx/
- /flash/project_46xxxxxxx/
- /scratch/project_46xxxxxxx/
- /users/<user_name>/
- /tmp/
- Object storage
https://docs.lumi-supercomputer.eu/storage/
Lustre
https://docs.lumi-supercomputer.eu/storage/parallel-filesystems/lustre/
Object Storage / S3
https://docs.lumi-supercomputer.eu/storage/lumio/clients-general/
Get LUMI-O Access
https://docs.lumi-supercomputer.eu/storage/lumio/auth-lumidata-eu/
Web Interface
https://docs.lumi-supercomputer.eu/runjobs/webui/#accessing-lumi-o
Configure
module load lumio
lumio-confCopy File
cd /scratch/project_46xxxxxxx
module load lumio
# List files in bucket
s3cmd ls s3://testbucket
# Copy file from scratch to bucket
s3cmd put test_file.txt s3://testbucket/test_file.txt
# Copy file from bucket to scratch
s3cmd get s3://testbucket/test_file.txt test_file.txtPrefixes
s3cmd put test_file.txt s3://testbucket/animals/mammals/black_cat.txt
s3cmd put test_file.txt s3://testbucket/animals/birds/bald_eagle.txt
# List all mammal files
s3cmd ls s3://testbucket/animals/mammals/
# List all animal files
s3cmd ls --recursive s3://testbucket/animals/Tags
Max 10 tags.
# Set tags
s3cmd settagging s3://testbucket/test_file.txt "Color=Red&Shape=Square"
# Get tags
s3cmd gettagging s3://testbucket/test_file.txtFile Formats
https://github.com/Lumi-supercomputer/LUMI-AI-Guide/tree/main/03-file-formats
SquashFS
https://docs.lumi-supercomputer.eu/storage/formats/FUSE/#squashfs
🟡 Containers
Pre-Made AI Containers
https://docs.lumi-supercomputer.eu/laif/software/ai-environment/#available-container-images
Get Into “Torch-Full” Container Shell.
# Load modules.
module purge
module use /appl/local/laifs/modules
module load lumi-aif-singularity-bindings
# Export the container path.
export SIF=/appl/local/laifs/containers/lumi-multitorch-u24r70f21m50t210-20260513_121430/lumi-multitorch-full-u24r70f21m50t210-20260513_121430.sif
# Get into the container shell
singularity shell $SIFRun Command From Container
singularity run $SIF echo "Hello world!"singularity run $SIF pip listAlternatively use exec instead of run.
Slurm + Container
srun --partition=small-g \
--account=project_46xxxxxxx \
--nodes=1 \
--gpus-per-node=1 \
--ntasks-per-node=1 \
--cpus-per-task=7 \
--mem-per-gpu=60G \
--time=00:05:00 \
singularity run $SIF echo "Hello world!"List All “Torch-Full” Containers From 2026.
ls -lh /appl/local/laifs/containers/lumi-multitorch-*-2026*/lumi-multitorch-full-*.sifDirectory Bindings
echo $SINGULARITY_BIND🔴 Multi-GPU, Multi-Node
📌 https://462000265.lumidata.eu/ai-20260611/files/LUMI-ai-20260611-08-Scaling_multiple_GPUs.pdf
📌 https://462000265.lumidata.eu/ai-20260611/files/LUMI-ai-20260611-09-Extreme_scale_AI.pdf
https://github.com/Lumi-supercomputer/LUMI-AI-Guide/tree/main/05-multi-gpu-and-node
https://github.com/Lumi-supercomputer/Getting_Started_with_AI_workshop/tree/main/08_Scaling_to_multiple_GPUs (Multi-GPU only)
https://github.com/Lumi-supercomputer/Getting_Started_with_AI_workshop/tree/main/09_Extreme_scale_AI
CPU Binding for Full Node Job
srun --cpu-bind=mask_cpu:0xfe000000000000,0xfe00000000000000,0xfe0000,0xfe000000,0xfe,0xfe00,0xfe00000000,0xfe0000000000 singularity run ...Multi-Node
Use All 4 High-Speed Interfaces
export NCCL_SOCKET_IFNAME=hsn0,hsn1,hsn2,hsn3Use GPU RDMA
export NCCL_NET_GDR_LEVEL=PHBNCCL Logs
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,COLL
export NCCL_DEBUG_FILE=/tmp/$(whoami)-rccl-rank$SLURM_PROCID.txt🟣 Inference
https://github.com/Lumi-supercomputer/LUMI-AI-Guide/tree/main/10-LLM-inference
Extra: Multi-Node vLLM
https://github.com/CSCfi/ai-inference-examples/blob/master/run-vllm-lumi16.sh
https://docs.gpustack.ai/latest/tutorials/running-distributed-vllm-with-multiprocessing/
🟢 Account and Project Management
https://docs.lumi-supercomputer.eu/runjobs/lumi_env/dailymanagement/
My User Info
lumi-ldap-userinfoMy Projects Info
lumi-ldap-projectinfo⚪ Other Sources
Training
https://lumi-supercomputer.github.io/LUMI-training-materials/ai-20260611/
https://github.com/Lumi-supercomputer/Getting_Started_with_AI_workshop
LUMI AI Guide
https://github.com/Lumi-supercomputer/LUMI-AI-Guide/tree/main


