it4i logoDocumentation
Lumi

Introduction to AI Workloads on LUMI

Introduction to AI workloads on LUMI

🔵 Slurm

LUMI-G Node Resources

1/8 of Node (1 GPU)

TotalAvailable
Cores87
RAM64 GB60 GB
VRAM64 GB64 GB

Whole Node

TotalAvailable
Cores6456
RAM512 GB480 GB
VRAM512 GB512 GB

Partitions

PartitionAllocation
standard-gWhole LUMI-G nodes
small-gFraction of LUMI-G node
standardWhole LUMI-C nodes
smallFraction of LUMI-C node

https://docs.lumi-supercomputer.eu/runjobs/scheduled-jobs/partitions/

Resources Allocation and Job Scheduling

Interactive Session

Allocate single GPU for 30 minutes.

salloc --partition=small-g \
    --account=project_46xxxxxxx \
    --nodes=1 \
    --gpus-per-node=1 \
    --ntasks-per-node=1 \
    --cpus-per-task=7 \
    --mem-per-gpu=60G \
    --time=00:30:00

Then run the code.

srun echo "Hello world!"

Interactive Session via Srun

Allocate single GPU and echo “Hello world!”.

srun --partition=small-g \
    --account=project_46xxxxxxx \
    --nodes=1 \
    --gpus-per-node=1 \
    --ntasks-per-node=1 \
    --cpus-per-task=7 \
    --mem-per-gpu=60G \
    --time=00:05:00 \
    echo "Hello world!"

Batch Script

hello_world.sh

#!/bin/bash
#SBATCH --job-name=hello-world
#SBATCH --partition=small-g
#SBATCH --account=project_46xxxxxxx
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=7
#SBATCH --mem-per-gpu=60G
#SBATCH --time=00:05:00
#SBATCH --output=slurm-%j.log

srun echo "Hello world!"

Schedule the job.

sbatch hello_world.sh

Other Slurm Commands

Show Availability of Nodes

sinfo -s

Allocated/Idle/Other/Total

List My Jobs

squeue --me

Cancel Job

scancel <job_id>

Get to Allocated Node

srun --overlap --pty --jobid=<job_id> bash

Then you can for example watch the gpu load.

watch -n 1 rocm-smi

Watch Logs of Job Scheduled via Sbatch

It’s not a slurm command but it’s useful.

tail -f slurm-<job_id>.log

Multiple Nodes, GPUs and Tasks

Count of tasks = count of processes

Single Node, 2 GPUs, Single Task

#!/bin/bash
#SBATCH --job-name=hello-world
#SBATCH --partition=small-g
#SBATCH --account=project_46xxxxxxx
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=14
#SBATCH --gpus-per-node=2
#SBATCH --mem-per-gpu=60G
#SBATCH --time=00:05:00
#SBATCH --output=slurm-%j.log

srun echo "Hello world!"

The file slurm-<job_id>.log will contain:

Hello world!

Single Node, 2 GPUs, 2 Tasks

#!/bin/bash
#SBATCH --job-name=hello-world
#SBATCH --partition=small-g
#SBATCH --account=project_46xxxxxxx
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=7
#SBATCH --gpus-per-node=2
#SBATCH --mem-per-gpu=60G
#SBATCH --time=00:05:00
#SBATCH --output=slurm-%j.log

srun echo "Hello world!"

The file slurm-<job_id>.log will contain:

Hello world!
Hello world!

2 Nodes, 2 Tasks

#!/bin/bash
#SBATCH --job-name=hello-world
#SBATCH --partition=standard-g
#SBATCH --account=project_46xxxxxxx
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=56
#SBATCH --gpus-per-node=8
#SBATCH --mem-per-gpu=60G
#SBATCH --time=00:05:00
#SBATCH --output=slurm-%j.log

srun echo "Hello world!"

The file slurm-<job_id>.log will contain:

Hello world!
Hello world!



⚫ Storage

  • /project/project_46xxxxxxx/
  • /flash/project_46xxxxxxx/
  • /scratch/project_46xxxxxxx/
  • /users/<user_name>/
  • /tmp/
  • Object storage

https://docs.lumi-supercomputer.eu/storage/

Lustre

https://docs.lumi-supercomputer.eu/storage/parallel-filesystems/lustre/

Object Storage / S3

https://docs.lumi-supercomputer.eu/storage/lumio/clients-general/

Get LUMI-O Access

https://docs.lumi-supercomputer.eu/storage/lumio/auth-lumidata-eu/

Web Interface

https://docs.lumi-supercomputer.eu/runjobs/webui/#accessing-lumi-o

Configure
module load lumio
lumio-conf
Copy File
cd /scratch/project_46xxxxxxx
module load lumio

# List files in bucket
s3cmd ls s3://testbucket

# Copy file from scratch to bucket
s3cmd put test_file.txt s3://testbucket/test_file.txt

# Copy file from bucket to scratch
s3cmd get s3://testbucket/test_file.txt test_file.txt
Prefixes
s3cmd put test_file.txt s3://testbucket/animals/mammals/black_cat.txt
s3cmd put test_file.txt s3://testbucket/animals/birds/bald_eagle.txt

# List all mammal files
s3cmd ls s3://testbucket/animals/mammals/

# List all animal files
s3cmd ls --recursive s3://testbucket/animals/
Tags

Max 10 tags.

# Set tags
s3cmd settagging s3://testbucket/test_file.txt "Color=Red&Shape=Square"

# Get tags
s3cmd gettagging s3://testbucket/test_file.txt

File Formats

https://github.com/Lumi-supercomputer/LUMI-AI-Guide/tree/main/03-file-formats

SquashFS

https://docs.lumi-supercomputer.eu/storage/formats/FUSE/#squashfs




🟡 Containers

Pre-Made AI Containers

https://docs.lumi-supercomputer.eu/laif/software/ai-environment/#available-container-images

Get Into “Torch-Full” Container Shell.

# Load modules.
module purge
module use /appl/local/laifs/modules
module load lumi-aif-singularity-bindings

# Export the container path.
export SIF=/appl/local/laifs/containers/lumi-multitorch-u24r70f21m50t210-20260513_121430/lumi-multitorch-full-u24r70f21m50t210-20260513_121430.sif

# Get into the container shell
singularity shell $SIF

Run Command From Container

singularity run $SIF echo "Hello world!"
singularity run $SIF pip list

Alternatively use exec instead of run.

Slurm + Container

srun --partition=small-g \
    --account=project_46xxxxxxx \
    --nodes=1 \
    --gpus-per-node=1 \
    --ntasks-per-node=1 \
    --cpus-per-task=7 \
    --mem-per-gpu=60G \
    --time=00:05:00 \
    singularity run $SIF echo "Hello world!"

List All “Torch-Full” Containers From 2026.

ls -lh /appl/local/laifs/containers/lumi-multitorch-*-2026*/lumi-multitorch-full-*.sif

Directory Bindings

echo $SINGULARITY_BIND



🔴 Multi-GPU, Multi-Node

📌 https://462000265.lumidata.eu/ai-20260611/files/LUMI-ai-20260611-08-Scaling_multiple_GPUs.pdf

📌 https://462000265.lumidata.eu/ai-20260611/files/LUMI-ai-20260611-09-Extreme_scale_AI.pdf

https://github.com/Lumi-supercomputer/LUMI-AI-Guide/tree/main/05-multi-gpu-and-node

https://github.com/Lumi-supercomputer/Getting_Started_with_AI_workshop/tree/main/08_Scaling_to_multiple_GPUs (Multi-GPU only)

https://github.com/Lumi-supercomputer/Getting_Started_with_AI_workshop/tree/main/09_Extreme_scale_AI

CPU Binding for Full Node Job

srun --cpu-bind=mask_cpu:0xfe000000000000,0xfe00000000000000,0xfe0000,0xfe000000,0xfe,0xfe00,0xfe00000000,0xfe0000000000 singularity run ...

Multi-Node

Use All 4 High-Speed Interfaces

export NCCL_SOCKET_IFNAME=hsn0,hsn1,hsn2,hsn3

Use GPU RDMA

export NCCL_NET_GDR_LEVEL=PHB

NCCL Logs

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,COLL
export NCCL_DEBUG_FILE=/tmp/$(whoami)-rccl-rank$SLURM_PROCID.txt



🟣 Inference

https://github.com/Lumi-supercomputer/LUMI-AI-Guide/tree/main/10-LLM-inference

Extra: Multi-Node vLLM

https://github.com/CSCfi/ai-inference-examples/blob/master/run-vllm-lumi16.sh

https://docs.gpustack.ai/latest/tutorials/running-distributed-vllm-with-multiprocessing/




🟢 Account and Project Management

https://docs.lumi-supercomputer.eu/runjobs/lumi_env/dailymanagement/

My User Info

lumi-ldap-userinfo

My Projects Info

lumi-ldap-projectinfo



⚪ Other Sources

Training

https://lumi-supercomputer.github.io/LUMI-training-materials/ai-20260611/

https://github.com/Lumi-supercomputer/Getting_Started_with_AI_workshop

LUMI AI Guide

https://github.com/Lumi-supercomputer/LUMI-AI-Guide/tree/main

© 2026 IT4Innovations – All rights reserved.

On this page