IT4I Agent User Guide

agent is an assistant for working on the IT4Innovations supercomputers. You ask in plain words, it looks at the real state of the cluster you are on - queues, nodes, your jobs, your quotas, your projects, the installed modules - and answers for that system, not in general.

What Is/Isn’t It For

What it is for:

• Answering questions about the queue, the nodes, the partitions and your jobs.
• Telling you why a job failed or why it is still waiting, and how efficiently it ran.
• Drafting a submit script and checking it against this cluster - module names against Lmod, #SBATCH directives against Slurm, walltime against the queue, rank counts, --account against your accounts.
• Handing you a ready-made template for a serial, OpenMP, MPI, hybrid, array or GPU job, built from this cluster’s real numbers.
• Reviewing a script you already have on disk and saying what is wrong with it.
• Writing the program or the input file the job runs.
• Telling you the current state of one job, and listing your job history.
• Showing storage quotas, what is filling a directory up, and how much of a project’s allocation is left.
• Finding a module, searching the IT4I documentation, and looking up scientific papers.
• Checking your SSH keys.

What it is not:

• It is not a shell. It runs nothing on your behalf except the read-only commands it needs for the answer. It submits a job only when you explicitly ask, and even then it asks first.
• It is not an authority. The answer is written by a language model. The numbers it takes from the cluster (queue, quotas, accounting) are reliable, but read the reasoning around them before you act on it.
• It does not replace support. When it does not know, the question belongs to support@it4i.cz.
• It does not run inside a batch job. It is a login-node tool, not something to call from a job script - see below.
• It does not read your data outside your home, scratch, project storage and the current directory, and it never shows a private key.
• It cannot watch anything in the background. It answers the question in front of it; it cannot notify you later that a job has finished.

Starting the Agent

usr0123@login1:~$ ml AGENT/1.0
usr0123@login1:~$ agent
it4i agent 2026.08.27.2046 - ask in your own words. Ctrl-D or 'exit' quits.
  You are on Barbora2.
  Projects for --account: defac, service
  6 queues, e.g. qcpu, qcpu_exp, qcpu_free, qcpu_long
Type /help for what it can do, /tools for the tool list.
>

The agent runs as a session: after an answer it waits for the next question and follows on from what was already said. The last six exchanges are kept in full; what falls out of that window is condensed into a short note of what you asked, so an instruction you gave earlier (“on project OPEN-XX-YY”) is not lost. That is what makes “change it to eight nodes” or “save it as mpi.sh” work without repeating yourself.

End the session with exit, quit, bye, q, :q (or the Czech konec, ukoncit) - and with Ctrl-D. Ctrl-C interrupts only the question in flight; the session stays open.

A question can also go straight on the command line; the agent answers it and then continues the session. With -1 it answers and exits:

usr0123@login1:~$ agent "how many core hours do I have left?"
usr0123@login1:~$ agent -1 "what time is it"

A session you saved with /save can be picked up later:

usr0123@login1:~$ agent --resume        # the most recent saved session
usr0123@login1:~$ agent --resume vasp   # a session saved by name

Login Nodes Only, Terminal Only

The agent starts only on login nodes (login*) and only when a human is at the terminal. It will not start inside a batch job, a pipe or a cron job:

usr0123@login1:~$ srun --pty agent
agent: the agent runs only on login nodes (login*); this is cn042

usr0123@login1:~$ agent "what time is it" < /dev/null
agent: the agent runs only interactively; there is no terminal here

Two reasons: on a compute node you would be paying for the question with your project’s core hours, and in a batch job there is nobody to confirm an action that changes something.

Diagnostics (--health, --config, --doc, --version, --list-models, --selftest) bypass both checks so they can be run from anywhere.

Language

The interface (help texts, messages) is English; -cs switches it to Czech, and so does AGENT_LANG=cs:

usr0123@login1:~$ agent -cs

The answer follows the language of the question. Ask in Czech and you get a Czech answer even without -cs. The flag is for when you want the interface itself in Czech - including this guide, which agent --doc then shows in Czech. Inside a session /cs and /en switch the interface without a restart.

Session Commands

Besides questions, the session takes twenty-one commands. They do not go to the model and answer immediately:

> /tools
Tools the agent can use (! = asks for your confirmation):
    check_jobscript  Check a job script that already exists on disk
    draft_jobscript  Write a Slurm job script for the user WITHOUT saving it
    job_template     Give the user a ready-made Slurm job script
  ! sbatch           Submit an EXISTING job script file to Slurm
  ! write_jobscript  Save a job script to a file
  ...

/facts is worth reaching for when an answer does not match reality - it shows what the agent actually knows about the cluster: partitions, node counts, the current queue, the accounts --account accepts, reservations and maintenance windows, announcements, and the names (never the contents) of the files in the directory you started in.

/context and /cost are the honest answer to “what exactly are you sending away?”. /context prints the whole system prompt; /cost gives the estimated token count for the next question against the input cap.

/copy uses the terminal’s own clipboard (OSC 52), so it works over SSH without X forwarding - some terminals have it switched off. /edit opens the last draft in $VISUAL, then $EDITOR, otherwise vi; your version replaces the draft, so a following “save it” saves what you edited.

/save writes the session under a name (posledni when you give none) and tells you the agent --resume command to continue it. /transcript writes the exchanges into a Markdown file in your home directory - useful for a support ticket - and never overwrites an existing file.

/good and /bad write a note next to the history in your home directory. Nothing is sent anywhere; it is material for improving the agent.

/memory, /remember, /forget, /save and /audit need your consent to write into the home directory (see section 8); without it they say so instead of storing anything.

Command-Line Options

Behavior

Memory

Diagnostics

--config prints the model, how the agent authenticates, the timeouts, the token caps and the state directory - never the backend address and never the credential. On the cluster --list-models is not available, because the credential is held by a system helper that only forwards questions; the model actually in use is in --config.

What It Can Do

Queue and Jobs

The agent sees your jobs and explains in plain words why a job is waiting (Priority, Resources, ReqNodeNotAvail, AssocGrpCPUMinsLimit and the other reasons). For pending jobs it also asks Slurm when it expects them to start - an estimate that moves with the queue.

> do I have anything running?
> why is my job 1178 still waiting?
> what is in the queue for everyone?
> cancel job 1180

By default you see only your own jobs; ask about “everyone” or “the whole queue” and it widens the view. The memory column in the queue is what a job asked for, not what it uses. Cancelling a job changes state, so the agent asks first.

Job History

The history needs no job ID - “what have I run” is enough. It lists the job ID, name, queue, account, state, exit code, submission time and how long the job ran, newest first. By default it looks 30 days back and returns fifteen jobs; say you want older ones or more of them and both numbers go up.

> what is my job history?
> what have I run in the last 90 days?

Cancelling a job changes state, so the agent asks first.

Why a Job Failed and How Efficiently It Reviewingan

The agent reads the Slurm accounting data and computes the reason from the numbers instead of guessing it: out of time, out of memory, a non-zero exit code, cancelled, a failed node. The same data gives the efficiency of the run: how much of the reserved memory was really used, whether the job shared its nodes with somebody else, and what share of the reserved core time it actually spent computing.

> why did my last job fail?
> what happened to job 48192?
> did that job use the memory I reserved for it?
> show me the end of slurm-48192.out

Without an ID the agent looks at your most recent job - that is usually what “why did it fail” means. It also names the job’s output file and can read it.

Submit Scripts

The agent drafts the script, shows it and submits nothing. The draft is checked against this cluster at the same time:

• module names against Lmod, and mixed toolchains,
• #SBATCH directives against sbatch --test-only,
• the queue against the requested walltime, and a needlessly long walltime,
• “all cores” against the real core count from sinfo,
• --account against the accounts Slurm knows for you,
• #SBATCH placed after the first command (Slurm ignores such a directive),
• --ntasks against --nodes and --ntasks-per-node,
• the MPI launcher against the MPI module that is loaded,
• job arrays (index range, %a in the output file) and --dependency (whether the job it waits for can still start),
• container calls and compilation commands inside the script,
• the program the script runs: whether the file exists, whether it is executable, whether it has a shebang,
• cd into a directory that does not exist, missing module purge, input files that are not there, no program actually being run.

> write me a submit script for 4 nodes, 2 hours, all cores
> change it to 8 nodes and add module purge
> save it as mpi.sh

When you keep changing the draft, the agent prints only what changed since the version you already saw.

The memory estimate from your earlier runs. When the script has both --job-name and --mem, the agent looks up how much memory jobs of that name really used over the last 90 days and says so: reserving several times more than you ever used only makes you wait longer in the queue, and reserving less than the earlier peak would get the job killed.

Checking a Script You Already Have

> check my job.sh
> go through ~/jobs/vasp.sh and tell me what is wrong

The same checks run over your own file, and the review is read-only - your file is not overwritten. A bare file name is looked for in the current directory and in your home; a script outside your own data is refused, and so is anything longer than 500 lines. The corrected version is kept aside as a draft, so you can have it saved under a different name.

Job Templates

For the common types of jobs the agent has ready-made templates:

> give me a template for an MPI job on 8 nodes for 4 hours
> I need an array job template

They are built from this cluster’s real numbers - cores per node, sockets, queues and GPUs from sinfo - so they are not a generic example from the internet. The template deliberately leaves out --account: the agent asks which project the job should be charged to instead of picking one. You are expected to replace the placeholder program and the module load line.

Companion Files: The Program and the Input

A job is almost never just the submit script. The agent will also write the program the job runs - Python, C, Fortran, shell - or an input file.

> also write a python program that prints the rank and the hostname
> I need an input file for it
> save that program as mpi_test.py

The same rules apply: the draft is only shown, it is saved when you ask, and an existing file is not overwritten.

Interactive Allocation

> I need an interactive session on one node for two hours

The agent assembles the salloc command including --account and prints it for you to run yourself - an interactive allocation needs a terminal, so the agent cannot start it. The result says NOT EXECUTED for exactly that reason: nothing has been allocated until you run the command.

If you have more than one Slurm account and named none of them in the question, the agent asks which project the job should run on first. It will not assemble the command until you tell it. The queue is chosen from the walltime you asked for, and the agent says when it changed the partition you named.

Watching a Job

> is job 48192 done yet?
> how far is my job?

You get one snapshot: pending (with the reason), running, or already finished (with the final state, the exit code and how long it ran). The agent does not wait and cannot notify you - ask again for a fresh state.

It does report, once per start, which of your jobs finished since the last time you ran agent - up to ten of them, looking back at most seven days, and nothing at all on the very first run.

Storage, Quotas and Disk Space

> how much free space do I have on scratch?
> where should I put two terabytes of data?
> what is filling up my home?

Quotas come from it4ifsusage and are watched not only by volume but also by file count - a quota can be exhausted by a million small files just as well. Project storage lines are reported too, and the agent warns about an approaching expiry. If a filesystem is nearly full, it says so on its own at the start of the session.

“What is filling it up” is a different question, and a different measurement: the agent runs du one level deep and lists the largest directories under the path. On a directory with millions of files it can take a while and may time out - then it tells you to run du yourself in screen or tmux. It measures only inside your own data, and it never proposes a command that deletes anything.

Project Consumption

> how many core hours are left?
> how much has project OPEN-XX-YY spent?

Allocation, usage and what is left come from it4ifree, together with an estimate of how long the remainder lasts at the current pace and how long the project is still valid. If you are the PI of a project, the agent says so - that is who requests more.

Watch the difference in names: it4ifree reports project IDs from the accounting portal (DD-13-5, SERVICE), while Slurm knows accounts (defac, service). Only what Slurm knows may go into --account; a portal ID there fails with “Invalid account”. The accounts that work are in /facts. A default Slurm account is also not a project, even though jobs without --account are charged to it.

Account Limits

A job Slurm refuses with MaxSubmit or AssocMax... looks like a broken script, and is not: it means you already have as many jobs queued as you are allowed. Ask “how many jobs can I submit” and the agent lists the limits on your account and QOS. An empty limit means no limit, never zero.

Scaling: Are More Nodes Worth It?

If you have already run the same job on different node counts, the agent works it out from the accounting: how the runtime shortened and what it cost in corehours. It also says where more nodes stop being worth the allocation they spend.

is it worth running my md-run job on eight nodes?

Only runs that finished are compared - a failed or killed job measures when it died, not how long it computed. With a single data point the agent says so and does not guess, and it never extrapolates beyond the largest node count you actually tried.

Cancelling Several Jobs at Once

cancel all my pending jobs

The agent first shows the list of jobs it would cancel (id, state, project, name) and cancels them only after you confirm. It does not touch running jobs unless you ask for that in so many words - a cancelled computation is lost work. If your jobs belong to several projects and you did not name one, it asks which.

Moving Data

how do I get 2 TB of data onto the cluster?

The advice is specific to this cluster: which tool (scp versus rsync), where to put it (and whether it fits), and what to watch out for - rsync -avP and screen for a large transfer, and packing a million small files first, because quotas limit the number of files too. The commands run on your own machine, not on the login node.

When the Software Is Not on the Cluster

GROMACS is not here, what now?

The agent checks whether something similar is installed after all, and if not, gives both ways forward: ask for an installation at support@it4i.cz (and what the request should say), or build it into your own home directory with EasyBuild.

Modules

> is GROMACS available here?
> which Python versions are installed?
> which module do I load for OpenMPI?

IT4I Documentation

The agent searches a knowledge base built from the IT4I documentation and cites the source URL so you can verify the answer. The Slurm man pages are in there too - for the exact syntax of a command - and are cited as man <command>, not as IT4I documentation.

> how do I apply for core hours?
> what does the AssocGrpCPUMinsLimit error mean?
> how do I transfer data to the cluster?

Every excerpt says which cluster it documents, and excerpts for the cluster you are on come first; when an answer rests on a passage written for another cluster, the agent has to say so. When the best match is weak, it says that and points you at support@it4i.cz instead of inventing an answer. When the index is more than a week old, it notes that the newest changes are not in it yet.

Scientific Papers

Search goes through OpenAlex (250M+ works). The agent returns title, authors, year, citation count, journal, DOI, and an open access link when there is one. It can sort by relevance (the default), by citation count, or by year.

> find papers on DFT calculations of perovskites
> next five
> save a hundred papers on molecular dynamics with LAMMPS to a file

Results come five at a time; “next five” moves to the following batch. Up to a hundred can be written to a text file in your home directory - that is a write, so the agent asks, and it never overwrites an existing file.

The query is translated into English for the search; scientific literature is overwhelmingly English and a non-English query returns almost nothing.

SSH Keys

> does my private key match the public one?
> are the permissions on ~/.ssh right?
> is my key in authorized_keys?

The agent reports key types, fingerprints, whether a private key matches its .pub, whether the key is in authorized_keys, and wrong file permissions. The private key itself is never read and never shown - only fingerprints are used, and a key protected by a passphrase simply cannot be paired without it.

How-to questions such as “how do I connect over SSH” or “how do I generate a key” are answered from the documentation, not by this tool.

Time and Cluster State

> what time is it
> which partitions are there?
> how many compute nodes does the cluster have?

The agent does not invent the time, it reads it from the login node. The cluster facts are measured when the session starts - /facts prints all of them.

New Cluster Announcements

You see the MOTD when you log in. When IT4I announces an outage in the middle of the day you are already logged in and will not see it - the agent keeps a fingerprint of the last announcement and reminds you when it changes. The full text comes from it4imotd.

Memory

The agent can remember facts about you between sessions - your projects, the software and the partitions you use.

> I usually work on project OPEN-XX-YY

Memory is opt-in: on first run the agent asks whether it may write anything into your home directory. Without consent nothing is created there - no notes, no history, no saved sessions. Every note it stores is printed to you immediately, so you can see what was written, and it will not store something you did not say yourself.

usr0123@login1:~$ agent --memory                    # what it remembers
usr0123@login1:~$ agent --remember project "OPEN-XX-YY"
usr0123@login1:~$ agent --edit-memory               # edit by hand
usr0123@login1:~$ agent --forget project            # drop one note
usr0123@login1:~$ agent --forget                    # drop all, revoke consent
usr0123@login1:~$ agent --no-memory "..."           # this once, without memory
usr0123@login1:~$ agent --init                      # wipe everything

The same three things work inside a session as /memory, /remember and /forget, so you do not have to leave what you are doing.

What the Agent Never Does on Its Own

These are not instructions in a prompt but guards in the code. Neither the wording of a question nor the content of a file the agent reads can change them.

• It does not submit a job until you explicitly ask. When you have it write a script, it shows the script and does not offer to submit it. Only when you say “submit it” does it call sbatch - and even then it first shows you the exact command and waits for y.
• It does not write or overwrite a file without confirmation. Saving a script, a program or a paper export changes state: you see what is about to happen and it waits for y. An existing file is not overwritten at all, and a path outside your own data is refused.
• Nothing is allocated by salloc either. That command is text for you to run; the result says NOT EXECUTED so that no answer can claim otherwise.
• It does not read files outside your data. Only your home, scratch, project storage and the current directory are readable. Anything else (/etc/passwd, say) is refused even where you have the rights. Lines that look like a password or a token are stripped from what it does read, and you are told how many.
• It never shows a private key. A file that looks like a private key is refused. SSH keys are handled through fingerprints only.
• It does not pick a project for you. With several Slurm accounts and none named, it will not build a command with --account; it asks.
• It uses only public models. Private models are blocked in the code, not in a setting, so no configuration can turn that off.
• It remembers nothing without your consent, and it will not store a note about something you did not say yourself.

At the confirmation prompt you have more than yes and no:

Proposed action:
sbatch /home/usr0123/jobs/mpi.sh

Run it? [y = yes, N = no, d = show it all, e = edit]

d prints the whole thing when the preview was shortened (only the first 40 lines are shown), and e opens a multi-line argument - a script or a file - in your editor, so “almost right” does not mean starting over. Anything other than y means no.

--dry-run shows what the agent would do and performs nothing. -y does the opposite and turns confirmation off - use it only where you know exactly what will happen. Declining does not end the session.

When Something Does Not Work

“the agent runs only on login nodes (login*); this is cn042” You are on a compute node or inside a job. The agent deliberately does not run there - go back to a login node.

“the agent runs only interactively; there is no terminal here” Input or output is not a terminal - a batch job, a pipe, cron. The agent needs a human at the terminal.

Backspace and the arrow keys do not work The readline module is missing; the agent says so at startup. The line is then handled by the terminal alone and Backspace may not delete. stty erase '^?' often helps; otherwise use agent -1 "question" and type the question on the command line. With readline present you also get Tab completion for the session commands, the queue names and your accounts.

The answer has no diacritics That is the default: the text is transliterated to ASCII, because in a terminal without UTF-8 every accented character would show up as _. If your terminal is fine, turn them on:

usr0123@login1:~$ AGENT_UTF8=1 agent

Turning colors off

usr0123@login1:~$ NO_COLOR=1 agent
usr0123@login1:~$ AGENT_COLOR=0 agent

Redirected output never gets colors anyway.

A long answer opens in a pager Answers longer than the screen go to $PAGER (less by default), so a list of a hundred papers does not scroll away. AGENT_NO_PAGER=1 switches that off.

The agent picked the wrong tool Only the tools that fit the question are offered to the model - measured to be both faster and more accurate. If the one you need is missing, force the whole set:

usr0123@login1:~$ AGENT_ALL_TOOLS=1 agent

Rephrasing more concretely (naming the job ID or the file) helps too, and --explain shows which tools were used.

“too many requests - please wait a while and try again” The limit is 60 questions per hour per user. The backend is shared by everyone on the cluster.

“the answer was cut off at the output limit” Ask for fewer items, or raise AGENT_MAX_OUTPUT_TOKENS.

“the model did not put an answer together. Here is what the agent found” Now and then the model returns an empty result instead of an answer - with a long question, or with unusual content. When the agent already has the data in hand (a file, the queue, your quotas), it shows them as the tool returned them instead of apologizing. It is a raw listing with no summary; ask again, more specifically, if you want one.

“the model did not return a usable answer this time” The same, but the agent had nothing to show - no tool was called. Ask again and be more specific, for example with a job ID.

The backend does not answer, or the question times out

usr0123@login1:~$ agent --health
usr0123@login1:~$ agent --timeout 300

On the cluster --health checks two things: that the system helper is in place, and that a trivial question comes back with a sensible answer - that second one covers the network, the credentials and the model at once. --no-probe skips asking the model. Nothing there is something you can fix yourself; if it fails, the report goes to support@it4i.cz.

Because the probe really does ask the model, the same guards apply to it as to a question: login node only, terminal only, and the limit of 60 questions an hour. For monitoring there is agent --health --no-probe, which never touches the model and runs anywhere.

Return codes (useful in a wrapper script):

The advice is wrong

The answer is written by a language model and it does get things wrong. Read the commands before you run them. --explain shows which tools were used, /facts shows what it knows about the cluster. Flag recurring problems with /bad and write to support@it4i.cz.

Where to go next

• IT4I documentation: https://docs.it4i.cz.
• Support: support@it4i.cz.
• Source code and bug reports: gitlab.it4i.cz (IT4I VPN needed).

Where the Agent Keeps State

~/.it4i-agent/ - everything that needs your consent:

The directory is 0700, the files 0600 - including the /transcript write-up, which holds whole answers and therefore parts of your files. Without consent nothing is created: no notes, no history, no audit log, no saved sessions, no transcripts and no feedback. The last draft is kept so that “save it” or “change it” still works in the next session. agent --init asks once and then deletes the whole directory, saved sessions and drafts included (with no terminal it deletes nothing); agent --forget without a key wipes the memory and revokes the consent.

~/.cache/it4i-agent/ - things that say nothing about you, so they need no consent: a fingerprint of the last announcement you were shown, the mark of when the agent last looked at your finished jobs, and a five-minute cache of the slowest cluster facts. When the home directory cannot be written, a private directory under /tmp named by your user ID is used instead.

The request counter for the 60-per-hour limit lives in /tmp as well, in a directory of its own per user; that is not user data either.

Your history, notes and sessions live in your home directory, which on the cluster is shared between login nodes - so --resume works from any of them.

Privacy

• Questions go to a language model running on the IT4I DGX - they do not leave the IT4I infrastructure.
• Only public models are used; private ones are blocked in the code.
• What goes to the model is the system prompt (the measured cluster facts, the tool definitions, and your memory notes if you enabled them), your question, and the results of the tools that ran. /context prints it, /cost measures it.
• Text the agent did not write itself is framed as untrusted before the model sees it - a file it read, the job queue, module lists, documentation excerpts, paper records, cluster announcements. The model is told to treat it as data and never to follow instructions found inside it.
• The agent does not know the backend credential - a system helper holds it and it cannot be extracted from the agent’s process. Neither the backend address nor the key is printed, not even by --config.
• Paper searches are the one query that leaves the cluster: they go to the public OpenAlex API. Nothing about you goes with them, only the search words.
• The agent runs as your user: it sees only what you have rights to, and submits jobs to your account.
• Feedback (/good, /bad) is stored next to the history in your home directory and is not sent anywhere.