Which of the following command can be used to remove a job from the list of pending jobs

This page will give you a list of the commonly used commands for SLURM. Although there are a few advanced ones in here, as you start making significant use of the cluster, you’ll find that these advanced ones are essential!

A good comparison of SLURM, LSF, PBS/Torque, and SGE commands can be found here.

Also useful:

• What is FairShare and What’s My Lab’s Score?
• Managing FairShare in Multiple Groups
• Common Cluster Pitfalls
• My Job is Pending

General commands

Get documentation on a command:

man

Try the following commands:

man sbatch
man squeue
man scancel

Submitting jobs

The following example script specifies a partition, time limit, memory allocation and number of cores. All your scripts should specify values for these four parameters. You can also set additional parameters as shown, such as jobname and output file. For This script performs a simple task — it generates of file of random numbers and then sorts it. A detailed explanation the script is available here.

#!/bin/bash
#
#SBATCH -p shared # partition [queue]
#SBATCH -c 1 # number of cores
#SBATCH --mem 100 # memory pool for all cores
#SBATCH -t 0-2:00 # time [D-HH:MM]
#SBATCH -o slurm.%N.%j.out # STDOUT
#SBATCH -e slurm.%N.%j.err # STDERR
for i in {1..100000}; do
echo $RANDOM >> SomeRandomNumbers.txt
donesort SomeRandomNumbers.txt

Now you can submit your job with the command:

sbatch myscript.sh

If you want to test your job and find out when your job is estimated to run use [note this does not actually submit the job]:

sbatch --test-only myscript.sh

Information on jobs

List all current jobs for a user:
squeue -u

List all running jobs for a user:
squeue -u -t RUNNING

List all pending jobs for a user:
squeue -u -t PENDING

List priority order of jobs for the current user [you] in a given partition:
showq-slurm -o -u -q

List all current jobs in the shared partition for a user:
squeue -u -p shared

List detailed information for a job [useful for troubleshooting]:
scontrol show jobid -dd

List status info for a currently running job:
sstat --format=AveCPU,AvePages,AveRSS,AveVMSize,JobID -j --allsteps

Once your job has completed, you can get additional information that was not available during the run. This includes run time, memory used, etc.
To get statistics on completed jobs by jobID:
sacct -j --format=JobID,JobName,MaxRSS,Elapsed

To view the same information for all jobs of a user:
sacct -u --format=JobID,JobName,MaxRSS,Elapsed

Controlling jobs

To cancel one job:
scancel

To cancel all the jobs for a user:
scancel -u

To cancel all the pending jobs for a user:
scancel -t PENDING -u

To cancel one or more jobs by name:
scancel --name myJobName

To hold a particular job from being scheduled:
scontrol hold

To release a particular job to be scheduled:
scontrol release

To requeue [cancel and rerun] a particular job:
scontrol requeue

Job arrays and useful commands

As shown in the commands above, its easy to refer to one job by its Job ID, or to all your jobs via your username. What if you want to refer to a subset of your jobs? The answer is to submit your job set as a job array. Then you can use the job array ID to refer to the set when running SLURM commands. See the following excellent resources for further information:
Running Jobs: Job Arrays
SLURM job arrays

To cancel an indexed job in a job array:
scancel _
e.g.
scancel 1234_4

To find the original submit time for your job array
sacct -j 32532756 -o submit -X --noheader | uniq

Advanced [but useful!] commands

The following commands work for individual jobs and for job arrays, and allow easy manipulation of large numbers of jobs. You can combine these commands with the parameters shown above to provide great flexibility and precision in job control. [Note that all of these commands are entered on one line]

Suspend all running jobs for a user [takes into account job arrays]:
squeue -ho %A -t R | xargs -n 1 scontrol suspend

Resume all suspended jobs for a user:
squeue -o "%.18A %.18t" -u | awk '{if [$2 =="S"]{print $1}}' | xargs -n 1 scontrol resume

After resuming, check if any are still suspended:
squeue -ho %A -u $USER -t S | wc -l

The following is useful if your group has its own queue and you want to quickly see utilization.
lsload |grep 'Hostname\|'

Example for the smith partition:
lsload |grep 'Hostname|smith'
Hostname Cores InUse Ratio Load Mem Alloc State
smith01 64 60 100.0 12.01 262 261 ALLOCATED
smith02 64 64 100.0 12.00 262 240 ALLOCATED
smith03 64 40 100.0 12.00 262 261 ALLOCATED

• Note that while node 03 has free cores, all its memory in use. So those cores are necessarily idle.
• Node 02 has a little free memory but all the cores are in use.
• The scheduler will shoot for 100% utilization, but jobs are generally stochastic; beginning and ending at different times with unpredictable amounts of CPU and RAM released/requested.

Custom Commands

We and users on the FASRC cluster have developed custom commands that also are deployed on the cluster for use.  If you have a command you want to contribute please contact us.  Below are the custom commands we have added.

spart shows you the partitions you have access to: //github.com/fasrc/spart

showq which shows cluster state: //github.com/fasrc/slurm_showq

scalc which does various Slurm calculations such as projected fairshare usage: //github.com/fasrc/scalc

find-best-partition shows which partition will schedule your job the quickest: //github.com/fasrc/best_slurm_partition

Which of the following commands will show a list of process names along with their process ID PID?

Which of the following can be used to move a job to the background?

What option can be used with the ps command to display an entire list of processes across all?

Which of the following commands will change the SELinux security context of a file?