Changing Job Attributes
It is possible to modify the attributes of a submitted job using the scontrol
command while the job is running or pending in the queue. However, some job attributes cannot be updated during runtime like GPU(s), node(s), and memory allocations.
Only a few job attributes can be changed after a job is submitted. These attributes include:
- wall clock limit
- job name
- job dependency
For some of the cases, these attributes can only be updated for the pending jobs.
See some of the examples here :
To increase a job’s time limit as it approaches its initial submission deadline.
scontrol update JobId=$JobID timelimit=<new timelimit>
The format set is Minutes, Minutes: Seconds, Hours:Minutes: Seconds, Days-Hours, Days-Hours: Minutes, or Days-Hours:Minutes: Seconds.
To change the job dependency:
scontrol update JobId=$JobID_1 dependency=afterany:$JobID_2
Controlling jobs
A User can prevent a pending job from being started by placing one’s jobs in a ‘hold’ state by using scontrol
command.
scontrol hold <job_id>
To release a queued job from the ‘hold’ state :
scontrol release <job_id>
To cancel and rerun (requeue) a particular job:
scontrol requeue <job_id>
Canceling jobs
To cancel either running or pending jobs:
scancel <job_id>
To cancel all jobs owned by a user
scancel -u <user>