Modifying Jobs

Changing Job Attributes

It is possible to modify the attributes of a submitted job using the scontrol command while the job is running or pending in the queue. However, some job attributes cannot be updated during runtime like GPU(s), node(s), and memory allocations.

Only a few job attributes can be changed after a job is submitted. These attributes include:

  • wall clock limit
  • job name
  • job dependency

For some of the cases, these attributes can only be updated for the pending jobs.

See some of the examples here :

To increase a job’s time limit as it approaches its initial submission deadline.

scontrol update JobId=$JobID timelimit=<new timelimit>The format set is Minutes, Minutes: Seconds, Hours:Minutes: Seconds, Days-Hours, Days-Hours: Minutes, or Days-Hours:Minutes: Seconds.

To change the job dependency:

scontrol update JobId=$JobID_1 dependency=afterany:$JobID_2

Controlling jobs

A User can prevent a pending job from being started by placing one’s jobs in a ‘hold’ state by using scontrol command.

scontrol hold <job_id>

To release a queued job from the ‘hold’ state :

scontrol release <job_id>

To cancel and rerun (requeue) a particular job:

scontrol requeue <job_id>

Canceling jobs

To cancel either running or pending jobs:

scancel <job_id>

To cancel all jobs owned by a user

scancel -u <user>