User Tools

Site Tools


Sun Grid Engine

The job scheduling system currently in use is Sun Grid Engine 6.2u4. There is vendor documentation available online:

Submit a Job

There is a command called qsub that will submit jobs into the batch queue. Here is a sample submit script written in bash (note the -S option):

#$ -S /bin/bash
uname -a

For MPI jobs once you load your MPI environment (via the module load command) mpirun will be in your PATH. Here is a parallel submit script:

#$ -S /bin/bash
# add any module load directives here:
module load compilers/gcc-4.4 mpi/openmpi-1.3-gcc-4.4
mpirun ./mpibinary

It is submitted using the qsub command like so: qsub -pe mpi 2 ./test.sub

:!: NOTE: The following options to mpirun are no longer used: -machinefile and -np. Grid Engine is configured to use tight integration and will handle that automatically.

Keep in mind that if a node has N CPUs that no single CPU job should take more than 1/N of the ram on that node. If you need more memory then you should ask for enough CPUs on that node to get the amount of memory you need. So for instance on a node with 4 CPU and 8GB ram if you want to run a 4GB single cpu job you should run:

qsub -pe threaded 2

If you have a MPI code and you only want a single job to run per node use the following mpirun flags instead of trying to play with the host list. In the below example we will ask for 4 nodes, but run only a single MPI process per node.

qsub -pe mpi 64 # 4 nodes worth in this example


mpirun -npernode 1 ./relay 1

Common Options

Here are a few submit script options that you might want to use. These options can be passed in via the command line or via the submit script. If you wish to put them in your submit script then just prefix them with #$.

  • -cwd: Execute the job from the current working directory.
  • -S /path/to/shell: Specifies the interpreting shell for the job.
  • -V: Specifies that all environment variables active within the qsub utility be exported to the context of the job.
  • -v variable[=value]: Defines or redefines the environment variables to be exported to the execution context of the job.
  • -m b|e|a|s|n: Defines or redefines under which circumstances mail is to be sent to the job owner or to the users defined with the -M option described below. The option arguments have the following meaning:
    • b: Mail is sent at the beginning of the job.
    • e: Mail is sent at the end of the job.
    • a: Mail is sent when the job is aborted or rescheduled.
    • s: Mail is sent when the job is suspended.
    • n: No mail is sent.
  • -M email_address: Defines or redefines the list of users to which the server that executes the job has to send mail, if the server sends mail about the job.
  • -N name: The name of the job.

The job script below requests an MPI job with 8 cores and 1.5 hours of run time:

#$ -V	 # Inherit the submission environment
#$ -cwd	 # Start job in submission directory
#$ -N myMPI	 # Job Name
#$ -j y	 # Combine stderr and stdout
#$ -o $JOB_NAME.o$JOB_ID  	 # Name of the output file (eg. myMPI.oJobID)
#$ -pe threaded 8
#$ -q normal	 # Queue name "normal"
#$ -l h_rt=01:30:00	 # Run time (hh:mm:ss) - 1.5 hours
#$ -M	 # Use email notification address
#$ -m be	 # Email at Begin and End of job
set -x	 # Echo commands, use "set echo" with csh

If you don't want stderr and stdout directed to the same file, replace the -j option line with a -e option to name a separate output file for stderr (but don't use both). By default, stderr and stdout are sent to files out.o and err.o, respectively.

For more information on the qsub command see the man page, type man qsub.

Monitor a Job

There is a command line utility called qstat that you can use to monitor jobs in Grid Engine.

To view only your jobs run the following:

$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
  40340 0.50976   bill         r     08/05/2008 14:12:07 ib.q@icompute-1-27.local           4        
  40339 0.50976    bill         r     08/05/2008 14:11:52 ib.q@icompute-1-29.local           4 

To view everyone's jobs you can use the -u option (note: you must escape the asterisk):

$ qstat -u \*
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
  40340 0.50976   bill         r     08/05/2008 14:12:07 ib.q@icompute-1-27.local           4        
  40339 0.50976    bill         r     08/05/2008 14:11:52 ib.q@icompute-1-29.local           4  
  38904 0.50976 QLOGIN     root         r     07/28/2008 18:22:02 ib.q@icompute-3-29.local           4 

For an overall queue summary you can use the -g option:

$ qstat -g c # display cluster queue summary
CLUSTER QUEUE                   CQLOAD   USED  AVAIL  TOTAL aoACDS  cdsuE  
all.q                             0.79      0    796    892      0     96 
broken                            0.00      0     80    100      0     20 
eth.q                             0.68    140      0    192      0     52 
ib.q                              0.93    556      8    588      0     24

For more information on the qstat command see the man page, type man qstat.


This section is intended for administrators. To backup and restore the SGE config see this howto.

Parallel Environments

There are more than one MPI layers installed on this cluster. OpenMPI is now very stable and many vendors are getting behind their implementation. We use it exclusively mostly because it offers features that the vendor-specific implementations don't offer.

support/hpc/software/gridengine.txt · Last modified: 2012/03/15 16:59 by tlknight