HOWTO: Submit multiple jobs using parameters

Often users want to submit a large amount of jobs all at once with each using different parameters for each job.  These parameters could be anything, including the path of a data file or different input values for a program.  This how-to will show you how you can do this using a simple python script, a CSV file, and a template script.  You will need to adapt this advice for your own situation.

Consider the following batch script:

#PBS -l nodes=1:ppn=12
#PBS -l walltime=80:00:00
#PBS -n week42_data8

# Copy input data to the nodes fast local disk 
cp ~/week42/data/source1/data8.out $TMPDIR
cd $TMPDIR

# Run the analysis 
full_analysis data8.in data8.out

# Copy results to proper folder
cp  data8.out ~/week42/results

Lets say you need to submit 100 of these jobs on a weekly basis.  Each job uses a different data file as input.  You recieve data from two different sources, and thus your data is located within two different folders.  All of the jobs from one week need to store their results in a single weekly results folder.  The output file name is based upon the input file name.

Creating a Template Script

As you can see, this job follows a general template.  There are three main parameters that change in each job:

1. The week 
   • Used as part of the job name
   • Used to find the proper data file to copy to the nodes local disk
   • Used to copy the results to the correct folder
2. The data source
   • Used to find the proper data file to copy to the nodes local disk
3. The data file's name
   • Used as part of the job name
   • Used to find the proper data file to copy to the nodes local disk
   • Used to specify both the input and output file to the program full_analysis
   • Used to copy the results to the correct folder

If we replace these parameters with variables, prefixed by the dollar sign $and surrounded by curly braces { }, we get the following template script:

#PBS -l nodes=1:ppn=12
#PBS -l walltime=80:00:00
#PBS -n ${WEEK}_${DATA}

# Copy input data to the nodes fast local disk 
cp ~/${WEEK}/data/${SOURCE}/${DATA}.out $TMPDIR
cd $TMPDIR

# Run the analysis 
full_analysis ${DATA}.in ${DATA}.out

# Copy results to proper folder
cp  ${DATA}.out ~/${WEEK}/results

Automating Job Submission

We can now use qsub's -v option to pass parameters to our template script.  The format for passing parameters is:

qsub -v par_name=par_value[,par_name=par_value...] script.sh

Submitting a 100 jobs using qsub -v option manually does not make our task much easier than modifying and submitting each job one by one.  To complete our task we need to automate the submission of our jobs.  We will do this by using a python script that submits our jobs using parameters it reads from a CSV file.  

Note that python was chosen for this task for its general ease of use and understandability -- if you feel more comfortable using another scritping language feel free to interpret/translate this python code for your own use.

Here is the script that submits the jobs using the parameters:

#!/usr/bin/env python
import csv, subprocess

parameter_file_full_path = "/nfs/12/user0123/week42/job_params.csv"

with open(parameter_file_full_path, "rb") as csvfile:
    reader = csv.reader(csvfile)
    for job in reader:
        qsub_command = """qsub -v WEEK={0},SOURCE={1},DATA={2} template_1.pbs""".format(*job)

        #print qsub_command # Uncomment this line when testing to view the qsub command

        # Comment the following 3 lines when testing to prevent jobs from being submitted
        exit_status = subprocess.call(qsub_command, shell=True)
        if exit_status is 1:  # Check to make sure the job submitted
            print "Job {0} failed to submit".format(qsub_command)
print "Done submitting jobs!"

This script will open the CSV file specified by the variable parameter_file_full_path and step through the file line by line, submitting a job for each line using the lines values.  If the qsub command returns a non-zero exit code, usually indicating it was not submitted, we will print this out to the display.  The jobs will be submitted using the general format:

qsub -v WEEK=WEEK_VALUE,SOURCE=SOURCE_VALUE,DATA=DATA_VALUE template_1.pbs

Where WEEK_VALUE ,SOURCE_VALUE, and DATA_VALUE are the first, second, and third comma separated values in the CSV file's current row, and template_1.pbs is the name of our template script.

Creating a CSV File

We now need to create a CSV file with parameters for each job.  This can be done with a regular text editor or using a spreadsheet editor such as excel.  By default you should use commas as your delimiter.  

Here is our CSV file with parameters:

week42,source1,data1
week42,source1,data2
week42,source1,data3
...
week42,source2,data98
week42,source2,data99
week42,source2,data100

The submit script would read in the first row of this CSV file and form and  submit the following qsub command:

qsub -v WEEK=week42,SOURCE=source1,DATA=data1 template_1.pbs

Submitting Jobs

Once all the above is done all you need to do to submit your jobs is make sure the CSV file is populated with the proper parameters and run the automatic submission script.