Often users want to submit a large amount of jobs all at once with each using different parameters for each job. These parameters could be anything, including the path of a data file or different input values for a program. This how-to will show you how you can do this using a simple python script, a CSV file, and a template script. You will need to adapt this advice for your own situation.
Consider the following batch script:
#PBS -l nodes=1:ppn=12 #PBS -l walltime=80:00:00 #PBS -n week42_data8 # Copy input data to the nodes fast local disk cp ~/week42/data/source1/data8.out $TMPDIR cd $TMPDIR # Run the analysis full_analysis data8.in data8.out # Copy results to proper folder cp data8.out ~/week42/results
Lets say you need to submit 100 of these jobs on a weekly basis. Each job uses a different data file as input. You recieve data from two different sources, and thus your data is located within two different folders. All of the jobs from one week need to store their results in a single weekly results folder. The output file name is based upon the input file name.
Creating a Template Script
As you can see, this job follows a general template. There are three main parameters that change in each job:
- The week
- Used as part of the job name
- Used to find the proper data file to copy to the nodes local disk
- Used to copy the results to the correct folder
- The data source
- Used to find the proper data file to copy to the nodes local disk
- The data file's name
- Used as part of the job name
- Used to find the proper data file to copy to the nodes local disk
- Used to specify both the input and output file to the program
full_analysis
- Used to copy the results to the correct folder
If we replace these parameters with variables, prefixed by the dollar sign $
and surrounded by curly braces { }
, we get the following template script:
#PBS -l nodes=1:ppn=12 #PBS -l walltime=80:00:00 #PBS -n ${WEEK}_${DATA} # Copy input data to the nodes fast local disk cp ~/${WEEK}/data/${SOURCE}/${DATA}.out $TMPDIR cd $TMPDIR # Run the analysis full_analysis ${DATA}.in ${DATA}.out # Copy results to proper folder cp ${DATA}.out ~/${WEEK}/results
Automating Job Submission
We can now use qsub'
s -v option to pass parameters to our template script. The format for passing parameters is:
qsub -v par_name=par_value[,par_name=par_value...] script.sh
Submitting a 100 jobs using qsub -v
option manually does not make our task much easier than modifying and submitting each job one by one. To complete our task we need to automate the submission of our jobs. We will do this by using a python script that submits our jobs using parameters it reads from a CSV file.
Note that python was chosen for this task for its general ease of use and understandability -- if you feel more comfortable using another scritping language feel free to interpret/translate this python code for your own use.
Here is the script that submits the jobs using the parameters:
#!/usr/bin/env python import csv, subprocess parameter_file_full_path = "/nfs/12/user0123/week42/job_params.csv" with open(parameter_file_full_path, "rb") as csvfile: reader = csv.reader(csvfile) for job in reader: qsub_command = """qsub -v WEEK={0},SOURCE={1},DATA={2} template_1.pbs""".format(*job) #print qsub_command # Uncomment this line when testing to view the qsub command # Comment the following 3 lines when testing to prevent jobs from being submitted exit_status = subprocess.call(qsub_command, shell=True) if exit_status is 1: # Check to make sure the job submitted print "Job {0} failed to submit".format(qsub_command) print "Done submitting jobs!"
This script will open the CSV file specified by the variable parameter_file_full_path
and step through the file line by line, submitting a job for each line using the lines values. If the qsub
command returns a non-zero exit code, usually indicating it was not submitted, we will print this out to the display. The jobs will be submitted using the general format:
qsub -v WEEK=WEEK_VALUE,SOURCE=SOURCE_VALUE,DATA=DATA_VALUE template_1.pbs
Where WEEK_VALUE
,SOURCE_VALUE
, and DATA_VALUE
are the first, second, and third comma separated values in the CSV file's current row, and template_1.pbs
is the name of our template script.
Creating a CSV File
We now need to create a CSV file with parameters for each job. This can be done with a regular text editor or using a spreadsheet editor such as excel. By default you should use commas as your delimiter.
Here is our CSV file with parameters:
week42,source1,data1 week42,source1,data2 week42,source1,data3 ... week42,source2,data98 week42,source2,data99 week42,source2,data100
The submit script would read in the first row of this CSV file and form and submit the following qsub
command:
qsub -v WEEK=week42,SOURCE=source1,DATA=data1 template_1.pbs
Submitting Jobs
Once all the above is done all you need to do to submit your jobs is make sure the CSV file is populated with the proper parameters and run the automatic submission script.