edu.isi.pegasus.planner.code.generator.condor
Class CondorGenerator

java.lang.Object
  edu.isi.pegasus.planner.code.generator.Abstract
      edu.isi.pegasus.planner.code.generator.condor.CondorGenerator

All Implemented Interfaces:

CodeGenerator

Direct Known Subclasses:

HashedFile

public class CondorGenerator
extends Abstract

This class generates the condor submit files for the DAG which has to be submitted to the Condor DagMan.

Version:

$Revision: 4782 $

Author:

Gaurang Mehta, Karan Vahi

Field Summary

static String	CONDOR_DAGMAN_LOGICAL_NAME The logical name with which to query the transformation catalog for the condor_dagman executable, that ends up running the mini dag as one job.
static String	CONDOR_DAGMAN_NAMESPACE The namespace to use for condor dagman.
static String	DAGMAN_PROPERTIES_PREFIX The prefix for DAGMan specific properties
static int	DEFAULT_CHMOD_PRIORITY_KEY The default priority key associated with chmod jobs.
static int	DEFAULT_CLEANUP_PRIORITY_KEY The default priority key associated with the cleanup jobs.
static int	DEFAULT_CREATE_DIR_PRIORITY_KEY The default priority key associated with the create dir jobs.
static int	DEFAULT_INTER_SITE_PRIORITY_KEY The default priority key associated with the inter site transfer jobs.
static int	DEFAULT_REPLICA_REG_PRIORITY_KEY The default priority key associated with the replica registration jobs.
static int	DEFAULT_STAGE_IN_PRIORITY_KEY The default priority key associated with the stagein jobs.
static int	DEFAULT_STAGE_OUT_PRIORITY_KEY The default priority key associated with the stage out jobs.
static String	DEFAULT_SUBDAG_CATEGORY_KEY The default category for the sub dax jobs.
private boolean	mAssignDefaultJobPriorities Boolean indicating whether to assign job priorities or not.
protected ADag	mConcreteWorkflow The workflow for which the code has to be generated.
private long	mCondorVersion The long value of condor version.
protected PrintWriter	mDagWriter The file handle to the .dag file.
protected boolean	mDone A boolean indicating whether the files have been generated or not.
protected GridStartFactory	mGridStartFactory The handle to the GridStart Factory.
protected boolean	mInitializeGridStart A boolean indicating whether grid start has been initialized or not.
protected String	mPoolClass Specifies the implementing class for the pool interface.
static String	mSeparator The nice separator, define once, use often.
private SiteStore	mSiteStore The handle to the site catalog store.
protected CondorStyleFactory	mStyleFactory Handle to the Style factory, that is used for this workflow.
protected TransformationCatalog	mTCHandle Handle to the Transformation Catalog.
protected String	mTempLogFile The name of the log file in the /tmp directory

Fields inherited from class edu.isi.pegasus.planner.code.generator.Abstract

mBag, mLogger, mPOptions, mProps, mSubmitFileDir

Fields inherited from interface edu.isi.pegasus.planner.code.CodeGenerator

VERSION

Constructor Summary

CondorGenerator()
The default constructor.

Method Summary

protected void	applyStyle(Job job, PrintWriter writer) Applies a submit file style to the job, according to the fact whether the job has to be submitted directly to condor or to a remote jobmanager via CondorG and GRAM.
protected void	concreteDagEmpty(String filename, ADag dag) A callout method that dictates what needs to be done in case the concrete plan that is generated is empty.
protected Job	constructDAGJob(String name, String directory, String dagBasename) Constructs a job that plans and submits the partitioned workflow, referred to by a Partition.
private Map	constructMap(String propValue) Constructs a map with the numbers/values to be passed in the RSL handle for certain pools.
private TransformationCatalogEntry	constructTCEntryFromEnvironment() Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION
private TransformationCatalogEntry	constructTCEntryFromEnvProfiles(ENV env) Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION
private TransformationCatalogEntry	defaultTCEntry(String site) Returns a default TC entry to be used in case entry is not found in the transformation catalog.
Collection<File>	generateCode(ADag dag) Generates the code for the concrete workflow in Condor DAGMAN and CondorG input format.
void	generateCode(ADag dag, Job job) Generates the code (condor submit file) for a single job.
protected boolean	generateLogFileSymlink(String logFile, String symlink) This method generates a symlink to the actual log file written in the local temp directory.
Map<String,String>	getAdditionalBraindumpEntries(ADag workflow) Returns a Map containing additional braindump entries that are specific to a Code Generator.
protected String	getBasename(String prefix, String suffix) A covenience method to construct the basename.
protected String	getCategoryDAGManKnobs(PegasusProperties properties) Write out the DAGMan knobs for each category the user mentions in the properties.
protected String	getCondorLogInSubmitDirectory() Returns the path to the condor log file in the submit directory.
protected String	getCondorLogInSubmitDirectory(ADag dag) Returns the path to the condor log file in the submit directory.
protected String	getCondorLogInTmpDirectory() Returns the name of Condor log file in a tmp directory that is created if generation of symlink for condor logs is turned on.
protected String	getDAGMANOutFilename(ADag dag) Returns the basename of the file, that contains the output of the dagman while running the dag generated for the workflow.
protected int	getJobPriority(Job job, int depth) Computes the priority for a job based on job type and depth in the workflow
protected int	getNaturalNumberValue(String value) Returns a natural number value ( > 0 ) if the parameter passed is an integer and greater than zero, else -1
private String	gridstart(PrintWriter writer, Job job, boolean isGlobusJob) This function creates the stdio handling with and without gridstart.
protected void	handleCondorVarForJob(Job job) It updates/adds the condor variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally.
protected void	handleEnvVarForJob(Job sinfo) It updates/adds the environment variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally.
protected void	handleGlobusRSLForJob(Job sinfo) It updates/adds the the Globus RSL parameters got through the dax that are in Job object.
void	initialize(PegasusBag bag) Initializes the Code Generator implementation.
protected File	initializeDagFileWriter(String filename, DagInfo dinfo) Initializes the file handler to the dag file and writes the header to it.
void	populatePeriodicReleaseAndRemoveInJob(Job job) Populates the periodic release and remove values in the job.
protected void	printDagString(String str) Writes a string to the dag file.
void	reset() Resets the Code Generator implementation.
boolean	startMonitoring() Starts monitoring of the workflow by invoking a workflow monitor daemon tailstatd.
protected void	writeDagFileTail(DagInfo dinfo) It writes the relations making up the DAG in the dag file and and closes the file handle to it.
protected void	writeDOTFile(String filename, ADag dag) Writes out the DOT file in the submit directory.
protected void	writeJobMapFile(String filename, ADag dag) Writes out the job map file in the submit directory.

Methods inherited from class edu.isi.pegasus.planner.code.generator.Abstract

getDAGFilename, getDAGFilename, getFileBaseName, getWriter, writeOutBraindump, writeOutDAXReplicaStore, writeOutNotifications, writeOutStampedeEvents, writeOutWorkflowMetrics

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_SUBDAG_CATEGORY_KEY

public static final String DEFAULT_SUBDAG_CATEGORY_KEY

The default category for the sub dax jobs.

See Also:

Constant Field Values

mSeparator

public static final String mSeparator

The nice separator, define once, use often.

See Also:

Constant Field Values

CONDOR_DAGMAN_NAMESPACE

public static final String CONDOR_DAGMAN_NAMESPACE

The namespace to use for condor dagman.

See Also:

Constant Field Values

CONDOR_DAGMAN_LOGICAL_NAME

public static final String CONDOR_DAGMAN_LOGICAL_NAME

The logical name with which to query the transformation catalog for the condor_dagman executable, that ends up running the mini dag as one job.

See Also:

Constant Field Values

DAGMAN_PROPERTIES_PREFIX

public static final String DAGMAN_PROPERTIES_PREFIX

The prefix for DAGMan specific properties

See Also:

Constant Field Values

DEFAULT_STAGE_IN_PRIORITY_KEY

public static final int DEFAULT_STAGE_IN_PRIORITY_KEY

The default priority key associated with the stagein jobs.

See Also:

Constant Field Values

DEFAULT_INTER_SITE_PRIORITY_KEY

public static final int DEFAULT_INTER_SITE_PRIORITY_KEY

The default priority key associated with the inter site transfer jobs.

See Also:

Constant Field Values

DEFAULT_CREATE_DIR_PRIORITY_KEY

public static final int DEFAULT_CREATE_DIR_PRIORITY_KEY

The default priority key associated with the create dir jobs.

See Also:

Constant Field Values

DEFAULT_CHMOD_PRIORITY_KEY

public static final int DEFAULT_CHMOD_PRIORITY_KEY

The default priority key associated with chmod jobs.

See Also:

Constant Field Values

DEFAULT_STAGE_OUT_PRIORITY_KEY

public static final int DEFAULT_STAGE_OUT_PRIORITY_KEY

The default priority key associated with the stage out jobs.

See Also:

Constant Field Values

DEFAULT_REPLICA_REG_PRIORITY_KEY

public static final int DEFAULT_REPLICA_REG_PRIORITY_KEY

The default priority key associated with the replica registration jobs.

See Also:

Constant Field Values

DEFAULT_CLEANUP_PRIORITY_KEY

public static final int DEFAULT_CLEANUP_PRIORITY_KEY

The default priority key associated with the cleanup jobs.

See Also:

Constant Field Values

mTCHandle

protected TransformationCatalog mTCHandle

Handle to the Transformation Catalog.

mSiteStore

private SiteStore mSiteStore

The handle to the site catalog store.

mPoolClass

protected String mPoolClass

Specifies the implementing class for the pool interface. Contains the name of the class that implements the pool interface the user has asked at runtime.

mDagWriter

protected PrintWriter mDagWriter

The file handle to the .dag file. A part of the dag file is printed as we write the submit files, to insert the appropriate postscripts for handling exit codes.

mTempLogFile

protected String mTempLogFile

The name of the log file in the /tmp directory

mDone

protected boolean mDone

A boolean indicating whether the files have been generated or not.

mConcreteWorkflow

protected ADag mConcreteWorkflow

The workflow for which the code has to be generated.

mStyleFactory

protected CondorStyleFactory mStyleFactory

Handle to the Style factory, that is used for this workflow.

mGridStartFactory

protected GridStartFactory mGridStartFactory

The handle to the GridStart Factory.

mInitializeGridStart

protected boolean mInitializeGridStart

A boolean indicating whether grid start has been initialized or not.

mCondorVersion

private long mCondorVersion

The long value of condor version.

mAssignDefaultJobPriorities

private boolean mAssignDefaultJobPriorities

Boolean indicating whether to assign job priorities or not.

Constructor Detail

CondorGenerator

public CondorGenerator()

The default constructor.

Method Detail

initialize

public void initialize(PegasusBag bag)
                throws CodeGeneratorException

Initializes the Code Generator implementation. Initializes the various writers.

Specified by:

initialize in interface CodeGenerator

Overrides:

initialize in class Abstract

Parameters:

bag - the bag of initialization objects.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

generateCode

public Collection<File> generateCode(ADag dag)
                              throws CodeGeneratorException

Generates the code for the concrete workflow in Condor DAGMAN and CondorG input format. Returns only the File object for the DAG file that is written out.

Parameters:

dag - the concrete workflow.

Returns:

the Collection of File objects for the files written out.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

generateCode

public void generateCode(ADag dag,
                         Job job)
                  throws CodeGeneratorException

Generates the code (condor submit file) for a single job.

Parameters:

dag - the dag of which the job is a part of.

job - the Job object holding the information about that particular job.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

startMonitoring

public boolean startMonitoring()
                        throws org.griphyn.vdl.euryale.VTorInUseException

Starts monitoring of the workflow by invoking a workflow monitor daemon tailstatd. The tailstatd is picked up from the default path of $PEGASUS_HOME/bin/tailstatd.

Specified by:

startMonitoring in interface CodeGenerator

Overrides:

startMonitoring in class Abstract

Returns:

boolean indicating whether could successfully start the monitor daemon or not.

Throws:

org.griphyn.vdl.euryale.VTorInUseException - in case the method is called before the submit files have been generated.

reset

public void reset()
           throws CodeGeneratorException

Resets the Code Generator implementation.

Specified by:

reset in interface CodeGenerator

Overrides:

reset in class Abstract

Throws:

CodeGeneratorException - in case of any error occuring code generation.

constructMap

private Map constructMap(String propValue)

Constructs a map with the numbers/values to be passed in the RSL handle for certain pools. The user ends up specifying these through the properties file. The value of the property is of the form poolname1=value,poolname2=value....

Parameters:

propValue - the value of the property got from the properties file.

Returns:

Map

constructDAGJob

protected Job constructDAGJob(String name,
                              String directory,
                              String dagBasename)

Constructs a job that plans and submits the partitioned workflow, referred to by a Partition. The main job itself is a condor dagman job that submits the concrete workflow. The concrete workflow is generated by running the planner in the prescript for the job.

Parameters:

name - the name to be assigned to the job.

directory - the submit directory where the submit files for the partition should reside. this is where the dag file is created

dagBasename - the basename of the dag file created.

Returns:

the constructed DAG job.

defaultTCEntry

private TransformationCatalogEntry defaultTCEntry(String site)

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Parameters:

site - the site for which the default entry is required.

Returns:

the default entry.

constructTCEntryFromEnvironment

private TransformationCatalogEntry constructTCEntryFromEnvironment()

Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION

Returns:

the constructed entry else null.

constructTCEntryFromEnvProfiles

private TransformationCatalogEntry constructTCEntryFromEnvProfiles(ENV env)

Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION

Parameters:

env - the environment profiles.

Returns:

the entry constructed else null if environment variables not defined.

getBasename

protected String getBasename(String prefix,
                             String suffix)

A covenience method to construct the basename.

Parameters:

prefix - the first half of basename

suffix - the latter half of basename

Returns:

basename

initializeDagFileWriter

protected File initializeDagFileWriter(String filename,
                                       DagInfo dinfo)
                                throws CodeGeneratorException

Initializes the file handler to the dag file and writes the header to it.

Parameters:

filename - basename of dag file to be written.

dinfo - object containing daginfo of type DagInfo .

Returns:

the File object for the DAG file.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

getCategoryDAGManKnobs

protected String getCategoryDAGManKnobs(PegasusProperties properties)

Write out the DAGMan knobs for each category the user mentions in the properties.

Parameters:

properties - the pegasus properties

Returns:

the String

writeDOTFile

protected void writeDOTFile(String filename,
                            ADag dag)
                     throws CodeGeneratorException

Writes out the DOT file in the submit directory.

Parameters:

filename - basename of dot file to be written .

dag - the ADag object.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

writeJobMapFile

protected void writeJobMapFile(String filename,
                               ADag dag)
                        throws CodeGeneratorException

Writes out the job map file in the submit directory.

Parameters:

filename - basename of dot file to be written .

dag - the ADag object.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

writeDagFileTail

protected void writeDagFileTail(DagInfo dinfo)
                         throws CodeGeneratorException

It writes the relations making up the DAG in the dag file and and closes the file handle to it.

Parameters:

dinfo - object containing daginfo of type DagInfo.

Throws:

CodeGeneratorException

printDagString

protected void printDagString(String str)
                       throws CodeGeneratorException

Writes a string to the dag file. When calling this function the file handle to file is already initialized.

Parameters:

str - The String to be printed to the dag file.

Throws:

CodeGeneratorException

getCondorLogInTmpDirectory

protected String getCondorLogInTmpDirectory()

Returns the name of Condor log file in a tmp directory that is created if generation of symlink for condor logs is turned on.

Returns:

the name of the log file.

getCondorLogInSubmitDirectory

protected String getCondorLogInSubmitDirectory()

Returns the path to the condor log file in the submit directory. It can be a symlink.

Parameters:

dag - the concrete workflow.

Returns:

the path to condor log file in the submit directory.

getCondorLogInSubmitDirectory

protected String getCondorLogInSubmitDirectory(ADag dag)

Returns the path to the condor log file in the submit directory. It can be a symlink.

Parameters:

dag - the concrete workflow.

Returns:

the path to condor log file in the submit directory.

getAdditionalBraindumpEntries

public Map<String,String> getAdditionalBraindumpEntries(ADag workflow)

Returns a Map containing additional braindump entries that are specific to a Code Generator.

Specified by:

getAdditionalBraindumpEntries in class Abstract

Parameters:

workflow - the executable workflow

Returns:

Map containing entries for dag and condor_log

generateLogFileSymlink

protected boolean generateLogFileSymlink(String logFile,
                                         String symlink)

This method generates a symlink to the actual log file written in the local temp directory. The symlink is placed in the dag directory.

Parameters:

logFile - the full path to the log file.

symlink - the full path to the symlink.

Returns:

boolean indicating if creation of symlink was successful or not

getDAGMANOutFilename

protected String getDAGMANOutFilename(ADag dag)

Returns the basename of the file, that contains the output of the dagman while running the dag generated for the workflow. The basename of the .out file is dependant on whether the basename prefix has been specified at runtime or not by command line options.

Parameters:

dag - the DAG containing the concrete workflow

Returns:

the name of the dagfile.

concreteDagEmpty

protected void concreteDagEmpty(String filename,
                                ADag dag)
                         throws CodeGeneratorException

A callout method that dictates what needs to be done in case the concrete plan that is generated is empty. It just logs a message saying the plan is empty.

Parameters:

filename - Filename of the dag to be written of type String.

dag - the concrete dag that is empty.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

handleCondorVarForJob

protected void handleCondorVarForJob(Job job)
                              throws CodeGeneratorException

It updates/adds the condor variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally. In case of clashes of Condor variables from various sources the following order is followed,property file, pool config file and then dax.

Parameters:

job - The Job object containing the information about the job.

Throws:

CodeGeneratorException

populatePeriodicReleaseAndRemoveInJob

public void populatePeriodicReleaseAndRemoveInJob(Job job)

Populates the periodic release and remove values in the job. If an integer value is specified it is used to construct the default expression, else the value specified in the profiles is used as is. The default expression for periodic_release and periodic_remove is

   periodic_release = (NumSystemHolds <= releasevalue)
   periodic_remove = (NumSystemHolds > removevalue)
 

where releasevalue is value of condor profile periodic_release and removevalue is value of condor profile periodic_remove

Parameters:

job - the job object.

getNaturalNumberValue

protected int getNaturalNumberValue(String value)

Returns a natural number value ( > 0 ) if the parameter passed is an integer and greater than zero, else -1

Parameters:

value - the String passed

Returns:

applyStyle

protected void applyStyle(Job job,
                          PrintWriter writer)
                   throws CodeGeneratorException

Applies a submit file style to the job, according to the fact whether the job has to be submitted directly to condor or to a remote jobmanager via CondorG and GRAM. If no style is associated with the job, then for the job running on local site, condor style is applied. For a job running on non local sites, globus style is applied if none is associated with the job.

Parameters:

job - the job on which the style needs to be applied.

writer - the PrintWriter stream to the submit file for the job.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

handleEnvVarForJob

protected void handleEnvVarForJob(Job sinfo)

It updates/adds the environment variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally. In case of clashes of environment variables from various sources the following order is followed,property file, transformation catalog, pool config file and then dax. At present values are not picked from the properties file.

Parameters:

job - The Job object containing the information about the job.

handleGlobusRSLForJob

protected void handleGlobusRSLForJob(Job sinfo)

It updates/adds the the Globus RSL parameters got through the dax that are in Job object. In addition inserts the additional rsl attributes that can be specified in the properties file or the pool config files in the profiles tags. In case of clashes of RSL attributes from various sources the following order is followed,property file, pool config file and then dax.

Parameters:

job - The Job object containing the information about the job.

getJobPriority

protected int getJobPriority(Job job,
                             int depth)

Computes the priority for a job based on job type and depth in the workflow

Parameters:

job - the job whose priority needs to be computed

depth - the depth in the workflow

Returns:

gridstart

private String gridstart(PrintWriter writer,
                         Job job,
                         boolean isGlobusJob)
                  throws CodeGeneratorException

This function creates the stdio handling with and without gridstart. Please note that gridstart will become the default by end 2003, and no gridstart support will be phased out.

Parameters:

writer - is an open stream for the Condor submit file.

job - is the job information structure.

isGlobusJob - is true, if the job generated a line universe = globus, and thus runs remotely. Set to false, if the job runs on the submit host in any way.

Returns:

A possibly empty string which contains things that need to be added to the "globusrsl" clause. The return value is only of interest for isGlobusJob==true calls.

Throws:

CodeGeneratorException - in case of any error occuring code generation.