edu.isi.pegasus.planner.client
Class CPlanner

java.lang.Object
  edu.isi.pegasus.planner.client.Executable
      edu.isi.pegasus.planner.client.CPlanner

public class CPlanner
extends Executable

This is the main program for the Pegasus. It parses the options specified by the user and calls out to the appropriate components to parse the abstract plan, concretize it and then write the submit files.

Version:

$Revision: 5255 $

Author:

Gaurang Mehta, Karan Vahi

Field Summary

static String	CLEANUP_DIR The basename of the directory that contains the submit files for the cleanup DAG that for the concrete dag generated for the workflow.
static String	DEFAULT_MEGADAG_MODE The default megadag mode that is used for generation of megadags in deferred planning.
private static String	EMPTY_FINAL_WORKFLOW_MESSAGE The final successful message that is to be logged.
static String	JAVA_COMMAND_LINE_PROPERTY_REGEX The regex used to match against a java property that is set using -Dpropertyname=value in the argument string
private PegasusBag	mBag The object containing the bag of pegasus objects
private NumberFormat	mNumFormatter The number formatter to format the run submit dir entries.
private PlannerMetrics	mPMetrics The PlannerMetrics object storing the metrics about this planning instance.
private PlannerOptions	mPOptions The object containing all the options passed to the Concrete Planner.
private String	mUser The user name of the user running Pegasus.
static String	NOOP_PREFIX The prefix for the NoOP jobs that are created.
static String	PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY The name of the property key that determines whether pegasus-run should monitord or not.
private static String	SUCCESS_MESSAGE The message to be logged in case of empty executable workflow.

Fields inherited from class edu.isi.pegasus.planner.client.Executable

mLogger, mLogMsg, mProps, mVersion

Constructor Summary

CPlanner()
Default constructor.

CPlanner(LogManager logger)
The overload constructor.

Method Summary

protected void	construct(Job job, String key, String value) Constructs a condor variable in the condor profile namespace associated with the job.
protected Job	createNoOPJob(String name) It creates a NoOP job that runs on the submit host.
protected boolean	createSymbolicLink(String source, String destination, File directory, boolean logErrorToDebug) This method generates a symlink between two files
protected String	determineRelativeSubmitDirectory(ADag dag, String dir, String user, String vogroup, boolean timestampBased) Creates the submit directory for the workflow.
protected String	determineRelativeSubmitDirectory(String label, String dir, String user, String vogroup, boolean timestampBased) Creates the submit directory for the workflow.
private String	determineRootWorkflowUUID(ADag dag, PlannerOptions options, PegasusProperties properties) Determines the workflow uuid for a workflow
protected String	doBackupAndCreateSymbolicLinkForSubmitDirectory(String baseDir, String relativeSubmitDir)
private Collection<File>	doDeferredPlanning() This ends up invoking the deferred planning code, that generates the MegaDAG that is used to submit the partitioned daxes in layers.
protected void	doPartitionAndPlan(PegasusProperties properties, PlannerOptions options) Partitions and plans the workflow.
void	executeCommand() Executes the command on the basis of the options specified.
Collection<File>	executeCommand(PlannerOptions options) Executes the command on the basis of the options specified.
private void	generateStampedeEventsForAbstractWorkflow(ADag workflow, PegasusBag bag) Generates events for the abstract workflow.
gnu.getopt.LongOpt[]	generateValidOptions() Tt generates the LongOpt which contain the valid options that the command will accept.
protected String	getDAGFilename(ADag dag, PlannerOptions options) Returns the basename of the dag file
String	getNOOPJobName(ADag dag) Returns the name of the noop job.
private String	getPegasusRunInvocation() Returns the pegasus-run invocation on the workflow planned.
protected String	getRandomDirectory(ADag dag) Sets the basename of the random directory that is created on the remote sites per workflow.
protected boolean	handleRescueDAG(ADag dag, PlannerOptions options) Checks for rescue dags, and determines whether to plan or not.
protected boolean	handleRescueDAG(String dag, String dir, int numOfRescues) Checks for rescue dags, and determines whether to submit a rescue dag or not.
void	initialize(String[] opts, char confChar) Initialize the executable object
void	loadProperties() Loads all the properties that are needed by this class.
private SiteStore	loadSiteStore(Set<String> sites) Loads the sites from the site catalog into the site store
private void	logIDHierarchyMessage(ADag dag, String parentType, String parentID) Logs a message that connects the jobs with DAX/DAG
private void	logSuccessfulCompletion(boolean emptyWorkflow) Logs the successful completion message.
static void	main(String[] args) The main program for the CPlanner.
PlannerOptions	parseCommandLineArguments(String[] args) Parses the command line arguments using GetOpt and returns a PlannerOptions contains all the options passed by the user at the command line.
PlannerOptions	parseCommandLineArguments(String[] args, boolean sanitizePath) Parses the command line arguments using GetOpt and returns a PlannerOptions contains all the options passed by the user at the command line.
void	printLongVersion() Prints the long description, displaying in detail what the various options to the command stand for.
void	printShortVersion() Prints out a short description of what the command does.
protected static void	sanityCheck(File dir) Checks the destination location for existence, if it can be created, if it is writable etc.
boolean	submitWorkflow(String invocation) Submits the workflow for execution using pegasus-run, a wrapper around pegasus-submit-dag.
protected boolean	writeOutMetrics(PlannerMetrics pm) Writes out the planner metrics to the global log.

Methods inherited from class edu.isi.pegasus.planner.client.Executable

convertException, convertException, getCommandLineOptions, getEnvValue, getGVDSVersion, initialize, log, lookupConfProperty, sanityCheckOnProperties, setupLogging

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_MEGADAG_MODE

public static final String DEFAULT_MEGADAG_MODE

The default megadag mode that is used for generation of megadags in deferred planning.

See Also:

Constant Field Values

CLEANUP_DIR

public static final String CLEANUP_DIR

The basename of the directory that contains the submit files for the cleanup DAG that for the concrete dag generated for the workflow.

See Also:

Constant Field Values

NOOP_PREFIX

public static final String NOOP_PREFIX

The prefix for the NoOP jobs that are created.

See Also:

Constant Field Values

PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY

public static final String PEGASUS_MONITORD_LAUNCH_PROPERTY_KEY

The name of the property key that determines whether pegasus-run should monitord or not.

See Also:

Constant Field Values

JAVA_COMMAND_LINE_PROPERTY_REGEX

public static final String JAVA_COMMAND_LINE_PROPERTY_REGEX

The regex used to match against a java property that is set using -Dpropertyname=value in the argument string

See Also:

Constant Field Values

EMPTY_FINAL_WORKFLOW_MESSAGE

private static final String EMPTY_FINAL_WORKFLOW_MESSAGE

The final successful message that is to be logged.

See Also:

Constant Field Values

SUCCESS_MESSAGE

private static final String SUCCESS_MESSAGE

The message to be logged in case of empty executable workflow.

See Also:

Constant Field Values

mPOptions

private PlannerOptions mPOptions

The object containing all the options passed to the Concrete Planner.

mBag

private PegasusBag mBag

The object containing the bag of pegasus objects

mPMetrics

private PlannerMetrics mPMetrics

The PlannerMetrics object storing the metrics about this planning instance.

mNumFormatter

private NumberFormat mNumFormatter

The number formatter to format the run submit dir entries.

mUser

private String mUser

The user name of the user running Pegasus.

Constructor Detail

CPlanner

public CPlanner()

Default constructor.

CPlanner

public CPlanner(LogManager logger)

The overload constructor.

Parameters:

logger - the logger object to use. can be null.

Method Detail

initialize

public void initialize(String[] opts,
                       char confChar)

Description copied from class: Executable

Initialize the executable object

Overrides:

initialize in class Executable

Parameters:

opts - the command line argument passed by the user

confChar - the short option corresponding the conf property.

main

public static void main(String[] args)

The main program for the CPlanner.

Parameters:

args - the main arguments passed to the planner.

loadProperties

public void loadProperties()

Loads all the properties that are needed by this class.

Specified by:

loadProperties in class Executable

executeCommand

public void executeCommand()

Executes the command on the basis of the options specified.

Parameters:

args - the command line options.

executeCommand

public Collection<File> executeCommand(PlannerOptions options)

Executes the command on the basis of the options specified.

Parameters:

options - the command line options.

Returns:

the Collection of File objects for the files written out.

getNOOPJobName

public String getNOOPJobName(ADag dag)

Returns the name of the noop job.

Parameters:

dag - the workflow

Returns:

the name

createNoOPJob

protected Job createNoOPJob(String name)

It creates a NoOP job that runs on the submit host.

Parameters:

name - the name to be assigned to the noop job

Returns:

the noop job.

construct

protected void construct(Job job,
                         String key,
                         String value)

Constructs a condor variable in the condor profile namespace associated with the job. Overrides any preexisting key values.

Parameters:

job - contains the job description.

key - the key of the profile.

value - the associated value.

parseCommandLineArguments

public PlannerOptions parseCommandLineArguments(String[] args)

Parses the command line arguments using GetOpt and returns a PlannerOptions contains all the options passed by the user at the command line.

Parameters:

args - the arguments passed by the user at command line.

Returns:

the options.

parseCommandLineArguments

public PlannerOptions parseCommandLineArguments(String[] args,
                                                boolean sanitizePath)

Parses the command line arguments using GetOpt and returns a PlannerOptions contains all the options passed by the user at the command line.

Parameters:

args - the arguments passed by the user at command line.

sanitizePath - whether to sanitize path during construction of options

Returns:

the options.

submitWorkflow

public boolean submitWorkflow(String invocation)

Submits the workflow for execution using pegasus-run, a wrapper around pegasus-submit-dag.

Parameters:

invocation - the pegasus run invocation

Returns:

boolean indicating whether could successfully submit the workflow or not.

doPartitionAndPlan

protected void doPartitionAndPlan(PegasusProperties properties,
                                  PlannerOptions options)

Partitions and plans the workflow. First step of merging DAGMan and Condor

Parameters:

properties - the properties passed to the planner.

options - the options passed to the planner.

getRandomDirectory

protected String getRandomDirectory(ADag dag)

Sets the basename of the random directory that is created on the remote sites per workflow. The name is generated by default from teh flow ID, unless a basename prefix is specifed at runtime in the planner options.

Parameters:

dag - the DAG containing the abstract workflow.

Returns:

the basename of the random directory.

generateValidOptions

public gnu.getopt.LongOpt[] generateValidOptions()

Tt generates the LongOpt which contain the valid options that the command will accept.

Specified by:

generateValidOptions in class Executable

Returns:

array of LongOpt objects , corresponding to the valid options

printShortVersion

public void printShortVersion()

Prints out a short description of what the command does.

Specified by:

printShortVersion in class Executable

printLongVersion

public void printLongVersion()

Prints the long description, displaying in detail what the various options to the command stand for.

Specified by:

printLongVersion in class Executable

determineRootWorkflowUUID

private String determineRootWorkflowUUID(ADag dag,
                                         PlannerOptions options,
                                         PegasusProperties properties)

Determines the workflow uuid for a workflow

Parameters:

dag - the workflow

options - the options passed to the planner

properties - the properties passed to the planner

Returns:

uuid for the root workflow instance

doDeferredPlanning

private Collection<File> doDeferredPlanning()

This ends up invoking the deferred planning code, that generates the MegaDAG that is used to submit the partitioned daxes in layers.

Returns:

the Collection of File objects for the files written out.

determineRelativeSubmitDirectory

protected String determineRelativeSubmitDirectory(ADag dag,
                                                  String dir,
                                                  String user,
                                                  String vogroup,
                                                  boolean timestampBased)
                                           throws IOException

Creates the submit directory for the workflow. This is not thread safe.

Parameters:

dag - the workflow being worked upon.

dir - the base directory specified by the user.

user - the username of the user.

vogroup - the vogroup to which the user belongs to.

timestampBased - boolean indicating whether to have a timestamp based dir or not

Returns:

the directory name created relative to the base directory passed as input.

Throws:

IOException - in case of unable to create submit directory.

determineRelativeSubmitDirectory

protected String determineRelativeSubmitDirectory(String label,
                                                  String dir,
                                                  String user,
                                                  String vogroup,
                                                  boolean timestampBased)
                                           throws IOException

Creates the submit directory for the workflow. This is not thread safe.

Parameters:

label - the label of the workflow

dir - the base directory specified by the user.

user - the username of the user.

vogroup - the vogroup to which the user belongs to.

timestampBased - boolean indicating whether to have a timestamp based dir or not

Returns:

the directory name created relative to the base directory passed as input.

Throws:

IOException - in case of unable to create submit directory.

sanityCheck

protected static void sanityCheck(File dir)
                           throws IOException

Checks the destination location for existence, if it can be created, if it is writable etc.

Parameters:

dir - is the new base directory to optionally create.

Throws:

IOException - in case of error while writing out files.

writeOutMetrics

protected boolean writeOutMetrics(PlannerMetrics pm)

Writes out the planner metrics to the global log.

Parameters:

pm - the metrics to be written out.

Returns:

boolean

getDAGFilename

protected String getDAGFilename(ADag dag,
                                PlannerOptions options)

Returns the basename of the dag file

Parameters:

dag - the dag that was parsed.

options - the planner options

Returns:

boolean true means submit the rescue false do the planning operation

handleRescueDAG

protected boolean handleRescueDAG(ADag dag,
                                  PlannerOptions options)

Checks for rescue dags, and determines whether to plan or not.

Parameters:

dag - the dag that was parsed.

options - the planner options

Returns:

boolean true means submit the rescue false do the planning operation

handleRescueDAG

protected boolean handleRescueDAG(String dag,
                                  String dir,
                                  int numOfRescues)

Checks for rescue dags, and determines whether to submit a rescue dag or not.

Parameters:

dag - the dag file for the dax

dir - the submit directory.

numOfRescues - the number of rescues to handle.

Returns:

true means submit the rescue false do the planning operation

createSymbolicLink

protected boolean createSymbolicLink(String source,
                                     String destination,
                                     File directory,
                                     boolean logErrorToDebug)

This method generates a symlink between two files

Parameters:

source - the file that has to be symlinked

destination - the destination of the symlink

directory - the directory in which to execute the command

logErrorToDebug - whether to log messeage to debug or not

Returns:

boolean indicating if creation of symlink was successful or not

generateStampedeEventsForAbstractWorkflow

private void generateStampedeEventsForAbstractWorkflow(ADag workflow,
                                                       PegasusBag bag)

Generates events for the abstract workflow.

Parameters:

workflow - the parsed dax

bag - the initialized object bag

loadSiteStore

private SiteStore loadSiteStore(Set<String> sites)

Loads the sites from the site catalog into the site store

Parameters:

sites -

Returns:

SiteStore object containing the information about the sites.

logIDHierarchyMessage

private void logIDHierarchyMessage(ADag dag,
                                   String parentType,
                                   String parentID)

Logs a message that connects the jobs with DAX/DAG

Parameters:

dag - the DAG object

parentType - the parent type

parentID - the parent id

logSuccessfulCompletion

private void logSuccessfulCompletion(boolean emptyWorkflow)

Logs the successful completion message.

Parameters:

emptyWorkflow - indicates whether the workflow created was empty or not.

getPegasusRunInvocation

private String getPegasusRunInvocation()

Returns the pegasus-run invocation on the workflow planned.

Returns:

the pegasus-run invocation

doBackupAndCreateSymbolicLinkForSubmitDirectory

protected String doBackupAndCreateSymbolicLinkForSubmitDirectory(String baseDir,
                                                                 String relativeSubmitDir)
                                                          throws IOException

Throws:

IOException