edu.isi.pegasus.planner.code.gridstart
Class Kickstart

java.lang.Object
  edu.isi.pegasus.planner.code.gridstart.Kickstart

All Implemented Interfaces:

GridStart

public class Kickstart
extends Object
implements GridStart

This enables a constituentJob to be run on the grid, by launching it through kickstart. The kickstart executable is a light-weight program which connects the stdin, stdout and stderr filehandles for Pegasus jobs on the remote site.

Sitting in between the remote scheduler and the executable, it is possible for kickstart to gather additional information about the executable run-time behavior, including the exit status of jobs.

Kickstart is an executable distributed with Pegasus that can generally be found at $PEGASUS_HOME/bin/kickstart

Version:

$Revision: 5152 $

Author:

Karan Vahi vahi@isi.edu

Field Summary

static String	CLASSNAME The basename of the class that is implmenting this.
static String	COMPLETE_TRANSFORMATION_NAME The complete TC name for kickstart.
static String	EXECUTABLE_BASENAME The basename of the kickstart executable.
static String	KICKSTART_CLEANUP The environment variable used to the set Kickstart CLEANUP JOB.
static String	KICKSTART_INPUT_SUFFIX The suffix for the kickstart input file, that is generated to use invoke at the remote end.
static String	KICKSTART_POSTJOB The environment variable used to the set Kickstart POSTJOB.
static String	KICKSTART_PREJOB The environment variable used to the set Kickstart PREJOB.
static String	KICKSTART_SETUP The environment variable used to the set Kickstart SETUP JOB.
private ADag	mConcDAG The handle to the workflow that is being enabled.
private boolean	mDisableInvokeFunctionality Boolean indicating whether to disable invoke functionality.
private boolean	mDoStat A boolean indicating whether to stat files or not.
private boolean	mDynamicDeployment A boolean indicating whether kickstart is deployed dynamically or not.
private boolean	mEnablingPartOfAggregatedJob An instance variable to track if enabling is happening as part of a clustered constituentJob.
private boolean	mGenerateLOF A boolean indicating whether to generate lof files or not.
private boolean	mInvokeAlways A boolean indicating whether to use invoke always or not.
private long	mInvokeLength The invoke limit trigger.
private String	mKickstartLabel The label that is passed to kickstart.
private LogManager	mLogger The LogManager object which is used to log all the messages.
private GridStart	mNoGridStartImpl Handle to NoGridStart implementation.
private PlannerOptions	mPOptions The options passed to the planner.
private PegasusProperties	mProps The object holding all the properties pertaining to Pegasus.
private boolean	mSetXBit Whether kickstart should set the X Bit on the staged executables.
private SiteStore	mSiteStore Handle to the site catalog store.
private SLS	mSLS The handle to the SLS implementor
private String	mSubmitDir The submit exectionSiteDirectory where the submit files are being generated for the workflow.
private TransformationCatalog	mTCHandle Handle to Transformation Catalog.
private boolean	mUseFullPathToGridStart Boolean indicating whether to use full path or not
(package private) boolean	mWorkerNodeExecution A boolean indicating whether to have worker node execution or not.
static String	SHORT_NAME The SHORTNAME for this implementation.
static String	TRANSFORMATION_NAME The logical name of kickstart
static String	TRANSFORMATION_NAMESPACE The transformation namespace for the kickstart
static String	TRANSFORMATION_VERSION The version number for kickstart.

Fields inherited from interface edu.isi.pegasus.planner.code.GridStart

mSeparator, VERSION

Constructor Summary

Kickstart()

Method Summary

private void	addCleanupPostScript(Job job, List files) Adds a /bin/rm post constituentJob to kickstart that removes the files passed.
boolean	canSetXBit() Indicates whether the enabling mechanism can set the X bit on the executable on the remote grid site, in addition to launching it on the remote grid site.
private void	construct(Job job, String key, String value) Constructs a condor variable in the condor profile namespace associated with the constituentJob.
protected String	constructCleanupJob(Job job, String workerNodeTmp) Constructs a kickstart setup constituentJob
protected String	constructPREJob(Job job, String headNodeURLPrefix, String headNodeDirectory, String workerNodeDirectory, String slsFile) Constructs the prejob that fetches sls file, and then invokes transfer again.
protected String	constructSetupJob(Job job, String workerNodeTmp) Constructs a kickstart setup constituentJob
String	defaultPOSTScript() Returns the SHORT_NAME for the POSTScript implementation that is used to be as default with this GridStart implementation.
boolean	enable(AggregatedJob job, boolean isGlobusJob) Enables a constituentJob to run on the grid.
boolean	enable(Job job, boolean isGlobusJob) Enables a constituentJob to run on the grid by launching it through kickstart.
protected boolean	enable(Job job, boolean isGlobusJob, boolean stat, boolean addPostScript, boolean partOfClusteredJob) Enables a constituentJob to run on the grid by launching it through kickstart.
String	generateListofFilenamesFile(Set files, String basename) Writes out the list of filenames file for the constituentJob.
protected String	getDirectory(Job job) Returns the directory in which the job should run.
private String	getDirectoryKey(Job job) Returns the exectionSiteDirectory that is associated with the constituentJob to specify the exectionSiteDirectory in which the constituentJob needs to run
String	getKickstartPath(SiteCatalogEntry site) Returns the default path to kickstart as constructed from the environment variable associated with a site in the site catalog
TransformationCatalogEntry	getTransformationCatalogEntry(String site) Returns the transformation catalog entry for kickstart on a site
String	getVDSKeyValue() Returns the value of the vds profile with key as Pegasus.GRIDSTART_KEY, that would result in the loading of this particular implementation.
String	getWorkerNodeDirectory(Job job) Returns the exectionSiteDirectory in which the constituentJob executes on the worker node.
protected String	handleTransferOfExecutable(Job job, String path) It changes the paths to the executable depending on whether we want to transfer the executable or not.
void	initialize(PegasusBag bag, ADag dag) Initializes the GridStart implementation.
private String	quote(String string) Condor Quotes a string
protected boolean	requiresToSetDirectory(Job job) Returns a boolean indicating whether we need to set the directory for the job or not.
String	shortDescribe() Returns a short textual description in the form of the name of the class.
void	useFullPathToGridStarts(boolean fullPath) Setter method to control whether a full path to Gridstart should be returned while wrapping a job or not.
private boolean	useInvoke(Job job, String executable, StringBuffer args) Triggers the creation of the kickstart input file, that contains the the remote executable and the arguments with which it has to be invoked.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TRANSFORMATION_NAMESPACE

public static final String TRANSFORMATION_NAMESPACE

The transformation namespace for the kickstart

See Also:

Constant Field Values

TRANSFORMATION_NAME

public static final String TRANSFORMATION_NAME

The logical name of kickstart

See Also:

Constant Field Values

TRANSFORMATION_VERSION

public static final String TRANSFORMATION_VERSION

The version number for kickstart.

EXECUTABLE_BASENAME

public static final String EXECUTABLE_BASENAME

The basename of the kickstart executable.

See Also:

Constant Field Values

COMPLETE_TRANSFORMATION_NAME

public static final String COMPLETE_TRANSFORMATION_NAME

The complete TC name for kickstart.

KICKSTART_INPUT_SUFFIX

public static final String KICKSTART_INPUT_SUFFIX

The suffix for the kickstart input file, that is generated to use invoke at the remote end.

See Also:

Constant Field Values

CLASSNAME

public static final String CLASSNAME

The basename of the class that is implmenting this. Could have been determined by reflection.

See Also:

Constant Field Values

SHORT_NAME

public static final String SHORT_NAME

The SHORTNAME for this implementation.

See Also:

Constant Field Values

KICKSTART_SETUP

public static final String KICKSTART_SETUP

The environment variable used to the set Kickstart SETUP JOB.

See Also:

Constant Field Values

KICKSTART_PREJOB

public static final String KICKSTART_PREJOB

The environment variable used to the set Kickstart PREJOB.

See Also:

Constant Field Values

KICKSTART_POSTJOB

public static final String KICKSTART_POSTJOB

The environment variable used to the set Kickstart POSTJOB.

See Also:

Constant Field Values

KICKSTART_CLEANUP

public static final String KICKSTART_CLEANUP

The environment variable used to the set Kickstart CLEANUP JOB.

See Also:

Constant Field Values

mLogger

private LogManager mLogger

The LogManager object which is used to log all the messages.

mProps

private PegasusProperties mProps

The object holding all the properties pertaining to Pegasus.

mPOptions

private PlannerOptions mPOptions

The options passed to the planner.

mConcDAG

private ADag mConcDAG

The handle to the workflow that is being enabled.

mSiteStore

private SiteStore mSiteStore

Handle to the site catalog store.

mTCHandle

private TransformationCatalog mTCHandle

Handle to Transformation Catalog.

mSubmitDir

private String mSubmitDir

The submit exectionSiteDirectory where the submit files are being generated for the workflow.

mInvokeAlways

private boolean mInvokeAlways

A boolean indicating whether to use invoke always or not.

mDoStat

private boolean mDoStat

A boolean indicating whether to stat files or not.

mGenerateLOF

private boolean mGenerateLOF

A boolean indicating whether to generate lof files or not.

mInvokeLength

private long mInvokeLength

The invoke limit trigger.

mWorkerNodeExecution

boolean mWorkerNodeExecution

A boolean indicating whether to have worker node execution or not.

mSLS

private SLS mSLS

The handle to the SLS implementor

mEnablingPartOfAggregatedJob

private boolean mEnablingPartOfAggregatedJob

An instance variable to track if enabling is happening as part of a clustered constituentJob. See Bug 21 comments on Pegasus Bugzilla

mDynamicDeployment

private boolean mDynamicDeployment

A boolean indicating whether kickstart is deployed dynamically or not.

mKickstartLabel

private String mKickstartLabel

The label that is passed to kickstart.

mSetXBit

private boolean mSetXBit

Whether kickstart should set the X Bit on the staged executables.

mNoGridStartImpl

private GridStart mNoGridStartImpl

Handle to NoGridStart implementation.

mUseFullPathToGridStart

private boolean mUseFullPathToGridStart

Boolean indicating whether to use full path or not

mDisableInvokeFunctionality

private boolean mDisableInvokeFunctionality

Boolean indicating whether to disable invoke functionality.

Constructor Detail

Kickstart

public Kickstart()

Method Detail

initialize

public void initialize(PegasusBag bag,
                       ADag dag)

Initializes the GridStart implementation.

Specified by:

initialize in interface GridStart

Parameters:

bag - the bag of objects that is used for initialization.

dag - the concrete dag so far.

useFullPathToGridStarts

public void useFullPathToGridStarts(boolean fullPath)

Setter method to control whether a full path to Gridstart should be returned while wrapping a job or not.

Specified by:

useFullPathToGridStarts in interface GridStart

Parameters:

fullPath - if set to true, indicates that full path would be used.

enable

public boolean enable(AggregatedJob job,
                      boolean isGlobusJob)

Enables a constituentJob to run on the grid. This also determines how the stdin,stderr and stdout of the constituentJob are to be propogated. To grid enable a constituentJob, the constituentJob may need to be wrapped into another constituentJob, that actually launches the constituentJob. It usually results in the constituentJob description passed being modified modified.

Specified by:

enable in interface GridStart

Parameters:

constituentJob - the Job object containing the constituentJob description of the constituentJob that has to be enabled on the grid.

isGlobusJob - is true, if the constituentJob generated a line universe = globus, and thus runs remotely. Set to false, if the constituentJob runs on the submit host in any way.

Returns:

boolean true if enabling was successful,else false.

enable

public boolean enable(Job job,
                      boolean isGlobusJob)

Enables a constituentJob to run on the grid by launching it through kickstart. Does the stdio, and stderr handling of the constituentJob to be run on the grid. It modifies the constituentJob description, and also constructs all the valid option to be passed to kickstart for launching the executable.

Specified by:

enable in interface GridStart

Parameters:

constituentJob - the Job object containing the constituentJob description of the constituentJob that has to be enabled on the grid.

isGlobusJob - is true, if the constituentJob generated a line universe = globus, and thus runs remotely. Set to false, if the constituentJob runs on the submit host in any way.

Returns:

boolean true if enabling was successful,else false in case when the path to kickstart could not be determined on the site where the constituentJob is scheduled.

enable

protected boolean enable(Job job,
                         boolean isGlobusJob,
                         boolean stat,
                         boolean addPostScript,
                         boolean partOfClusteredJob)

Enables a constituentJob to run on the grid by launching it through kickstart. Does the stdio, and stderr handling of the constituentJob to be run on the grid. It modifies the constituentJob description, and also constructs all the valid option to be passed to kickstart for launching the executable.

Parameters:

constituentJob - the Job object containing the constituentJob description of the constituentJob that has to be enabled on the grid.

isGlobusJob - is true, if the constituentJob generated a line universe = globus, and thus runs remotely. Set to false, if the constituentJob runs on the submit host in any way.

stat - boolean indicating whether to generate the lof files for kickstart stat option or not.

addPostScript - boolean indicating whether to add a postscript or not.

partOfClusteredJob - boolean indicating whether the job being enabled is part of a clustered job or not.

Returns:

boolean true if enabling was successful,else false in case when the path to kickstart could not be determined on the site where the constituentJob is scheduled.

handleTransferOfExecutable

protected String handleTransferOfExecutable(Job job,
                                            String path)

It changes the paths to the executable depending on whether we want to transfer the executable or not. If the transfer_executable is set to true, then the executable needs to be shipped from the submit host meaning the local pool. This function changes the path of the executable to the one on the local pool, so that it can be shipped. If the worker package is being deployed dynamically, then the path is set to the exectionSiteDirectory where the worker package is deployed. Else, we pick up the path from the site catalog that is passed as input

Parameters:

constituentJob - the Job containing the constituentJob description.

path - the path to kickstart on the remote compute site, as determined from the site catalog.

Returns:

the path that needs to be set as the executable

getTransformationCatalogEntry

public TransformationCatalogEntry getTransformationCatalogEntry(String site)

Returns the transformation catalog entry for kickstart on a site

Parameters:

site - the site on which the entry is required

Returns:

the entry if found else null

getKickstartPath

public String getKickstartPath(SiteCatalogEntry site)

Returns the default path to kickstart as constructed from the environment variable associated with a site in the site catalog

Parameters:

site - the SiteCatalogEntry object for the site.

Returns:

value if set else null

getWorkerNodeDirectory

public String getWorkerNodeDirectory(Job job)

Returns the exectionSiteDirectory in which the constituentJob executes on the worker node.

Specified by:

getWorkerNodeDirectory in interface GridStart

Parameters:

constituentJob -

Returns:

the full path to the exectionSiteDirectory where the constituentJob executes

canSetXBit

public boolean canSetXBit()

Indicates whether the enabling mechanism can set the X bit on the executable on the remote grid site, in addition to launching it on the remote grid site.

Specified by:

canSetXBit in interface GridStart

Returns:

true indicating Kickstart can set the X bit or not.

getVDSKeyValue

public String getVDSKeyValue()

Returns the value of the vds profile with key as Pegasus.GRIDSTART_KEY, that would result in the loading of this particular implementation. It is usually the name of the implementing class without the package name.

Specified by:

getVDSKeyValue in interface GridStart

Returns:

the value of the profile key.

See Also:

org.griphyn.cPlanner.namespace.Pegasus#GRIDSTART_KEY

shortDescribe

public String shortDescribe()

Returns a short textual description in the form of the name of the class.

Specified by:

shortDescribe in interface GridStart

Returns:

short textual description.

defaultPOSTScript

public String defaultPOSTScript()

Returns the SHORT_NAME for the POSTScript implementation that is used to be as default with this GridStart implementation.

Specified by:

defaultPOSTScript in interface GridStart

Returns:

the identifier for the PegasusExitCode POSTScript implementation.

See Also:

POSTScript.shortDescribe()

requiresToSetDirectory

protected boolean requiresToSetDirectory(Job job)

Returns a boolean indicating whether we need to set the directory for the job or not.

Parameters:

job - the job for which to set directory.

Returns:

getDirectory

protected String getDirectory(Job job)

Returns the directory in which the job should run.

Parameters:

job - the job in which the directory has to run.

Returns:

getDirectoryKey

private String getDirectoryKey(Job job)

Returns the exectionSiteDirectory that is associated with the constituentJob to specify the exectionSiteDirectory in which the constituentJob needs to run

Parameters:

constituentJob - the constituentJob

Returns:

the condor key . can be initialdir or remote_initialdir

useInvoke

private boolean useInvoke(Job job,
                          String executable,
                          StringBuffer args)

Triggers the creation of the kickstart input file, that contains the the remote executable and the arguments with which it has to be invoked. The kickstart input file is created in the submit directory.

Parameters:

constituentJob - the Job object containing the constituentJob description.

executable - the path to the executable used.

args - the arguments buffer for gridstart invocation so far.

Returns:

boolean indicating whether kickstart input file was generated or not. false in case of any error.

constructSetupJob

protected String constructSetupJob(Job job,
                                   String workerNodeTmp)

Constructs a kickstart setup constituentJob

Parameters:

constituentJob - the constituentJob to be run.

workerNodeTmp - the worker node tmp to run the constituentJob in.

Returns:

String

constructCleanupJob

protected String constructCleanupJob(Job job,
                                     String workerNodeTmp)

Constructs a kickstart setup constituentJob

Parameters:

constituentJob - the constituentJob to be run.

workerNodeTmp - the worker node tmp to run the constituentJob in.

Returns:

String

constructPREJob

protected String constructPREJob(Job job,
                                 String headNodeURLPrefix,
                                 String headNodeDirectory,
                                 String workerNodeDirectory,
                                 String slsFile)

Constructs the prejob that fetches sls file, and then invokes transfer again.

Parameters:

constituentJob - the constituentJob for which the prejob is being created

headNodeURLPrefix - String

headNodeDirectory - String

workerNodeDirectory - String

slsFile - String

Returns:

String containing the prescript invocation

generateListofFilenamesFile

public String generateListofFilenamesFile(Set files,
                                          String basename)

Writes out the list of filenames file for the constituentJob.

Parameters:

files - the list of PegasusFile objects contains the files whose stat information is required.

basename - the basename of the file that is to be created

Returns:

the full path to lof file created, else null if no file is written out.

construct

private void construct(Job job,
                       String key,
                       String value)

Constructs a condor variable in the condor profile namespace associated with the constituentJob. Overrides any preexisting key values.

Parameters:

constituentJob - contains the constituentJob description.

key - the key of the profile.

value - the associated value.

quote

private String quote(String string)

Condor Quotes a string

Parameters:

string - the string to be quoted.

Returns:

quoted string.

addCleanupPostScript

private void addCleanupPostScript(Job job,
                                  List files)

Adds a /bin/rm post constituentJob to kickstart that removes the files passed. The post jobs is added as an environment variable.

Parameters:

constituentJob - the constituentJob in which the post constituentJob needs to be added.

files - the files to be deleted.