edu.isi.pegasus.planner.refiner
Class DeployWorkerPackage

java.lang.Object
  edu.isi.pegasus.planner.refiner.Engine
      edu.isi.pegasus.planner.refiner.DeployWorkerPackage

public class DeployWorkerPackage
extends Engine

The refiner that is responsible for adding - setup nodes that deploy a worker package on each deployment site at start of workflow execution - cleanup nodes that undeploy a worker package on each deployment site at end workflow execution

Version:

$Revision: 538 $

Author:

Karan Vahi, Gaurang Mehta

Field Summary

static String	BASE_BUILD_DIRECTORY_URL The base directory URL for the builds.
static String	CLEANUP_PREFIX Constant suffix for the names of the deployment nodes.
static String	COMPLETE_TRANSFORMATION_NAME The complete TC name for pegasus worker package.
static String	COMPLETE_UNTAR_TRANSFORMATION_NAME The complete TC name for untar.
static String	DEFAULT_REFINER The default transfer refiner name.
static String	DEPLOY_WORKER_PREFIX Constant suffix for the names of the deployment nodes.
static String	DERIVATION_NAME The logical name of the transformation for the worker package
static String	DERIVATION_NAMESPACE The derivation namespace for the worker package.
static String	DERIVATION_VERSION The version number for the derivations for worker package.
protected ADag	mCurrentDag It is a reference to the Concrete Dag so far.
protected Map<String,FileTransfer>	mFTMap The FileTransfer map indexed by site id.
protected String	mJobPrefix The job prefix that needs to be applied to the job file basenames.
protected Map<String,Boolean>	mLocalTransfers Map that indicates whether we need local setup transfer jobs for a site or not.
private static Map<SysInfo.OS,String>	mOSToNMIOSReleaseAndVersion The map storing OS to corresponding NMI OS platforms.
private static Pattern	mPattern Stores compiled patterns at first use, quasi-Singleton.
protected String	mPlannerMajorMinorVersion The major minor version that is used to construct the URL for the pegasus website.
private static String	mRegexExpression Store the regular expressions necessary to parse the basename from the worker package url to retrieve the version of pegasus.
protected Implementation	mSetupTransferImplementation The transfer implementation to be used for staging in the data as part of setup job.
protected Map<String,String>	mSiteToPegasusHomeMap Maps a site to the the directory where the pegasus worker package has been untarred during workflow execution.
protected boolean	mTransferWorkerPackage Boolean indicating whether user wants the worker package to be transferred or not.
protected String	mUserSpecifiedSourceLocation The user specified location from where to stage the worker packages.
protected boolean	mUseUserSpecifiedSourceLocation Boolean indicating whether to use the user specified location or not
protected boolean	mWorkerNodeExecution Boolean indicating worker node execution.
static String	PACKAGE_NAME The name of the package in which all the implementing classes are.
static String	PATH_VALUE The path to be set for create dir jobs.
static String	PEGASUS_VERSION The version of pegasus matching the planner.
static String[][]	PEGASUS_WORKER_EXECUTABLES Array storing the names of the executables in the $PEGASUS_HOME/bin directory Associates the transformation name with the executable basenames
static String	TRANSFORMATION_NAME The logical name of the worker package
static String	TRANSFORMATION_NAMESPACE The transformation namespace for the worker package
static String	TRANSFORMATION_VERSION The version number for the worker package.
static String	UNTAR_DERIVATION_NAME The logical name of the transformation for the untar job.
static String	UNTAR_DERIVATION_NAMESPACE The derivation namespace for the untar job.
static String	UNTAR_DERIVATION_VERSION The version number for the derivations for untar job.
static String	UNTAR_PREFIX Constant suffix for the names of the deployment nodes.
static String	UNTAR_TRANSFORMATION_NAME The logical name of the worker package
static String	UNTAR_TRANSFORMATION_NAMESPACE The transformation namespace for the worker package
static String	UNTAR_TRANSFORMATION_VERSION The version number for the worker package.

Fields inherited from class edu.isi.pegasus.planner.refiner.Engine

mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE

Constructor Summary

DeployWorkerPackage(PegasusBag bag)
A pratically nothing constructor !

Method Summary

ADag	addCleanupNodesForWorkerPackage(ADag dag) Adds cleanup nodes in the workflow for sites specified.
private TransformationCatalogEntry	addDefaultTCEntry(String site, String pegasusHome, SysInfo sysinfo, boolean useFullPath, String name, String executable) Returns a default TC entry to be used in case entry is not found in the transformation catalog.
protected TransformationCatalogEntry	addDefaultTCEntryForPegasusWebsite(String site, String name) Returns a default TC entry for the pegasus site.
ADag	addSetupNodes(ADag dag) Adds a setup node per execution site in the workflow that will stage the worker node executables to the workdirectory on the sites the workflow has been scheduled to.
private Graph	addSetupNodesWithoutUntarNodes(ADag dag, Set<String> deploymentSites) Adds only the stage worker nodes to the workflow.
private Graph	addSetupNodesWithUntarNodes(ADag dag, Set<String> deploymentSites) Adds untar nodes to the workflow, in addition to the stage worker nodes
protected String	constructDefaultURLToPegasusWorkerPackage(String name, SysInfo sysinfo) Constructs the default URL's for the pegasus worker package.
private TransformationCatalogEntry	defaultUntarTCEntry(SiteCatalogEntry site) Returns a default TC entry to be used in case entry is not found in the transformation catalog.
protected String	getBasename(String url) Returns the basename of the URL using substring.
protected String	getCleanupJobname(ADag dag, String site) It returns the name of the untar job, that is to be assigned.
protected String	getDeployJobName(ADag dag, String site, boolean localTransfer) It returns the name of the deployment job, that is to be assigned.
protected Set	getDeploymentSites(ADag dag) Retrieves the sites for which the deployment jobs need to be created.
protected String	getRootDirectoryNameForPegasus(String url) Does regex magic to figure out the version of pegasus from the url, and use it to construct the name of pegasus directory, when worker package is untarred.
private String	getStagingSite(String site) Returns the staging site for a particular execution site.
protected String	getUntarJobName(ADag dag, String site) It returns the name of the untar job, that is to be assigned.
void	initialize(ADag scheduledDAG) Initialize with the scheduled graph.
static DeployWorkerPackage	loadDeployWorkerPackage(PegasusBag bag) Loads the implementing class corresponding to the mode specified by the user at runtime.
protected Job	makeUntarJob(String site, String jobName, String wpBasename) It creates a untar job , that untars the worker package that is staged by the setup transfer job.
private static Map<SysInfo.OS,String>	osToOSReleaseAndVersion() Maps each to OS to a specific OS release for purposes of picking up the correct worker package for a site.
boolean	runTransferOnLocalSite(Refiner refiner, String site, String destinationURL, int type) Returns whether to run a transfer job on local site or not.

Methods inherited from class edu.isi.pegasus.planner.refiner.Engine

addVector, appendArrayList, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEPLOY_WORKER_PREFIX

public static final String DEPLOY_WORKER_PREFIX

Constant suffix for the names of the deployment nodes.

See Also:

Constant Field Values

UNTAR_PREFIX

public static final String UNTAR_PREFIX

Constant suffix for the names of the deployment nodes.

See Also:

Constant Field Values

CLEANUP_PREFIX

public static final String CLEANUP_PREFIX

Constant suffix for the names of the deployment nodes.

See Also:

Constant Field Values

PEGASUS_WORKER_EXECUTABLES

public static final String[][] PEGASUS_WORKER_EXECUTABLES

Array storing the names of the executables in the $PEGASUS_HOME/bin directory Associates the transformation name with the executable basenames

mRegexExpression

private static final String mRegexExpression

Store the regular expressions necessary to parse the basename from the worker package url to retrieve the version of pegasus.

See Also:

Constant Field Values

PATH_VALUE

public static final String PATH_VALUE

The path to be set for create dir jobs.

See Also:

Constant Field Values

DEFAULT_REFINER

public static final String DEFAULT_REFINER

The default transfer refiner name.

See Also:

Constant Field Values

TRANSFORMATION_NAMESPACE

public static final String TRANSFORMATION_NAMESPACE

The transformation namespace for the worker package

See Also:

Constant Field Values

TRANSFORMATION_NAME

public static final String TRANSFORMATION_NAME

The logical name of the worker package

See Also:

Constant Field Values

TRANSFORMATION_VERSION

public static final String TRANSFORMATION_VERSION

The version number for the worker package.

UNTAR_TRANSFORMATION_NAMESPACE

public static final String UNTAR_TRANSFORMATION_NAMESPACE

The transformation namespace for the worker package

UNTAR_TRANSFORMATION_NAME

public static final String UNTAR_TRANSFORMATION_NAME

The logical name of the worker package

See Also:

Constant Field Values

UNTAR_TRANSFORMATION_VERSION

public static final String UNTAR_TRANSFORMATION_VERSION

The version number for the worker package.

COMPLETE_UNTAR_TRANSFORMATION_NAME

public static final String COMPLETE_UNTAR_TRANSFORMATION_NAME

The complete TC name for untar.

COMPLETE_TRANSFORMATION_NAME

public static final String COMPLETE_TRANSFORMATION_NAME

The complete TC name for pegasus worker package.

DERIVATION_NAMESPACE

public static final String DERIVATION_NAMESPACE

The derivation namespace for the worker package.

See Also:

Constant Field Values

DERIVATION_NAME

public static final String DERIVATION_NAME

The logical name of the transformation for the worker package

See Also:

Constant Field Values

DERIVATION_VERSION

public static final String DERIVATION_VERSION

The version number for the derivations for worker package.

See Also:

Constant Field Values

UNTAR_DERIVATION_NAMESPACE

public static final String UNTAR_DERIVATION_NAMESPACE

The derivation namespace for the untar job.

UNTAR_DERIVATION_NAME

public static final String UNTAR_DERIVATION_NAME

The logical name of the transformation for the untar job.

See Also:

Constant Field Values

UNTAR_DERIVATION_VERSION

public static final String UNTAR_DERIVATION_VERSION

The version number for the derivations for untar job.

See Also:

Constant Field Values

PACKAGE_NAME

public static final String PACKAGE_NAME

The name of the package in which all the implementing classes are.

See Also:

Constant Field Values

BASE_BUILD_DIRECTORY_URL

public static final String BASE_BUILD_DIRECTORY_URL

The base directory URL for the builds.

See Also:

Constant Field Values

PEGASUS_VERSION

public static final String PEGASUS_VERSION

The version of pegasus matching the planner.

mPattern

private static Pattern mPattern

Stores compiled patterns at first use, quasi-Singleton.

mOSToNMIOSReleaseAndVersion

private static Map<SysInfo.OS,String> mOSToNMIOSReleaseAndVersion

The map storing OS to corresponding NMI OS platforms.

mCurrentDag

protected ADag mCurrentDag

It is a reference to the Concrete Dag so far.

mJobPrefix

protected String mJobPrefix

The job prefix that needs to be applied to the job file basenames.

mSetupTransferImplementation

protected Implementation mSetupTransferImplementation

The transfer implementation to be used for staging in the data as part of setup job.

mFTMap

protected Map<String,FileTransfer> mFTMap

The FileTransfer map indexed by site id.

mLocalTransfers

protected Map<String,Boolean> mLocalTransfers

Map that indicates whether we need local setup transfer jobs for a site or not.

mSiteToPegasusHomeMap

protected Map<String,String> mSiteToPegasusHomeMap

Maps a site to the the directory where the pegasus worker package has been untarred during workflow execution.

mUserSpecifiedSourceLocation

protected String mUserSpecifiedSourceLocation

The user specified location from where to stage the worker packages.

mUseUserSpecifiedSourceLocation

protected boolean mUseUserSpecifiedSourceLocation

Boolean indicating whether to use the user specified location or not

mTransferWorkerPackage

protected boolean mTransferWorkerPackage

Boolean indicating whether user wants the worker package to be transferred or not.

mPlannerMajorMinorVersion

protected String mPlannerMajorMinorVersion

The major minor version that is used to construct the URL for the pegasus website.

mWorkerNodeExecution

protected boolean mWorkerNodeExecution

Boolean indicating worker node execution.

Constructor Detail

DeployWorkerPackage

public DeployWorkerPackage(PegasusBag bag)

A pratically nothing constructor !

Parameters:

bag - bag of initialization objects

Method Detail

osToOSReleaseAndVersion

private static Map<SysInfo.OS,String> osToOSReleaseAndVersion()

Maps each to OS to a specific OS release for purposes of picking up the correct worker package for a site. The mapping is to be kept consistent with the NMI builds for the releases.

Returns:

map

loadDeployWorkerPackage

public static DeployWorkerPackage loadDeployWorkerPackage(PegasusBag bag)
                                                   throws FactoryException

Loads the implementing class corresponding to the mode specified by the user at runtime.

Parameters:

bag - bag of initialization objects

Returns:

instance of a DeployWorkerPackage implementation

Throws:

FactoryException - that nests any error that might occur during the instantiation of the implementation.

initialize

public void initialize(ADag scheduledDAG)

Initialize with the scheduled graph. Results in the appropriate population of the transformation catalog with pegasus-worker executables.

Parameters:

scheduledDAG - the scheduled workflow.

runTransferOnLocalSite

public boolean runTransferOnLocalSite(Refiner refiner,
                                      String site,
                                      String destinationURL,
                                      int type)

Returns whether to run a transfer job on local site or not.

Parameters:

site - the site handle associated with the destination URL.

destURL - the destination URL

type - the type of transfer job for which the URL is being constructed.

Returns:

true indicating if the associated transfer job should run on local site or not.

getRootDirectoryNameForPegasus

protected String getRootDirectoryNameForPegasus(String url)

Does regex magic to figure out the version of pegasus from the url, and use it to construct the name of pegasus directory, when worker package is untarred.

Parameters:

url - the url.

Returns:

basename for pegasus directory

addSetupNodes

public ADag addSetupNodes(ADag dag)

Adds a setup node per execution site in the workflow that will stage the worker node executables to the workdirectory on the sites the workflow has been scheduled to.

Parameters:

dag - the scheduled workflow.

Returns:

the workflow with setup jobs added

addSetupNodesWithUntarNodes

private Graph addSetupNodesWithUntarNodes(ADag dag,
                                          Set<String> deploymentSites)

Adds untar nodes to the workflow, in addition to the stage worker nodes

Parameters:

dag - the dag

deploymentSites - the sites for which the worker package has to be deployed

Returns:

the workflow in the graph representation with the nodes added.

addSetupNodesWithoutUntarNodes

private Graph addSetupNodesWithoutUntarNodes(ADag dag,
                                             Set<String> deploymentSites)

Adds only the stage worker nodes to the workflow. This is used when Pegasus Lite is used to launch the jobs on the execution sites.

Parameters:

dag - the dag

deploymentSites - the sites for which the worker package has to be deployed

Returns:

the workflow in the graph representation with the nodes added.

addCleanupNodesForWorkerPackage

public ADag addCleanupNodesForWorkerPackage(ADag dag)

Adds cleanup nodes in the workflow for sites specified.

Parameters:

dag - the workflow

Returns:

workflow with cleanup jobs added

getDeploymentSites

protected Set getDeploymentSites(ADag dag)

Retrieves the sites for which the deployment jobs need to be created.

Parameters:

dag - the dag on which the jobs need to execute.

Returns:

a Set containing a list of siteID's of the sites where the dag has to be run.

getDeployJobName

protected String getDeployJobName(ADag dag,
                                  String site,
                                  boolean localTransfer)

It returns the name of the deployment job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.

Parameters:

dag - the workflow so far.

site - the execution pool for which the create directory job is responsible.

localTransfer - whether the transfer needs to run locally or not.

Returns:

String corresponding to the name of the job.

getUntarJobName

protected String getUntarJobName(ADag dag,
                                 String site)

It returns the name of the untar job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.

Parameters:

dag - the workflow so far.

site - the execution pool for which the create directory job is responsible.

Returns:

String corresponding to the name of the job.

getCleanupJobname

protected String getCleanupJobname(ADag dag,
                                   String site)

It returns the name of the untar job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.

Parameters:

dag - the workflow so far.

site - the execution pool for which the create directory job is responsible.

Returns:

String corresponding to the name of the job.

makeUntarJob

protected Job makeUntarJob(String site,
                           String jobName,
                           String wpBasename)

It creates a untar job , that untars the worker package that is staged by the setup transfer job.

Parameters:

site - the execution pool for which the create dir job is to be created.

jobName - the name that is to be assigned to the job.

wpBasename - the basename of the worker package that is staged to remote site.

Returns:

create dir job.

addDefaultTCEntry

private TransformationCatalogEntry addDefaultTCEntry(String site,
                                                     String pegasusHome,
                                                     SysInfo sysinfo,
                                                     boolean useFullPath,
                                                     String name,
                                                     String executable)

Returns a default TC entry to be used in case entry is not found in the transformation catalog. It also attempts to add the transformation catalog entry to the underlying TC store.

Parameters:

site - the site for which the default entry is required.

pegasusHome - the path to deployed worker package

sysinfo - the system information of that site.

useFullPath - boolean indicating whether to use just the basename or the full path

name - the logical name of the transformation

executable - the basename of the executable

Returns:

the default entry.

addDefaultTCEntryForPegasusWebsite

protected TransformationCatalogEntry addDefaultTCEntryForPegasusWebsite(String site,
                                                                        String name)

Returns a default TC entry for the pegasus site. The entry points to the http webserver on the pegasus website. It also attempts to add the transformation catalog entry to the TC store.

Parameters:

site - the execution site for which we need a matching static binary.

name - logical name of the transformation

Returns:

the default entry.

constructDefaultURLToPegasusWorkerPackage

protected String constructDefaultURLToPegasusWorkerPackage(String name,
                                                           SysInfo sysinfo)

Constructs the default URL's for the pegasus worker package. If the user has not specified the URL to the source directory in Pegaus Properties then the URL constructed points to the pegasus website. The version of Pegasus retrieved is the one against which the planner is executing.

Parameters:

name - the logical name of the executable, usually worker|binary.

sysinfo - the sysinfo for which the path is required.

Returns:

url

defaultUntarTCEntry

private TransformationCatalogEntry defaultUntarTCEntry(SiteCatalogEntry site)

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Parameters:

site - the site for which the default entry is required.

Returns:

the default entry.

getBasename

protected String getBasename(String url)

Returns the basename of the URL using substring.

Parameters:

url -

Returns:

basename

getStagingSite

private String getStagingSite(String site)

Returns the staging site for a particular execution site. If worker node execution is enabled, then the staging site is the submit directory for the workflow on the local sit.e

Parameters:

site - the execution site.

Returns:

the staging site