edu.isi.pegasus.planner.refiner
Class RemoveDirectory

java.lang.Object
  edu.isi.pegasus.planner.refiner.Engine
      edu.isi.pegasus.planner.refiner.RemoveDirectory

public class RemoveDirectory
extends Engine

Ends up creating a cleanup dag that deletes the remote directories that were created by the create dir jobs. The cleanup dag is generated in a sub directory from the main directory containing the submit files of the dag. The dag consists of independant jobs, with each job responsible for deleting directory for a execution pool. The current way of generating the dag is tied to the fact, that the directories in which a job are executed is tied to the pool not the job itself.

Version:

$Revision: 4716 $

Author:

Karan Vahi

See Also:

CreateDirectory

Field Summary

static String	CLEANUP_DAG_PREFIX The prefix that is attached to the name of the dag for which the cleanup Dag is being generated, to generate the name of the cleanup Dag.
static String	DERIVATION_NAME The logical name of the transformation that removes directories on the remote execution pools.
static String	DERIVATION_NAMESPACE The derivation namespace for the create dir jobs.
static String	DERIVATION_VERSION The version number for the derivations for create dir jobs.
static String	FILE_URL_SCHEME The scheme name for file url.
private ADag	mConcDag The concrete dag so far, for which the clean up dag needs to be generated.
private String	mSubmitDirectory The submit directory for the workflow.
private boolean	mTransferFromSubmitHost Boolean indicating whether we need to transfer dirmanager from the submit host.
static String	REMOVE_DIR_EXECUTABLE_BASENAME The basename of the pegasus dirmanager executable.
static String	REMOVE_DIR_SUFFIX Constant suffix for the names of the remote directory nodes.
static String	TRANSFORMATION_NAME The logical name of the transformation that removes directories on the remote execution pools.
static String	TRANSFORMATION_NAMESPACE The transformation namespace for the create dir jobs.
static String	TRANSFORMATION_VERSION The version number for the derivations for create dir jobs.

Fields inherited from class edu.isi.pegasus.planner.refiner.Engine

mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE

Constructor Summary

RemoveDirectory(ADag concDag, PegasusBag bag, String submitDirectory)
The overloaded constructor that sets the dag for which we have to generated the cleanup dag for.

Method Summary

private TransformationCatalogEntry	defaultTCEntry(SiteCatalogEntry site) Returns a default TC entry to be used in case entry is not found in the transformation catalog.
ADag	generateCleanUPDAG() Generates a cleanup DAG for the dag associated with the class.
ADag	generateCleanUPDAG(ADag dag) Generates a cleanup DAG for the dag object passed.
static String	getCompleteTranformationName() A convenience method to return the complete transformation name being used to construct jobs in this class.
protected Set	getCreateDirSites(ADag dag) Retrieves the sites for which the create dir jobs need to be created.
private String	getRemoveDirJobName(ADag dag, String pool) It returns the name of the remove directory job, that is to be assigned.
Job	makeRemoveDirJob(String site, String jobName) It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote.
Job	makeRemoveDirJob(String site, String jobName, List<String> files) It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote.

Methods inherited from class edu.isi.pegasus.planner.refiner.Engine

addVector, appendArrayList, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FILE_URL_SCHEME

public static final String FILE_URL_SCHEME

The scheme name for file url.

See Also:

Constant Field Values

CLEANUP_DAG_PREFIX

public static final String CLEANUP_DAG_PREFIX

The prefix that is attached to the name of the dag for which the cleanup Dag is being generated, to generate the name of the cleanup Dag.

See Also:

Constant Field Values

REMOVE_DIR_SUFFIX

public static final String REMOVE_DIR_SUFFIX

Constant suffix for the names of the remote directory nodes.

See Also:

Constant Field Values

TRANSFORMATION_NAME

public static final String TRANSFORMATION_NAME

The logical name of the transformation that removes directories on the remote execution pools.

See Also:

Constant Field Values

REMOVE_DIR_EXECUTABLE_BASENAME

public static final String REMOVE_DIR_EXECUTABLE_BASENAME

The basename of the pegasus dirmanager executable.

See Also:

Constant Field Values

TRANSFORMATION_NAMESPACE

public static final String TRANSFORMATION_NAMESPACE

The transformation namespace for the create dir jobs.

See Also:

Constant Field Values

TRANSFORMATION_VERSION

public static final String TRANSFORMATION_VERSION

The version number for the derivations for create dir jobs.

DERIVATION_NAMESPACE

public static final String DERIVATION_NAMESPACE

The derivation namespace for the create dir jobs.

See Also:

Constant Field Values

DERIVATION_NAME

public static final String DERIVATION_NAME

The logical name of the transformation that removes directories on the remote execution pools.

See Also:

Constant Field Values

DERIVATION_VERSION

public static final String DERIVATION_VERSION

The version number for the derivations for create dir jobs.

See Also:

Constant Field Values

mConcDag

private ADag mConcDag

The concrete dag so far, for which the clean up dag needs to be generated.

mTransferFromSubmitHost

private boolean mTransferFromSubmitHost

Boolean indicating whether we need to transfer dirmanager from the submit host.

mSubmitDirectory

private String mSubmitDirectory

The submit directory for the workflow.

Constructor Detail

RemoveDirectory

public RemoveDirectory(ADag concDag,
                       PegasusBag bag,
                       String submitDirectory)

The overloaded constructor that sets the dag for which we have to generated the cleanup dag for.

Parameters:

concDag - the concrete dag for which cleanup is reqd.

bag - the bag of initialization objects

submitDirectory - the submit directory for the cleanup workflow

Method Detail

getCompleteTranformationName

public static String getCompleteTranformationName()

A convenience method to return the complete transformation name being used to construct jobs in this class.

Returns:

the complete transformation name

generateCleanUPDAG

public ADag generateCleanUPDAG()

Generates a cleanup DAG for the dag associated with the class. Creates a cleanup node per remote pool. It looks at the ADAG, to determine the sites at which the jobs in the dag have been scheduled.

Returns:

the cleanup DAG.

generateCleanUPDAG

public ADag generateCleanUPDAG(ADag dag)

Generates a cleanup DAG for the dag object passed. Creates a cleanup node per remote pool. It looks at the ADAG, to determine the sites at which the jobs in the dag have been scheduled.

Parameters:

dag - the dag for which cleanup dag needs to be generated.

Returns:

the cleanup DAG.

See Also:

org.griphyn.cPlanner.classes.ADag#getExecutionSites()

getCreateDirSites

protected Set getCreateDirSites(ADag dag)

Retrieves the sites for which the create dir jobs need to be created. It returns all the sites where the compute jobs have been scheduled.

Returns:

a Set containing a list of siteID's of the sites where the dag has to be run.

getRemoveDirJobName

private String getRemoveDirJobName(ADag dag,
                                   String pool)

It returns the name of the remove directory job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.

Parameters:

dag - the dag for which the cleanup DAG is being generated.

pool - the execution pool for which the remove directory job is responsible.

Returns:

String corresponding to the name of the job.

makeRemoveDirJob

public Job makeRemoveDirJob(String site,
                            String jobName)

It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote. It access mkdir underneath. It gets the name of the random directory from the Pool handle.

Parameters:

site - the execution pool for which the create dir job is to be created.

jobName - the name that is to be assigned to the job.

Returns:

the remove dir job.

makeRemoveDirJob

public Job makeRemoveDirJob(String site,
                            String jobName,
                            List<String> files)

It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote. It access mkdir underneath. It gets the name of the random directory from the Pool handle.

Parameters:

site - the site from where the directory need to be removed.

jobName - the name that is to be assigned to the job.

files - the list of files to be cleaned up.

Returns:

the remove dir job.

defaultTCEntry

private TransformationCatalogEntry defaultTCEntry(SiteCatalogEntry site)

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Parameters:

site - the SiteCatalogEntry for the site for which the default entry is required.

Returns:

the default entry.