edu.isi.pegasus.planner.transfer.refiner
Class Bundle

java.lang.Object
  edu.isi.pegasus.planner.transfer.AbstractRefiner
      edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner
          edu.isi.pegasus.planner.transfer.refiner.Default
              edu.isi.pegasus.planner.transfer.refiner.Bundle

All Implemented Interfaces:

Refiner

Direct Known Subclasses:

Cluster

public class Bundle
extends Default

An extension of the default refiner, that allows the user to specify the number of transfer nodes per execution site for stagein and stageout.

Version:

$Revision: 4642 $

Author:

Karan Vahi

Nested Class Summary

protected class	Bundle.BundleValue
protected class	Bundle.PoolTransfer A container to store the transfers that need to be done on a single pool.
protected class	Bundle.TransferContainer A container class for storing the name of the transfer job, the list of file transfers that the job is responsible for.

Field Summary

static String	DEFAULT_LOCAL_STAGE_IN_BUNDLE_FACTOR The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.
static String	DEFAULT_LOCAL_STAGE_OUT_BUNDLE_FACTOR The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.
static String	DEFAULT_REMOTE_STAGE_IN_BUNDLE_FACTOR The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.
static String	DEFAULT_REMOTE_STAGE_OUT_BUNDLE_FACTOR The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.
static String	DESCRIPTION A short description of the transfer refinement.
protected boolean	mAddNodesForSettingXBit A boolean indicating whether chmod jobs should be created that set the xbit in case of executable staging.
private int	mCurrentSOLevel The current level of the jobs being traversed.
protected String	mJobPrefix The job prefix that needs to be applied to the job file basenames.
protected Pegasus	mPegasusProfilesInProperties Pegasus Profiles specified in the properties.
private ReplicaCatalogBridge	mRCB The handle to the replica catalog bridge.
private Map	mRelationsMap The map indexed by compute jobnames that contains the list of stagin job names that are being added during the traversal of the workflow.
protected Map	mSetupMap The map indexed by staged executable logical name.
protected SiteStore	mSiteStore Handle to the SiteStore
protected Bundle.BundleValue	mStageinLocalBundleValue The BundleValue that evaluates for local stage in jobs.
private Map	mStageInLocalMap The map containing the list of stage in transfer jobs that are being created for the workflow indexed by the execution poolname.
protected Bundle.BundleValue	mStageInRemoteBundleValue The BundleValue that evaluates for remote stage-in jobs.
private Map	mStageInRemoteMap The map containing the list of stage in transfer jobs that are being created for the workflow indexed by the execution poolname.
protected Bundle.BundleValue	mStageOutLocalBundleValue The BundleValue that evaluates for local stage out jobs.
private Map<String,Bundle.PoolTransfer>	mStageOutLocalMapPerLevel A map indexed by site name, that contains the pointer to the local stage out PoolTransfer objects for that site.
protected Bundle.BundleValue	mStageOutRemoteBundleValue The BundleValue that evaluates for remote stage out jobs.
private Map<String,Bundle.PoolTransfer>	mStageOutRemoteMapPerLevel A map indexed by site name, that contains the pointer to the remote stage out PoolTransfer objects for that site.

Fields inherited from class edu.isi.pegasus.planner.transfer.refiner.Default

mCreateRegistrationJobs, mFileTable, mLogMsg, mPPS

Fields inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner

mDAG, mLogger, mPOptions, mProps, mRemoteTransfers, mTPT, mTXInterImplementation, mTXStageInImplementation, mTXStageOutImplementation, mTXSymbolicLinkImplementation, mXMLStore

Fields inherited from interface edu.isi.pegasus.planner.transfer.Refiner

INTER_POOL_PREFIX, LOCAL_PREFIX, REGISTER_PREFIX, REMOTE_PREFIX, STAGE_IN_PREFIX, STAGE_OUT_PREFIX

Fields inherited from interface edu.isi.pegasus.planner.refiner.Refiner

VERSION

Constructor Summary

Bundle(ADag dag, PegasusBag bag)
The overloaded constructor.

Method Summary

void	addStageInXFERNodes(Job job, boolean localTransfer, Collection files, int type, Map<String,Bundle.PoolTransfer> stageInMap, Bundle.BundleValue bundleValue, Implementation implementation) Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.
void	addStageInXFERNodes(Job job, Collection<FileTransfer> files, Collection<FileTransfer> symlinkFiles) Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.
void	addStageOutXFERNodes(Job job, Collection files, ReplicaCatalogBridge rcb, boolean localTransfer, boolean deletedLeaf) Adds the stageout transfer nodes, that stage data to an output site specified by the user.
void	done() Signals that the traversal of the workflow is done.
void	doneStageIn(Map<String,Bundle.PoolTransfer> stageInMap, Implementation implementation, int stageInJobType, boolean localTransfer) Signals that the traversal of the workflow is done.
protected String	getComputeJobBundleValue(Job job) Returns the bundle value associated with a compute job as a String.
protected String	getDefaultBundleValueFromProperties(String key, String defaultKey, String defaultValue) Returns the default value for the clustering/bundling of jobs to be used.
String	getDescription() Returns a textual description of the transfer mode.
Bundle.PoolTransfer	getStageOutPoolTransfer(String site, boolean localTransfer, int num) Returns the appropriate pool transfer for a particular site.
protected void	initializeBundleValues() Initializes the bundle value variables, that are responsible determining the bundle values.
protected Map	resetStageOutMap(Map<String,Bundle.PoolTransfer> map, boolean localTransfer) Resets a single map
protected void	resetStageOutMaps() Resets the local and remote stage out maps.

Methods inherited from class edu.isi.pegasus.planner.transfer.refiner.Default

addInterSiteTXNodes, addJob, addRelation, addRelation, addStageInXFERNodes, addStageOutXFERNodes, appendAttribute, constructFileKey, createRegistrationJob, logRefinerAction

Methods inherited from class edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner

loadImplementations

Methods inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner

addStageInXFERNodes, getWorkflow, getXMLProducer, isSiteThirdParty, logConfigMessages, refinerPreferenceForLocalTransferJobs, refinerPreferenceForTransferJobLocation, runTPTOnRemoteSite, runTransferRemotely

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DESCRIPTION

public static final String DESCRIPTION

A short description of the transfer refinement.

See Also:

Constant Field Values

DEFAULT_LOCAL_STAGE_IN_BUNDLE_FACTOR

public static final String DEFAULT_LOCAL_STAGE_IN_BUNDLE_FACTOR

The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.

See Also:

Constant Field Values

DEFAULT_REMOTE_STAGE_IN_BUNDLE_FACTOR

public static final String DEFAULT_REMOTE_STAGE_IN_BUNDLE_FACTOR

The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.

See Also:

Constant Field Values

DEFAULT_LOCAL_STAGE_OUT_BUNDLE_FACTOR

public static final String DEFAULT_LOCAL_STAGE_OUT_BUNDLE_FACTOR

The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.

See Also:

Constant Field Values

DEFAULT_REMOTE_STAGE_OUT_BUNDLE_FACTOR

public static final String DEFAULT_REMOTE_STAGE_OUT_BUNDLE_FACTOR

The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.

See Also:

Constant Field Values

mStageInLocalMap

private Map mStageInLocalMap

The map containing the list of stage in transfer jobs that are being created for the workflow indexed by the execution poolname.

mStageInRemoteMap

private Map mStageInRemoteMap

The map containing the list of stage in transfer jobs that are being created for the workflow indexed by the execution poolname.

mRelationsMap

private Map mRelationsMap

The map indexed by compute jobnames that contains the list of stagin job names that are being added during the traversal of the workflow. This is used to construct the relations that need to be added to workflow, once the traversal is done.

mStageinLocalBundleValue

protected Bundle.BundleValue mStageinLocalBundleValue

The BundleValue that evaluates for local stage in jobs.

mStageInRemoteBundleValue

protected Bundle.BundleValue mStageInRemoteBundleValue

The BundleValue that evaluates for remote stage-in jobs.

mStageOutLocalBundleValue

protected Bundle.BundleValue mStageOutLocalBundleValue

The BundleValue that evaluates for local stage out jobs.

mStageOutRemoteBundleValue

protected Bundle.BundleValue mStageOutRemoteBundleValue

The BundleValue that evaluates for remote stage out jobs.

mSetupMap

protected Map mSetupMap

The map indexed by staged executable logical name. Each entry is the name of the corresponding setup job, that changes the XBit on the staged file.

mStageOutLocalMapPerLevel

private Map<String,Bundle.PoolTransfer> mStageOutLocalMapPerLevel

A map indexed by site name, that contains the pointer to the local stage out PoolTransfer objects for that site. This is per level of the workflow.

mStageOutRemoteMapPerLevel

private Map<String,Bundle.PoolTransfer> mStageOutRemoteMapPerLevel

A map indexed by site name, that contains the pointer to the remote stage out PoolTransfer objects for that site. This is per level of the workflow.

mCurrentSOLevel

private int mCurrentSOLevel

The current level of the jobs being traversed.

mRCB

private ReplicaCatalogBridge mRCB

The handle to the replica catalog bridge.

mJobPrefix

protected String mJobPrefix

The job prefix that needs to be applied to the job file basenames.

mPegasusProfilesInProperties

protected Pegasus mPegasusProfilesInProperties

Pegasus Profiles specified in the properties.

mSiteStore

protected SiteStore mSiteStore

Handle to the SiteStore

mAddNodesForSettingXBit

protected boolean mAddNodesForSettingXBit

A boolean indicating whether chmod jobs should be created that set the xbit in case of executable staging.

Constructor Detail

Bundle

public Bundle(ADag dag,
              PegasusBag bag)

The overloaded constructor.

Parameters:

dag - the workflow to which transfer nodes need to be added.

bag - the bag of initialization objects

Method Detail

initializeBundleValues

protected void initializeBundleValues()

Initializes the bundle value variables, that are responsible determining the bundle values.

getDefaultBundleValueFromProperties

protected String getDefaultBundleValueFromProperties(String key,
                                                     String defaultKey,
                                                     String defaultValue)

Returns the default value for the clustering/bundling of jobs to be used. The factor is computed by looking up the pegasus profiles in the properties.

    return value of pegasus profile key if it exists,
    else return value of pegasus profile defaultKey if it exists, 
    else the defaultValue
 

Parameters:

key - the pegasus profile key

defaultKey - the default pegasus profile key

defaultValue - the default value.

Returns:

the value as string.

addStageInXFERNodes

public void addStageInXFERNodes(Job job,
                                Collection<FileTransfer> files,
                                Collection<FileTransfer> symlinkFiles)

Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.

Overrides:

addStageInXFERNodes in class Default

Parameters:

job - Job object corresponding to the node to which the files are to be transferred to.

files - Collection of FileTransfer objects containing the information about source and destURL's.

symlinkFiles - Collection of FileTransfer objects containing source and destination file url's for symbolic linking on compute site.

addStageInXFERNodes

public void addStageInXFERNodes(Job job,
                                boolean localTransfer,
                                Collection files,
                                int type,
                                Map<String,Bundle.PoolTransfer> stageInMap,
                                Bundle.BundleValue bundleValue,
                                Implementation implementation)

Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.

Parameters:

job - Job object corresponding to the node to which the files are to be transferred to.

localTransfer - boolean indicating whether transfer has to happen on local site.

files - Collection of FileTransfer objects containing the information about source and destURL's.

type - the type of transfer job being created

stageInMap - Map indexed by site name that gives all the transfers for that site.

bundleValue - used to determine the bundling factor to employ for a job.

implementation - the transfer implementation to use.

addStageOutXFERNodes

public void addStageOutXFERNodes(Job job,
                                 Collection files,
                                 ReplicaCatalogBridge rcb,
                                 boolean localTransfer,
                                 boolean deletedLeaf)

Adds the stageout transfer nodes, that stage data to an output site specified by the user.

Overrides:

addStageOutXFERNodes in class Default

Parameters:

job - Job object corresponding to the node to which the files are to be transferred to.

files - Collection of FileTransfer objects containing the information about source and destURL's.

rcb - bridge to the Replica Catalog. Used for creating registration nodes in the workflow.

localTransfer - whether the transfer should be on local site or not.

deletedLeaf - to specify whether the node is being added for a deleted node by the reduction engine or not. default: false

getComputeJobBundleValue

protected String getComputeJobBundleValue(Job job)

Returns the bundle value associated with a compute job as a String.

Parameters:

job -

Returns:

value as String or NULL

done

public void done()

Signals that the traversal of the workflow is done. At this point the transfer nodes are actually constructed traversing through the transfer containers and the stdin of the transfer jobs written.

Overrides:

done in class Default

doneStageIn

public void doneStageIn(Map<String,Bundle.PoolTransfer> stageInMap,
                        Implementation implementation,
                        int stageInJobType,
                        boolean localTransfer)

Signals that the traversal of the workflow is done. At this point the transfer nodes are actually constructed traversing through the transfer containers and the stdin of the transfer jobs written.

Parameters:

stageInMap - maps site names to PoolTransfer

implementation - the transfer implementation to use

stageInJobType - whether a stagein or symlink stagein job

localTransfer - indicates whether transfer job needs to run on local site or not.

getDescription

public String getDescription()

Returns a textual description of the transfer mode.

Overrides:

getDescription in class Default

Returns:

a short textual description

getStageOutPoolTransfer

public Bundle.PoolTransfer getStageOutPoolTransfer(String site,
                                                   boolean localTransfer,
                                                   int num)

Returns the appropriate pool transfer for a particular site.

Parameters:

site - the site for which the PT is reqd.

localTransfer - whethe the associated transfer job runs on local site or remote.

num - the number of Stageout jobs required for that Pool.

Returns:

the PoolTransfer

resetStageOutMaps

protected void resetStageOutMaps()

Resets the local and remote stage out maps.

resetStageOutMap

protected Map resetStageOutMap(Map<String,Bundle.PoolTransfer> map,
                               boolean localTransfer)

Resets a single map

Parameters:

map - the map to be reset

localTransfer - whether the transfer jobs need to run on local site or not

Returns:

the reset map