edu.isi.pegasus.planner.transfer.refiner
Class Chain

java.lang.Object
  edu.isi.pegasus.planner.transfer.AbstractRefiner
      edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner
          edu.isi.pegasus.planner.transfer.refiner.Default
              edu.isi.pegasus.planner.transfer.refiner.Chain

All Implemented Interfaces:

Refiner

public class Chain
extends Default

This transfer refiner builds upon the Default Refiner. The defaul Refiner allows the transfer of multiple files in a single condor job. However, it adds the stage in transfer nodes in parallel leading to multiple invocation of the globus-url-copy at remote execution pools, while running huge workflows. This refiner, tries to circumvent this problem by chaining up the stagein jobs instead of scheduling in parallel. This works best only when the top level of the workflow requires stage in jobs. The correct way is that the traversal needs to be done breath first in the TransferEngine.java.

Version:

$Revision: 4720 $

Author:

Karan Vahi, Gaurang Mehta

Nested Class Summary

private class	Chain.SiteTransfer A container to manage the transfer jobs that are needed to be done on a single site.
private class	Chain.TransferChain A shallow container class, that contains the list of the names of the transfer jobs and can return the last job in the list.

Field Summary

static String	DEFAULT_BUNDLE_FACTOR The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for the workflow.
static String	DESCRIPTION A short description of the transfer refinement.
private Map	mSIBundleMap The map containing the stage in bundle values indexed by the name of the site.
protected SiteStore	mSiteStore The handle to the Site Catalog.
private Map	mStageInMap A map indexed by execution sites.

Fields inherited from class edu.isi.pegasus.planner.transfer.refiner.Default

mCreateRegistrationJobs, mFileTable, mLogMsg, mPPS

Fields inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner

mDAG, mLogger, mPOptions, mProps, mRemoteTransfers, mTPT, mTXInterImplementation, mTXStageInImplementation, mTXStageOutImplementation, mTXSymbolicLinkImplementation, mXMLStore

Fields inherited from interface edu.isi.pegasus.planner.transfer.Refiner

INTER_POOL_PREFIX, LOCAL_PREFIX, REGISTER_PREFIX, REMOTE_PREFIX, STAGE_IN_PREFIX, STAGE_OUT_PREFIX

Fields inherited from interface edu.isi.pegasus.planner.refiner.Refiner

VERSION

Constructor Summary

Chain(ADag dag, PegasusBag bag)
The overloaded constructor.

Method Summary

void	addRelation(String parent, String child, String site, boolean parentNew) Adds a new relation to the workflow.
void	done() Prints out the bundles and chains that have been constructed.
String	getDescription() Returns a textual description of the transfer mode.
int	getSiteBundleValue(String site, String key) Determines the bundle factor for a particular site on the basis of the key associcated with the underlying transfer transformation in the transformation catalog.

Methods inherited from class edu.isi.pegasus.planner.transfer.refiner.Default

addInterSiteTXNodes, addJob, addRelation, addStageInXFERNodes, addStageInXFERNodes, addStageOutXFERNodes, addStageOutXFERNodes, appendAttribute, constructFileKey, createRegistrationJob, logRefinerAction

Methods inherited from class edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner

loadImplementations

Methods inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner

addStageInXFERNodes, getWorkflow, getXMLProducer, isSiteThirdParty, logConfigMessages, refinerPreferenceForLocalTransferJobs, refinerPreferenceForTransferJobLocation, runTPTOnRemoteSite, runTransferRemotely

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BUNDLE_FACTOR

public static final String DEFAULT_BUNDLE_FACTOR

The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for the workflow.

See Also:

Constant Field Values

mSiteStore

protected SiteStore mSiteStore

The handle to the Site Catalog. It is instantiated in this class.

mSIBundleMap

private Map mSIBundleMap

The map containing the stage in bundle values indexed by the name of the site. If the bundle value is not specified, then null is stored.

mStageInMap

private Map mStageInMap

A map indexed by execution sites. Each value is a SiteTransfer object, that contains the Bundles of stagin transfer jobs.

See Also:

Chain.TransferChain

DESCRIPTION

public static final String DESCRIPTION

A short description of the transfer refinement.

See Also:

Constant Field Values

Constructor Detail

Chain

public Chain(ADag dag,
             PegasusBag bag)

The overloaded constructor.

Parameters:

dag - the workflow to which transfer nodes need to be added.

bag - the bag of initialization objects

Method Detail

addRelation

public void addRelation(String parent,
                        String child,
                        String site,
                        boolean parentNew)

Adds a new relation to the workflow. In the case when the parent is a transfer job that is added, the parentNew should be set only the first time a relation is added. For subsequent compute jobs that maybe dependant on this, it needs to be set to false.

Overrides:

addRelation in class Default

Parameters:

parent - the jobname of the parent node of the edge.

child - the jobname of the child node of the edge.

site - the execution site where the transfer node is to be run.

parentNew - the parent node being added, is the new transfer job and is being called for the first time.

getSiteBundleValue

public int getSiteBundleValue(String site,
                              String key)

Determines the bundle factor for a particular site on the basis of the key associcated with the underlying transfer transformation in the transformation catalog. If none specified in transformation catalog then one is picked up from the site catalog. If the key is not found in the site catalog too , then the global default is returned.

Parameters:

site - the site at which the transfer job is being run.

key - the bundle key whose value needs to be searched.

Returns:

the bundle factor.

See Also:

DEFAULT_BUNDLE_FACTOR

done

public void done()

Prints out the bundles and chains that have been constructed.

Overrides:

done in class Default

getDescription

public String getDescription()

Returns a textual description of the transfer mode.

Overrides:

getDescription in class Default

Returns:

a short textual description