edu.isi.pegasus.planner.transfer.implementation
Class AbstractMultipleFTPerXFERJob

java.lang.Object
  edu.isi.pegasus.planner.transfer.implementation.Abstract
      edu.isi.pegasus.planner.transfer.implementation.AbstractMultipleFTPerXFERJob

All Implemented Interfaces:

Implementation, MultipleFTPerXFERJob

Direct Known Subclasses:

GUC, T2, Transfer

public abstract class AbstractMultipleFTPerXFERJob
extends Abstract
implements MultipleFTPerXFERJob

An abstract implementation for implementations that can handle multiple file transfers in a single file transfer job.

Version:

$Revision: 4778 $

Author:

Karan Vahi

Field Summary

Fields inherited from class edu.isi.pegasus.planner.transfer.implementation.Abstract

CHANGE_XBIT_TRANSFORMATION, mAddNodesForSettingXBit, mChmodDisabledForAllSites, mDisabledChmodSites, mLocalIrodsEnv, mLocalIrodsEnvBasename, mLocalS3cfg, mLocalS3cfgBasename, mLocalUserProxy, mLocalUserProxyBasename, mLogger, mPOptions, mProps, mRefiner, mSiteStore, mTCHandle, NOOP_PREFIX, SET_XBIT_PREFIX, XBIT_DERIVATION_NS, XBIT_DERIVATION_VERSION, XBIT_EXECUTABLE_BASENAME, XBIT_TRANSFORMATION_NS, XBIT_TRANSFORMATION_VERSION

Fields inherited from interface edu.isi.pegasus.planner.transfer.Implementation

TRANSFER_UNIVERSE, VERSION

Constructor Summary

AbstractMultipleFTPerXFERJob(PegasusBag bag)
The overloaded constructor, that is called by the Factory to load the class.

Method Summary

TransferJob	createTransferJob(Job job, String site, Collection files, Collection execFiles, String txJobName, int jobClass) Constructs a general transfer job that handles multiple transfers per transfer job.
protected TransformationCatalogEntry	defaultTCEntry(String namespace, String name, String version, String executableBasename, String site) Returns a default TC entry to be used in case entry is not found in the transformation catalog.
protected abstract String	generateArgumentString(TransferJob job) It constructs the arguments to the transfer executable that need to be passed to the executable referred to in this transfer mode.
protected abstract String	getCompleteTCName() Returns the complete name for the transformation that the implementation is using..
protected abstract String	getDerivationName() Returns the logical name of the derivation that this implementation refers to.
protected abstract String	getDerivationNamespace() Returns the namespace of the derivation that this implementation refers to.
protected abstract String	getDerivationVersion() Returns the version of the derivation that this implementation refers to.
protected abstract List<Profile>	getEnvironmentVariables(String site) Returns the environment profiles that are required for the default entry to sensibly work.
void	postProcess(TransferJob job) An optional method that allows the derived classes to do their own post processing on the the transfer job before it is returned to the calling module.
protected String	prepareSTDINAndAssociateCredentials(TransferJob job, Collection files, String stagingSite, int jobClass) Prepares the stdin for the transfer job.
protected abstract void	writeStdInAndAssociateCredentials(TransferJob job, FileWriter stdIn, Collection files, String stagingSite, int jobClass) Writes to a FileWriter stream the stdin which goes into the magic script via standard input

Methods inherited from class edu.isi.pegasus.planner.transfer.implementation.Abstract

addSetXBitJobs, addSetXBitJobs, addSetXBitJobs, applyPriority, checkAndTransferIrodsEnvFile, checkAndTransferProxy, construct, createNoOPJob, createSetXBitJob, createSetXBitJob, createSetXBitJob, determineDisabledChmodSites, disableChmodJobCreation, getNOOPJobName, getPriority, getSetXBitJobName, setRefiner

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.isi.pegasus.planner.transfer.Implementation

doesPreserveXBit, getDescription, getTransformationCatalogEntry, useThirdPartyTransferAlways

Constructor Detail

AbstractMultipleFTPerXFERJob

public AbstractMultipleFTPerXFERJob(PegasusBag bag)

The overloaded constructor, that is called by the Factory to load the class.

Parameters:

bag - the bag of Pegasus initialization objects

Method Detail

createTransferJob

public TransferJob createTransferJob(Job job,
                                     String site,
                                     Collection files,
                                     Collection execFiles,
                                     String txJobName,
                                     int jobClass)

Constructs a general transfer job that handles multiple transfers per transfer job. There are appropriate callouts to generate the implementation specific details.

Specified by:

createTransferJob in interface Implementation

Parameters:

job - the Job object for the job, in relation to which the transfer node is being added. Either the transfer node can be transferring this jobs input files to the execution pool, or transferring this job's output files to the output pool.

site - the site where the transfer job should run.

files - collection of FileTransfer objects representing the data files and staged executables to be transferred.

execFiles - subset collection of the files parameter, that identifies the executable files that are being transferred.

txJobName - the name of transfer node.

jobClass - the job Class for the newly added job. Can be one of the following: stage-in stage-out inter-pool transfer stage-in worker transfer

Returns:

the created TransferJob.

defaultTCEntry

protected TransformationCatalogEntry defaultTCEntry(String namespace,
                                                    String name,
                                                    String version,
                                                    String executableBasename,
                                                    String site)

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Parameters:

namespace - the namespace of the transfer transformation

name - the logical name of the transfer transformation

version - the version of the transfer transformation

executableBasename - the basename of the executable

site - the site for which the default entry is required.

Returns:

the default entry.

getEnvironmentVariables

protected abstract List<Profile> getEnvironmentVariables(String site)

Returns the environment profiles that are required for the default entry to sensibly work.

Parameters:

site - the site where the job is going to run.

Returns:

List of environment variables, else null in case where the required environment variables could not be found.

postProcess

public void postProcess(TransferJob job)

An optional method that allows the derived classes to do their own post processing on the the transfer job before it is returned to the calling module.

Parameters:

job - the TransferJob that has been created.

prepareSTDINAndAssociateCredentials

protected String prepareSTDINAndAssociateCredentials(TransferJob job,
                                                     Collection files,
                                                     String stagingSite,
                                                     int jobClass)
                                              throws Exception

Prepares the stdin for the transfer job. Usually involves writing out a text file that Condor transfers to the remote end. Additionally, it associates credentials with the job that are requried to for the transfers.

Parameters:

name - the name of the transfer job.

files - Collection of FileTransfer objects containing the information about sourceam fin and destURL's.

stagingSite - the site where the data will be populated by first level staging jobs.

jobClass - the job Class for the newly added job. Can be one of the following: stage-in stage-out inter-pool transfer

Returns:

the path to the prepared stdin file.

Throws:

Exception - in case of error.

getDerivationNamespace

protected abstract String getDerivationNamespace()

Returns the namespace of the derivation that this implementation refers to.

Returns:

the namespace of the derivation.

getDerivationName

protected abstract String getDerivationName()

Returns the logical name of the derivation that this implementation refers to.

Returns:

the name of the derivation.

getDerivationVersion

protected abstract String getDerivationVersion()

Returns the version of the derivation that this implementation refers to.

Returns:

the version of the derivation.

generateArgumentString

protected abstract String generateArgumentString(TransferJob job)

It constructs the arguments to the transfer executable that need to be passed to the executable referred to in this transfer mode.

Parameters:

job - the object containing the transfer node.

Returns:

the argument string

writeStdInAndAssociateCredentials

protected abstract void writeStdInAndAssociateCredentials(TransferJob job,
                                                          FileWriter stdIn,
                                                          Collection files,
                                                          String stagingSite,
                                                          int jobClass)
                                                   throws Exception

Writes to a FileWriter stream the stdin which goes into the magic script via standard input

Parameters:

job - the transfer job .

stdIn - the writer to the stdin file.

files - Collection of FileTransfer objects containing the information about sourceam fin and destURL's.

stagingSite - the site where the data will be populated by first level staging jobs.

jobClass - the job Class for the newly added job. Can be one of the following: stage-in stage-out inter-pool transfer

Throws:

Exception

getCompleteTCName

protected abstract String getCompleteTCName()

Returns the complete name for the transformation that the implementation is using..

Returns:

the complete name.