edu.isi.pegasus.planner.transfer.implementation
Class Abstract

java.lang.Object
  edu.isi.pegasus.planner.transfer.implementation.Abstract

All Implemented Interfaces:

Implementation

Direct Known Subclasses:

AbstractMultipleFTPerXFERJob, AbstractSingleFTPerXFERJob

public abstract class Abstract
extends Object
implements Implementation

An abstract implementation that implements some of the common functions in the Implementation Interface that are required by all the implementations.

Version:

$Revision: 4778 $

Author:

Karan Vahi

Field Summary

static String	CHANGE_XBIT_TRANSFORMATION The logical name of the transformation that creates directories on the remote execution pools.
protected boolean	mAddNodesForSettingXBit A boolean indicating whether chmod jobs should be created that set the xbit in case of executable staging.
protected boolean	mChmodDisabledForAllSites A boolean indicating whether chmod is disabled for all sites.
protected Set	mDisabledChmodSites The set of sites for which chmod job creation has to be disabled while doing executable staging.
protected String	mLocalIrodsEnv The path to the irodsEnv file on the submit host (local pool).
protected String	mLocalIrodsEnvBasename The basename of the user irodsEnv file
protected String	mLocalS3cfg The path to the s3cfg file on the submit host (local pool).
protected String	mLocalS3cfgBasename The basename of the user s3cfg file
protected String	mLocalUserProxy The path to the user proxy on the submit host (local pool), that is picked up for use in transfer of proxies.
protected String	mLocalUserProxyBasename The basename of the user proxy , that is picked up for use in transfer of proxies.
protected LogManager	mLogger The logging object which is used to log all the messages.
protected PlannerOptions	mPOptions Contains the various options to the Planner as passed by the user at runtime.
protected PegasusProperties	mProps The handle to the properties object holding the properties relevant to Pegasus.
protected Refiner	mRefiner The handle to the refiner that loaded this implementation.
protected SiteStore	mSiteStore The handle to the Pool Info Provider.
protected TransformationCatalog	mTCHandle The handle to the Transformation Catalog.
static String	NOOP_PREFIX The prefix for the NoOP jobs that are created.
static String	SET_XBIT_PREFIX The prefix for the jobs which are added to set X bit for the staged executables.
static String	XBIT_DERIVATION_NS The derivation namespace for the setXBit jobs.
static String	XBIT_DERIVATION_VERSION The version number for the derivations for setXBit jobs.
static String	XBIT_EXECUTABLE_BASENAME The basename of the pegasus dirmanager executable.
static String	XBIT_TRANSFORMATION_NS The transformation namespace for the setXBit jobs.
static String	XBIT_TRANSFORMATION_VERSION The version number for the derivations for setXBit jobs.

Fields inherited from interface edu.isi.pegasus.planner.transfer.Implementation

TRANSFER_UNIVERSE, VERSION

Constructor Summary

Abstract(PegasusBag bag)
The overloaded constructor, that is called by the Factory to load the class.

Method Summary

protected boolean	addSetXBitJobs(Job computeJob, Job txJob, Collection execFiles) Adds the dirmanager to the workflow, that do a chmod on the files being staged.
boolean	addSetXBitJobs(Job computeJob, String txJobName, Collection execFiles, int transferClass) Adds the dirmanager job to the workflow, that do a chmod on the files being staged.
boolean	addSetXBitJobs(Job computeJob, String txJobName, Collection execFiles, int transferClass, int xbitIndex) Adds the dirmanager job to the workflow, that do a chmod on the files being staged.
void	applyPriority(TransferJob job) Applies priorities to the transfer jobs if a priority is specified in the properties file.
boolean	checkAndTransferIrodsEnvFile(TransferJob job) Deprecated.
boolean	checkAndTransferProxy(TransferJob job) Deprecated.
protected void	construct(Job job, String key, String value) Constructs a condor variable in the condor profile namespace associated with the job.
Job	createNoOPJob(String name) It creates a NoOP job that runs on the submit host.
protected Job	createSetXBitJob(Collection<FileTransfer> files, String name, String site) Creates a dirmanager job, that does a chmod on the file being staged.
protected Job	createSetXBitJob(FileTransfer file, String name) Creates a dirmanager job, that does a chmod on the file being staged.
Job	createSetXBitJob(Job computeJob, Collection<FileTransfer> execFiles, int transferClass, int xbitIndex) Adds the dirmanager job to the workflow, that do a chmod on the files being staged.
private TransformationCatalogEntry	defaultXBitTCEntry(String site) Returns a default TC entry to be used in case entry is not found in the transformation catalog.
protected Set	determineDisabledChmodSites(String sites) Builds up a set of disabled chmod sites
protected boolean	disableChmodJobCreation(String site) Returns a boolean indicating whether to disable chmod job creation for a site or not.
String	getNOOPJobName(String name, int counter) Generates the name of the noop job , that is unique for the given workflow.
protected String	getPriority(TransferJob job) Returns the priority for the transfer job as specified in the properties file.
String	getSetXBitJobName(String name, int counter) Generates the name of the setXBitJob , that is unique for the given workflow.
void	setRefiner(Refiner refiner) Sets the callback to the refiner, that has loaded this implementation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.isi.pegasus.planner.transfer.Implementation

createTransferJob, doesPreserveXBit, getDescription, getTransformationCatalogEntry, useThirdPartyTransferAlways

Field Detail

CHANGE_XBIT_TRANSFORMATION

public static final String CHANGE_XBIT_TRANSFORMATION

The logical name of the transformation that creates directories on the remote execution pools.

See Also:

Constant Field Values

XBIT_EXECUTABLE_BASENAME

public static final String XBIT_EXECUTABLE_BASENAME

The basename of the pegasus dirmanager executable.

See Also:

Constant Field Values

XBIT_TRANSFORMATION_NS

public static final String XBIT_TRANSFORMATION_NS

The transformation namespace for the setXBit jobs.

See Also:

Constant Field Values

XBIT_TRANSFORMATION_VERSION

public static final String XBIT_TRANSFORMATION_VERSION

The version number for the derivations for setXBit jobs.

XBIT_DERIVATION_NS

public static final String XBIT_DERIVATION_NS

The derivation namespace for the setXBit jobs.

See Also:

Constant Field Values

XBIT_DERIVATION_VERSION

public static final String XBIT_DERIVATION_VERSION

The version number for the derivations for setXBit jobs.

SET_XBIT_PREFIX

public static final String SET_XBIT_PREFIX

The prefix for the jobs which are added to set X bit for the staged executables.

See Also:

Constant Field Values

NOOP_PREFIX

public static final String NOOP_PREFIX

The prefix for the NoOP jobs that are created.

See Also:

Constant Field Values

mLocalUserProxy

protected String mLocalUserProxy

The path to the user proxy on the submit host (local pool), that is picked up for use in transfer of proxies.

mLocalUserProxyBasename

protected String mLocalUserProxyBasename

The basename of the user proxy , that is picked up for use in transfer of proxies.

mLocalS3cfg

protected String mLocalS3cfg

The path to the s3cfg file on the submit host (local pool).

mLocalS3cfgBasename

protected String mLocalS3cfgBasename

The basename of the user s3cfg file

mLocalIrodsEnv

protected String mLocalIrodsEnv

The path to the irodsEnv file on the submit host (local pool).

mLocalIrodsEnvBasename

protected String mLocalIrodsEnvBasename

The basename of the user irodsEnv file

mProps

protected PegasusProperties mProps

The handle to the properties object holding the properties relevant to Pegasus.

mPOptions

protected PlannerOptions mPOptions

Contains the various options to the Planner as passed by the user at runtime.

mSiteStore

protected SiteStore mSiteStore

The handle to the Pool Info Provider. It is instantiated in this class

mTCHandle

protected TransformationCatalog mTCHandle

The handle to the Transformation Catalog. It must be instantiated in the implementing class

mRefiner

protected Refiner mRefiner

The handle to the refiner that loaded this implementation.

mLogger

protected LogManager mLogger

The logging object which is used to log all the messages.

mDisabledChmodSites

protected Set mDisabledChmodSites

The set of sites for which chmod job creation has to be disabled while doing executable staging.

mChmodDisabledForAllSites

protected boolean mChmodDisabledForAllSites

A boolean indicating whether chmod is disabled for all sites.

mAddNodesForSettingXBit

protected boolean mAddNodesForSettingXBit

A boolean indicating whether chmod jobs should be created that set the xbit in case of executable staging.

Constructor Detail

Abstract

public Abstract(PegasusBag bag)

The overloaded constructor, that is called by the Factory to load the class.

Parameters:

bag - the bag of initialization objects.

Method Detail

applyPriority

public void applyPriority(TransferJob job)

Applies priorities to the transfer jobs if a priority is specified in the properties file.

Specified by:

applyPriority in interface Implementation

Parameters:

job - the transfer job .

checkAndTransferProxy

public boolean checkAndTransferProxy(TransferJob job)

Deprecated.

Determines if there is a need to transfer proxy for the transfer job or not. If there is a need to transfer proxy, then the job is modified to create the correct condor commands to transfer the proxy. Proxy is usually transferred if the Pegasus profile TRANSFER_PROXY is set, or the job is being run in the condor vanilla universe. The proxy is transferred from the submit host (i.e site local). The location is determined from the value of the X509_USER_PROXY profile key associated in the env namespace.

Specified by:

checkAndTransferProxy in interface Implementation

Parameters:

job - the transfer job .

Returns:

boolean true job was modified to transfer the proxy, else false when job is not modified.

checkAndTransferIrodsEnvFile

public boolean checkAndTransferIrodsEnvFile(TransferJob job)

Deprecated.

Determines if there is a need to transfer the irodsEnvFile for the transfer job or not. If there is a need to transfert the file, then the job is modified to create the correct condor commands to transfer the file. The file is transferred from the submit host (i.e site local).

Parameters:

job - the transfer job .

Returns:

boolean true job was modified to transfer the irodsEnvFile, else false when job is not modified.

setRefiner

public void setRefiner(Refiner refiner)

Sets the callback to the refiner, that has loaded this implementation.

Specified by:

setRefiner in interface Implementation

Parameters:

refiner - the transfer refiner that loaded the implementation.

addSetXBitJobs

protected boolean addSetXBitJobs(Job computeJob,
                                 Job txJob,
                                 Collection execFiles)

Adds the dirmanager to the workflow, that do a chmod on the files being staged.

Parameters:

computeJob - the computeJob for which the files are being staged.

txJob - the transfer job that is staging the files.

execFiles - the executable files that are being staged.

Returns:

boolean indicating whether any XBitJobs were succesfully added or not.

addSetXBitJobs

public boolean addSetXBitJobs(Job computeJob,
                              String txJobName,
                              Collection execFiles,
                              int transferClass)

Adds the dirmanager job to the workflow, that do a chmod on the files being staged.

Parameters:

computeJob - the computeJob for which the files are being staged.

txJobName - the name of the transfer job that is staging the files.

execFiles - the executable files that are being staged.

transferClass - the class of transfer job

Returns:

boolean indicating whether any XBitJobs were succesfully added or not.

createSetXBitJob

public Job createSetXBitJob(Job computeJob,
                            Collection<FileTransfer> execFiles,
                            int transferClass,
                            int xbitIndex)

Adds the dirmanager job to the workflow, that do a chmod on the files being staged.

Specified by:

createSetXBitJob in interface Implementation

Parameters:

computeJob - the computeJob for which the files are being staged.

execFiles - the executable files that are being staged.

transferClass - the class of transfer job

xbitIndex - index to be used for creating the name of XBitJob.

Returns:

the job object for the xBitJob

addSetXBitJobs

public boolean addSetXBitJobs(Job computeJob,
                              String txJobName,
                              Collection execFiles,
                              int transferClass,
                              int xbitIndex)

Adds the dirmanager job to the workflow, that do a chmod on the files being staged.

Specified by:

addSetXBitJobs in interface Implementation

Parameters:

computeJob - the computeJob for which the files are being staged.

txJobName - the name of the transfer job that is staging the files.

execFiles - the executable files that are being staged.

transferClass - the class of transfer job

xbitIndex - index to be used for creating the name of XBitJob.

Returns:

boolean indicating whether any XBitJobs were succesfully added or not.

getSetXBitJobName

public String getSetXBitJobName(String name,
                                int counter)

Generates the name of the setXBitJob , that is unique for the given workflow.

Specified by:

getSetXBitJobName in interface Implementation

Parameters:

name - the name of the compute job

counter - the index for the setXBit job.

Returns:

the name of the setXBitJob .

getNOOPJobName

public String getNOOPJobName(String name,
                             int counter)

Generates the name of the noop job , that is unique for the given workflow.

Parameters:

name - the name of the compute job

counter - the index for the noop job.

Returns:

the name of the setXBitJob .

createNoOPJob

public Job createNoOPJob(String name)

It creates a NoOP job that runs on the submit host.

Parameters:

name - the name to be assigned to the noop job

Returns:

the noop job.

createSetXBitJob

protected Job createSetXBitJob(Collection<FileTransfer> files,
                               String name,
                               String site)

Creates a dirmanager job, that does a chmod on the file being staged. The file being staged should be of type executable. Though no explicit check is made for that. The staged file is the one whose X bit would be set on execution of this job. The site at which job is executed, is determined from the site associated with the destination URL.

Parameters:

files - the collection FileTransfer containing the file that has to be X Bit Set.

name - the name that has to be assigned to the job.

site - the site at which the job has to be created

Returns:

the chmod job, else null if it is not able to be created for some reason.

createSetXBitJob

protected Job createSetXBitJob(FileTransfer file,
                               String name)

Creates a dirmanager job, that does a chmod on the file being staged. The file being staged should be of type executable. Though no explicit check is made for that. The staged file is the one whose X bit would be set on execution of this job. The site at which job is executed, is determined from the site associated with the destination URL.

Parameters:

file - the FileTransfer containing the file that has to be X Bit Set.

name - the name that has to be assigned to the job.

Returns:

the chmod job, else null if it is not able to be created for some reason.

defaultXBitTCEntry

private TransformationCatalogEntry defaultXBitTCEntry(String site)

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Parameters:

site - the site for which the default entry is required.

Returns:

the default entry.

determineDisabledChmodSites

protected Set determineDisabledChmodSites(String sites)

Builds up a set of disabled chmod sites

Parameters:

sites - comma separated list of sites.

Returns:

a Set containing the site names.

disableChmodJobCreation

protected boolean disableChmodJobCreation(String site)

Returns a boolean indicating whether to disable chmod job creation for a site or not.

Parameters:

site - the name of the site

Returns:

boolean

getPriority

protected String getPriority(TransferJob job)

Returns the priority for the transfer job as specified in the properties file.

Parameters:

job - the Transfer job.

Returns:

the priority of the job as determined from properties, can be null if invalid value passed or property not set.

construct

protected void construct(Job job,
                         String key,
                         String value)

Constructs a condor variable in the condor profile namespace associated with the job. Overrides any preexisting key values.

Parameters:

job - contains the job description.

key - the key of the profile.

value - the associated value.