edu.isi.pegasus.planner.transfer.sls
Class Transfer

java.lang.Object
  edu.isi.pegasus.planner.transfer.sls.Transfer

All Implemented Interfaces:

SLS

public class Transfer
extends Object
implements SLS

This uses the transfer executable distributed with Pegasus to do the second level staging.

Version:

$Revision: 4780 $

Author:

Karan Vahi

Field Summary

static String	DERIVATION_NAME The name of the underlying derivation.
static String	DERIVATION_NAMESPACE The derivation namespace for for the transfer job.
static String	DERIVATION_VERSION The derivation version number for the transfer job.
static String	DESCRIPTION A short description of the transfer implementation.
static String	EXECUTABLE_BASENAME The executable basename
protected String	mExtraArguments Any extra arguments that need to be passed ahead to the s3 client invocation.
protected String	mLocalURLPrefix The local url prefix for the submit host.
protected LogManager	mLogger The handle to the logging manager.
protected PegasusProperties	mProps The handle to the properties.
protected boolean	mSeqExecGridStartUsed Boolean to track whether the gridstart used in PegasusLite or not
protected SiteStore	mSiteStore The handle to the site catalog.
protected boolean	mStageSLSFile Boolean to track whether to stage sls file or not
protected TransformationCatalog	mTCHandle The handle to the transformation catalog.
protected ReplicaCatalog	mTransientRC The handle to the transient replica catalog.
static String	TRANSFORMATION_NAME The name of the underlying transformation that is queried for in the Transformation Catalog.
static String	TRANSFORMATION_NAMESPACE The transformation namespace for the transfer job.
static String	TRANSFORMATION_VERSION The version number for the transfer job.

Fields inherited from interface edu.isi.pegasus.planner.transfer.SLS

VERSION

Constructor Summary

Transfer()
The default constructor.

Method Summary

protected TransformationCatalogEntry	defaultTCEntry(String namespace, String name, String version, String executableBasename, String site) Returns a default TC entry to be used in case entry is not found in the transformation catalog.
Collection<FileTransfer>	determineSLSInputTransfers(Job job, String fileName, String submitDir, String stagingSiteDirectory, String workerNodeDirectory) Generates a second level staging file of the input files to the worker node directory.
Collection<FileTransfer>	determineSLSOutputTransfers(Job job, String fileName, String submitDir, String stagingSiteDirectory, String workerNodeDirectory) Generates a second level staging file of the input files to the worker node directory.
boolean	doesCondorModifications() Returns a boolean whether the SLS implementation does a condor based modification or not.
protected List	getEnvironmentVariables(String site) Returns the environment profiles that are required for the default entry to sensibly work.
protected String	getExecutableBasename() Return the executable basename for transfer executable used.
String	getSLSInputLFN(Job job) Returns the LFN of sls input file.
String	getSLSOutputLFN(Job job) Returns the LFN of sls output file.
TransformationCatalogEntry	getTransformationCatalogEntry(String siteHandle) Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation.
void	initialize(PegasusBag bag) Initializes the SLS implementation.
String	invocationString(Job job, File slsFile) Constructs a command line invocation for a job, with a given sls file.
boolean	modifyJobForFirstLevelStaging(Job job, String submitDir, String slsInputLFN, String slsOutputLFN) Modifies a job for the first level staging to headnode.This is to add any files that needs to be staged to the head node for a job specific to the SLS implementation.
boolean	modifyJobForWorkerNodeExecution(Job job, String stagingSiteURLPrefix, String stagingSitedirectory, String workerNodeDirectory) Modifies a compute job for second level staging.
boolean	needsSLSInputTransfers(Job job) Returns a boolean indicating whether it will an input file for a job to do the transfers.
boolean	needsSLSOutputTransfers(Job job) Returns a boolean indicating whether it will an output file for a job to do the transfers.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TRANSFORMATION_NAMESPACE

public static final String TRANSFORMATION_NAMESPACE

The transformation namespace for the transfer job.

See Also:

Constant Field Values

TRANSFORMATION_NAME

public static final String TRANSFORMATION_NAME

The name of the underlying transformation that is queried for in the Transformation Catalog.

See Also:

Constant Field Values

TRANSFORMATION_VERSION

public static final String TRANSFORMATION_VERSION

The version number for the transfer job.

DERIVATION_NAMESPACE

public static final String DERIVATION_NAMESPACE

The derivation namespace for for the transfer job.

See Also:

Constant Field Values

DERIVATION_NAME

public static final String DERIVATION_NAME

The name of the underlying derivation.

See Also:

Constant Field Values

DERIVATION_VERSION

public static final String DERIVATION_VERSION

The derivation version number for the transfer job.

See Also:

Constant Field Values

DESCRIPTION

public static final String DESCRIPTION

A short description of the transfer implementation.

See Also:

Constant Field Values

EXECUTABLE_BASENAME

public static final String EXECUTABLE_BASENAME

The executable basename

See Also:

Constant Field Values

mSiteStore

protected SiteStore mSiteStore

The handle to the site catalog.

mTCHandle

protected TransformationCatalog mTCHandle

The handle to the transformation catalog.

mProps

protected PegasusProperties mProps

The handle to the properties.

mLogger

protected LogManager mLogger

The handle to the logging manager.

mLocalURLPrefix

protected String mLocalURLPrefix

The local url prefix for the submit host.

mTransientRC

protected ReplicaCatalog mTransientRC

The handle to the transient replica catalog.

mExtraArguments

protected String mExtraArguments

Any extra arguments that need to be passed ahead to the s3 client invocation.

mStageSLSFile

protected boolean mStageSLSFile

Boolean to track whether to stage sls file or not

mSeqExecGridStartUsed

protected boolean mSeqExecGridStartUsed

Boolean to track whether the gridstart used in PegasusLite or not

Constructor Detail

Transfer

public Transfer()

The default constructor.

Method Detail

initialize

public void initialize(PegasusBag bag)

Initializes the SLS implementation.

Specified by:

initialize in interface SLS

Parameters:

bag - the bag of objects. Contains access to catalogs etc.

doesCondorModifications

public boolean doesCondorModifications()

Returns a boolean whether the SLS implementation does a condor based modification or not. By condor based modification we mean whether it uses condor specific classads to achieve the second level staging or not.

Specified by:

doesCondorModifications in interface SLS

Returns:

false

invocationString

public String invocationString(Job job,
                               File slsFile)

Constructs a command line invocation for a job, with a given sls file. The SLS maybe null. In the case where SLS impl does not read from a file, it is advised to create a file in generateSLSXXX methods, and then read the file in this function and put it on the command line.

Specified by:

invocationString in interface SLS

Parameters:

job - the job that is being sls enabled

slsFile - the slsFile can be null

Returns:

invocation string

needsSLSInputTransfers

public boolean needsSLSInputTransfers(Job job)

Returns a boolean indicating whether it will an input file for a job to do the transfers. Transfer reads from stdin the file transfers that it needs to do. Always returns true, as we need to transfer the proxy always.

Specified by:

needsSLSInputTransfers in interface SLS

Parameters:

job - the job being detected.

Returns:

true

needsSLSOutputTransfers

public boolean needsSLSOutputTransfers(Job job)

Returns a boolean indicating whether it will an output file for a job to do the transfers. Transfer reads from stdin the file transfers that it needs to do.

Specified by:

needsSLSOutputTransfers in interface SLS

Parameters:

job - the job being detected.

Returns:

true

getSLSInputLFN

public String getSLSInputLFN(Job job)

Returns the LFN of sls input file.

Specified by:

getSLSInputLFN in interface SLS

Parameters:

job - Job

Returns:

the name of the sls input file.

getSLSOutputLFN

public String getSLSOutputLFN(Job job)

Returns the LFN of sls output file.

Specified by:

getSLSOutputLFN in interface SLS

Parameters:

job - Job

Returns:

the name of the sls input file.

determineSLSInputTransfers

public Collection<FileTransfer> determineSLSInputTransfers(Job job,
                                                           String fileName,
                                                           String submitDir,
                                                           String stagingSiteDirectory,
                                                           String workerNodeDirectory)

Generates a second level staging file of the input files to the worker node directory.

Specified by:

determineSLSInputTransfers in interface SLS

Parameters:

job - job for which the file is being created

fileName - name of the file that needs to be written out.

submitDir - submit directory where it has to be written out.

stagingSiteDirectory - directory on the head node of the staging site.

workerNodeDirectory - worker node directory

Returns:

a Collection of FileTransfer objects listing the transfers that need to be done.

See Also:

needsSLSInputTransfers( Job)

determineSLSOutputTransfers

public Collection<FileTransfer> determineSLSOutputTransfers(Job job,
                                                            String fileName,
                                                            String submitDir,
                                                            String stagingSiteDirectory,
                                                            String workerNodeDirectory)

Generates a second level staging file of the input files to the worker node directory.

Specified by:

determineSLSOutputTransfers in interface SLS

Parameters:

job - the job for which the file is being created

fileName - the name of the file that needs to be written out.

submitDir - the submit directory where it has to be written out.

stagingSiteDirectory - the directory on the head node of the staging site.

workerNodeDirectory - the worker node directory

Returns:

a Collection of FileTransfer objects listing the transfers that need to be done.

See Also:

needsSLSOutputTransfers( Job)

modifyJobForFirstLevelStaging

public boolean modifyJobForFirstLevelStaging(Job job,
                                             String submitDir,
                                             String slsInputLFN,
                                             String slsOutputLFN)

Modifies a job for the first level staging to headnode.This is to add any files that needs to be staged to the head node for a job specific to the SLS implementation. If any file needs to be added, a FileTransfer object should be created and added as an input or an output file.

Specified by:

modifyJobForFirstLevelStaging in interface SLS

Parameters:

job - the job

submitDir - the submit directory

slsInputLFN - the sls input file if required, that is used for staging in from the head node to worker node directory.

slsOutputLFN - the sls output file if required, that is used for staging in from the head node to worker node directory.

Returns:

boolean

modifyJobForWorkerNodeExecution

public boolean modifyJobForWorkerNodeExecution(Job job,
                                               String stagingSiteURLPrefix,
                                               String stagingSitedirectory,
                                               String workerNodeDirectory)

Modifies a compute job for second level staging. The only modification it does is add the appropriate environment varialbes to the job

Specified by:

modifyJobForWorkerNodeExecution in interface SLS

Parameters:

job - the job to be modified.

stagingSiteURLPrefix - the url prefix for the server on the staging site

stagingSitedirectory - the directory on the staging site, where the inp

workerNodeDirectory - the directory in the worker node tmp

Returns:

boolean indicating whether job was successfully modified or not.

getTransformationCatalogEntry

public TransformationCatalogEntry getTransformationCatalogEntry(String siteHandle)

Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation. If an entry is not specified in the Transformation Catalog, then null is returned.

Parameters:

siteHandle - the handle of the site where the transformation is to be searched.

Returns:

the transformation catalog entry if found, else null.

defaultTCEntry

protected TransformationCatalogEntry defaultTCEntry(String namespace,
                                                    String name,
                                                    String version,
                                                    String executableBasename,
                                                    String site)

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Parameters:

namespace - the namespace of the transfer transformation

name - the logical name of the transfer transformation

version - the version of the transfer transformation

executableBasename - the basename of the executable

site - the site for which the default entry is required.

Returns:

the default entry.

getEnvironmentVariables

protected List getEnvironmentVariables(String site)

Returns the environment profiles that are required for the default entry to sensibly work. Tries to retrieve the following variables

 PEGASUS_HOME
 GLOBUS_LOCATION
 LD_LIBRARY_PATH
 

Parameters:

site - the site where the job is going to run.

Returns:

List of environment variables, else empty list if none are found

getExecutableBasename

protected String getExecutableBasename()

Return the executable basename for transfer executable used.

Returns:

the executable basename.