edu.isi.pegasus.planner.refiner
Class InterPoolEngine

java.lang.Object
  edu.isi.pegasus.planner.refiner.Engine
      edu.isi.pegasus.planner.refiner.InterPoolEngine

All Implemented Interfaces:

Refiner

public class InterPoolEngine
extends Engine
implements Refiner

This engine calls out to the Site Selector selected by the user and maps the jobs in the workflow to the execution pools.

Version:

$Revision: 5256 $

Author:

Karan Vahi, Gaurang Mehta

Field Summary

private ADag	mDag ADag object corresponding to the Dag whose jobs we want to schedule.
protected TransformationStore	mDAXTransformationStore Handle to the transformation store that stores the transformation catalog user specifies in the DAX
private Set	mExecPools Set of the execution pools which the user has specified.
private SiteSelector	mSiteSelector Handle to the site selector.
private SLS	mSLS The handle to the SLS implementor
private Mapper	mTCMapper The handle to the transformation catalog mapper object that caches the queries to the transformation catalog, and indexes them according to lfn's.
private TransformationSelector	mTXSelector The handle to the transformation selector, that ends up selecting what transformations to pick up.
private boolean	mWorkerNodeExecution A boolean indicating whether to have worker node execution or not.
private XMLProducer	mXMLStore The XML Producer object that records the actions.

Fields inherited from class edu.isi.pegasus.planner.refiner.Engine

mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE

Fields inherited from interface edu.isi.pegasus.planner.refiner.Refiner

VERSION

Constructor Summary

InterPoolEngine(ADag aDag, PegasusBag bag)
Overloaded constructor.

InterPoolEngine(PegasusBag bag)
Default constructor.

Method Summary

List	convertToList(Set s) Converts a Set to a List.
List	convertToList(Vector v) Converts a Vector to a List.
void	determineSites() This is where the callout to the Partitioner should take place, that partitions the workflow into clusters and sends to the site selector only those list of jobs that are ready to be scheduled.
private String	getJobManager(String site, String universe) It returns a jobmanager for the given pool.
PegasusBag	getPegasusBag() Returns the bag of intialization objects.
String	getStagingSite(Job job) Returns the staging site to be used for a job.
ADag	getWorkflow() Returns a reference to the workflow that is being refined by the refiner.
XMLProducer	getXMLProducer() Returns a reference to the XMLProducer, that generates the XML fragment capturing the actions of the refiner.
private void	handleDependantExecutables(Job job) Handles the dependant executables that need to be staged.
private boolean	incorporateHint(Job job, String key) It incorporates a hint in the namespace to the job.
private boolean	incorporateProfiles(Job job) Incorporates the profiles from the various sources into the job.
protected void	logRefinerAction(Job job) Logs the action taken by the refiner on a job as a XML fragment in the XML Producer.
void	scheduleJobs(ADag dag, List sites) It schedules a list of jobs on the execution pools by calling out to the site selector specified.
private TransformationCatalogEntry	selectTCEntry(List entries, Job job, String selector) Calls out to the transformation selector to select an entry from a list of valid transformation catalog entries.

Methods inherited from class edu.isi.pegasus.planner.refiner.Engine

addVector, appendArrayList, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mDag

private ADag mDag

ADag object corresponding to the Dag whose jobs we want to schedule.

mExecPools

private Set mExecPools

Set of the execution pools which the user has specified.

mSiteSelector

private SiteSelector mSiteSelector

Handle to the site selector.

mTXSelector

private TransformationSelector mTXSelector

The handle to the transformation selector, that ends up selecting what transformations to pick up.

mTCMapper

private Mapper mTCMapper

The handle to the transformation catalog mapper object that caches the queries to the transformation catalog, and indexes them according to lfn's. There is no purge policy in the TC Mapper, so per se it is not a classic cache.

mXMLStore

private XMLProducer mXMLStore

The XML Producer object that records the actions.

mSLS

private SLS mSLS

The handle to the SLS implementor

mWorkerNodeExecution

private boolean mWorkerNodeExecution

A boolean indicating whether to have worker node execution or not.

mDAXTransformationStore

protected TransformationStore mDAXTransformationStore

Handle to the transformation store that stores the transformation catalog user specifies in the DAX

Constructor Detail

InterPoolEngine

public InterPoolEngine(PegasusBag bag)

Default constructor.

Parameters:

bag - the bag of initialization objects.

InterPoolEngine

public InterPoolEngine(ADag aDag,
                       PegasusBag bag)

Overloaded constructor.

Parameters:

aDag - the ADag object corresponding to the Dag for which we want to determine on which pools to run the nodes of the Dag.

bag - the bag of initialization objects

Method Detail

getPegasusBag

public PegasusBag getPegasusBag()

Returns the bag of intialization objects.

Returns:

PegasusBag

getWorkflow

public ADag getWorkflow()

Returns a reference to the workflow that is being refined by the refiner.

Specified by:

getWorkflow in interface Refiner

Returns:

ADAG object.

getXMLProducer

public XMLProducer getXMLProducer()

Returns a reference to the XMLProducer, that generates the XML fragment capturing the actions of the refiner. This is used for provenace purposes.

Specified by:

getXMLProducer in interface Refiner

Returns:

XMLProducer

determineSites

public void determineSites()

This is where the callout to the Partitioner should take place, that partitions the workflow into clusters and sends to the site selector only those list of jobs that are ready to be scheduled.

scheduleJobs

public void scheduleJobs(ADag dag,
                         List sites)

It schedules a list of jobs on the execution pools by calling out to the site selector specified. It is upto to the site selector to determine if the job can be run on the list of sites passed.

Parameters:

dag - the abstract workflow.

sites - the list of execution sites, specified by the user.

getStagingSite

public String getStagingSite(Job job)

Returns the staging site to be used for a job. If a staging site is not determined from the options it is set to be the execution site for the job

Parameters:

job - the job for which to determine the staging site

Returns:

the staging site

incorporateProfiles

private boolean incorporateProfiles(Job job)

Incorporates the profiles from the various sources into the job. The profiles are incorporated in the order pool, transformation catalog, and properties file, with the profiles from the properties file having the highest priority. It is here where the transformation selector is called to select amongst the various transformations returned by the TC Mapper.

Parameters:

job - the job into which the profiles have been incorporated.

Returns:

true profiles were successfully incorporated. false otherwise

handleDependantExecutables

private void handleDependantExecutables(Job job)

Handles the dependant executables that need to be staged.

Parameters:

job - Job

selectTCEntry

private TransformationCatalogEntry selectTCEntry(List entries,
                                                 Job job,
                                                 String selector)

Calls out to the transformation selector to select an entry from a list of valid transformation catalog entries.

Parameters:

entries - list of TransformationCatalogEntry objects.

job - the job.

selectors - the selector to be called

Returns:

the selected TransformationCatalogEntry object null when transformation selector is unable to select any transformation

getJobManager

private String getJobManager(String site,
                             String universe)

It returns a jobmanager for the given pool.

Parameters:

site - the name of the pool.

universe - the universe for which you need the scheduler on that particular pool.

Returns:

the jobmanager for that pool and universe. null if not found.

incorporateHint

private boolean incorporateHint(Job job,
                                String key)

It incorporates a hint in the namespace to the job. After the hint is incorporated the key is deleted from the hint namespace for that job.

Parameters:

job - the job that needs the hint to be incorporated.

key - the key in the hint namespace.

Returns:

true the hint was successfully incorporated. false the hint was not set in job or was not successfully incorporated.

convertToList

public List convertToList(Vector v)

Converts a Vector to a List. It only copies by reference.

Parameters:

v - Vector

Returns:

a ArrayList

convertToList

public List convertToList(Set s)

Converts a Set to a List. It only copies by reference.

Parameters:

s - Set

Returns:

a ArrayList

logRefinerAction

protected void logRefinerAction(Job job)

Logs the action taken by the refiner on a job as a XML fragment in the XML Producer.

Parameters:

job - the Job containing the job that was mapped to a site.