Task farming

brian.tools.taskfarm.run_tasks(dataman, task, items, gui=True, poolsize=0, initargs=None, initkwds=None, verbose=None, numitems=None)

Run a series of tasks using multiple CPUs on a single computer.

Initialised with arguments:

dataman

The DataManager object used to store the results in, see below.

task

The task function or class (see below).

items

A sequence (e.g. list or iterator) of arguments to be passed to the task.

gui=True

Whether or not to use a Tkinter based GUI to show progress and terminate the task run.

poolsize=0

The number of CPUs to use. If the value is 0, use all available CPUs, if it is -1 use all but one CPU, etc.

initargs, initkwds

If task is a class, these are the initialisation arguments and keywords for the class.

verbose=None

Specify True or False to print out every progress message (defaults to False if the GUI is used, or True if not).

numitems=None

For iterables (rather than fixed length sequences), if you specify the number of items, an estimate of the time remaining will be given.

The task (defined by a function or class, see below) will be called on each item in items, and the results saved to dataman. Results are stored in the format (key, val) where key is a unique but meaningless identifier. Results can be retrieved using dataman.values() or (for large data sets that should be iterated over) dataman.itervalues().

The task can either be a function or a class. If it is a function, it will be called for each item in items. If the items are tuples, the function will be called with those tuples as arguments (e.g. if the item is (1,2,3) the function will be called as task(1, 2, 3)). If the task is a class, it can have an __init__ method that is called once for each process (each CPU) at the beginning of the task run. If the __init__ method has a process_number argument, it will be passed an integer value from 0 to numprocesses-1 giving the number of the process (note, this is not the process ID). The class should define a __call__ method that behaves the same as above for task being a function. In both cases (function or class), if the arguments include a keyword report then it will be passed a value that can be passed as the report keyword in Brian’s run() function to give feedback on the simulation as it runs. A task function can also set self.taskname as a string that will be displayed on the GUI to give additional information.

Warning

On Windows, make sure that run_tasks() is only called from within a block such as:

if __name__=='__main__':
    run_tasks(...)

Otherwise, the program will go into a recursive loop.

Note that this class only allows you to run tasks on a single computer, to distribute work over multiple computers, we suggest using Playdoh.