edbaskerville.com

GridSweeper’s file transfer mechanism is designed to allow transfer of input and output files in a grid system-independent way. In environments without a real IT infrastructure, such as my own ad-hoc Xgrid setup, a shared filesystem will not necessarily be available, so you need a way to stage and retrieve files.

The GridSweeper code is file transfer system-agnostic, providing a simple interface that can be implemented for a particular file transfer system (e.g., FTP, which is included as a working example). The interface requires just a few basic methods: connect(), disconnect(), uploadFile, deleteFile, makeDirectory(), removeDirectory, list(), and isDirectory(), which all do pretty much what you’d expect. There is no notion of a working directory, so all paths are relative to the implicit root of the file system. Particular file transfer systems can define custom properties to affect setup—e.g., the FTP system provides properties for setting the hostname, username/password, root directory (so the GridSweeper root need not be the same as the FTP server’s default working directory), etc.

Here’s how a GridSweeper run interacts with the file system:

1. The experiment setup data includes a list of input files, mapping (absolute) paths on the local filesystem for the submit host to relative paths in the working directory of the running job. When the job is submitted, those files are copied into an input-file directory on the file transfer system, within the location experimentName/submissionDate/submissionTime/input/.
2. The experiment setup data also includes a list of output file paths, relative to the working directory of the running job. This list is part of the input data for a running GridSweeper job.
3. The GridSweeperRunner tool, which is the process actually executed by the grid system, begins the process by transferring any files in the input directory on the file transfer system into the working directory as specified. After the run is complete, it copies the specified output files back to the file transfer system to the location experimentName/submissionDate/submissionTime/caseDir/filename, where caseDir is a directory name representing the particular parameter settings for the run (“b=0.1-g=25″). If the filename includes the wildcard $gs_rn_ph$, that will be replaced by the current run number. If it does not, the run number will be appended as an extension (filename.runNumber).
4. When each run is complete, the submit host, which is monitoring the activity, retrieves files back to the local experiments directory. If the submit host stopped monitoring, there should be a way to go back and retrieve files not yet retrieved; I haven’t designed this mechanism yet.

Note: as of 4:30 PM, July 5, 2007, this is not all implemented correctly.

Thought: it’s possible, though unlikely, that file transfer system collisions may occur from multiple people submitting identically-named experiments at the same time. I can imagine a lab class with people following the same tutorial instructions all submitting identically-named jobs at the same time. So maybe it’s better to name these directories with unique hashes. Assuming no collisions, though, it doesn’t matter from the user perspective, so this can be changed later; the current naming scheme is nice for debugging.

If you have a shared filesystem, of course, none of this is necessary!

Between driving from San Francisco and selling furniture on Craigslist, this weekend I wrote preliminary Javadoc for all of last summer’s GridSweeper work. A very valuable exercise before diving into coding: it made me look through every single method I wrote and say something about it. It also brought a number of design flaws to my attention, duly noted in TODO comments.

The purpose of GridSweeper is to take a simple user-provided description of what parameter settings to run a model with, run the model on a grid, and return results to the user.

The user will be able to manipulate the parameter-sweep description in three ways: (1) using an XML specification file, (2) with command-line arguments, and (3) with a graphical user interface. These three mechanisms can be mixed: command-line arguments can augment or override XML as well as be saved back out to XML, and the GUI tool will serve to edit and save XML files as well.

Ultimately, user action will result in running the GridSweeper program, which turns parameter sweep specifications into job specifications for the grid system via DRMAA. Specifically, the program does the following:

1. Parses the XML specification and command-line arguments to generate an Experiment object.
2. Generates a list of ExperimentCase objects (parameter value settings) from the Experiment class.
3. Sets up an output directory for the files generated by this experiment. If a shared filesystem is not present, this can be done via FTP or other file-transfer mechanism supported by a plugin implementing a subclass of FileTransferSystem.
4. Starts a DRMAA session and submits a job for each experiment case, using an archived RunSetup object for each job’s standard input.
5. Still unimplemented: monitors the results of jobs and reports status changes to the user.

The way things are set up now, GridSweeper requires support on both the submission end and the execution end of the grid. The DRMAA job specification specifies that the execution host run not the model itself, but the GridSweeperRunner program, which takes input data and uses that to actually run the model. Specifically, it does the following:

1. Unarchives the RunSetup object from standard input.
2. If necessary, downloads input files via the file transfer mechanism.
3. Actually runs the model using an instance of the Adapter class specified by the user (explicitly, or implicitly by using, e.g., gdrone for the Drone compatibility adapter). The Adapter object knows how to take a set of parameters and send it to a particular type of model executable.
4. If necessary, uploads output files via the file transfer mechanism.

One problem with this mechanism is that it submits a separate job for every experiment case, bypassing DRMAA’s notion of batch jobs. DRMAA batch jobs let you submit a whole bunch of jobs at the same time by specifying that each job is the same except for an integer specifier, and that specifier can be used as a variable in command-line arguments. Because some systems may be faster at accepting batch jobs than a pile of individual jobs, it might be worth using the batch job mechanism.

One way to do this would be to defer the calculation of parameter assignments and random seeds to the execution host, but that makes it impossible to generate a file for reproducing the experiment as soon as it is submitted. A better way is to generate a series of input files in the experiment directory, named with the batch run index, and have the GridSweeperRunner tool read those files at runtime rather than reading an object from standard input.

As currently conceived, GridSweeper will consist of a set of Java classes in JAR files, additional Java classes as plugins (plugin format to be determined, but will include a JAR file), and shell scripts to simplify this:

java -cp ${GRIDSWEEPER_ROOT}/classes/GridSweeper.jar \\
    com.edbaskerville.gridsweeper.GridSweeper [args]

into this:

gsweep [args]

The top level of the hierarchy will be designated by the environment variable $GRIDSWEEPER_ROOT, within which the following tree will exist:

$GRIDSWEEPER_ROOT/
    bin/
        gsweep
            (main GridSweeper submission executable)
        gdrone
            (shortcut to gsweep -a com.edbaskerville.gridsweeper.DroneAdapter)
        grunner
            (wrapper to actually execute jobs on the agent machine)
        ...
            (other scripts to shortcut, e.g., the Repast adapter)
    classes/
        classes.jar
            (all classes except those with main methods)
        GridSweeper.jar
            (app/tool class)
        GridSweeperRunner.jar
            (class to actually run simulations on agents)
    plugins/
        (contains add-on adapters and file-transfer systems)

First things first: this post covers how to get the GridSweeper build environment set up on your machine. I’ve developed GridSweeper entirely with Eclipse, but the build process uses Ant, so it can be run from the command line as well (or, theoretically, any other Ant-compatible IDE).

To get GridSweeper building on your machine, you’ll need to get threefour things:

1. The code distribution (trunk), checked out into your Eclipse workspace. Soon to be hosted at CSCS.
2. An implementation of the Java Distributed Resource Management Application API (DRMAA). For CSCS/Linux, you should use the one provided by the Sun Grid Engine (in /appl/sge/drmaa.jar on CSCS machines). For building on my Mac, I’m using my XgridDRMAA implementation.
3. Jakarta Commons Net (download page). This is for FTP file transfer, which won’t actually be relevant for CSCS—maybe I can modify the build system to make this optional.
4. Jakarta ORO (download page), also for FTP. You won’t even realize you’re missing this until you get an obscure class not found error at runtime when using any of the FTP directory methods.

If you’re using Eclipse (recommended), open up the project in your workspace. Add the DRMAA and Jakarta Commons Net jar files (Project > Properties > Java Build Path > Add External JARs…), and, in theory, the project should build.

Next: evaluating the code.