In a sophisticated network setup like a typical Sun Grid Engine installation, a GridSweeper user will have the luxury of a shared filesystem, a network home directory, etc., etc., meaning that no files will need to be transferred as part of job submission. However, this isn’t always the case. With extra work, it’s apparently possible to set up SGE without a shared filesystem. And many Xgrid users, especially if they’re installing Xgrid for the sake of using GridSweeper on their simple Repast model and network of four Macs, won’t have any shared file system at all.
Although Xgrid provides built-in facilities for transferring files, SGE and DRMAA do not—today, the typical user of these systems is on a well managed network. But I want GridSweeper to be easy to set up for any Joe Repast modeler with a few computers. Although it might seem like too much network overhead to send an model’s executable code, plus input data, and retrieve output data on the other end. But compared to typical runtimes for ABMs, the time it takes to transfer a little executable is nothing. So providing a general, easy solution to this problem, I think, is vital.
All the obvious solutions to the file transfer problem come down to requiring the user to set up some kind of file-transfer infrastructure—NFS, AFP, FTP, SCP, etc., etc. But this defeats the whole purpose of ease of use: now they have to set something up!
All roads, in my view, point to including a simple file server as part of GridSweeper. This daemon will typically run on the same machine as, say, the Xgrid controller or SGE qmaster. The client GUI and command line will provide tools to add files to the GridSweeper file daemon. Additionally, clients will be able to upload files on a per-batch basis. When the agent/execution host* starts running, first it will see if it needs to download any files from the file daemon. (Timestamp checking & caching will ensure that if five runs on the same host all need the same file, it will only be downloaded once.)
At the end of a run, the GridSweeper monitor will send any output files back to the file daemon. Using a monitoring tool/GUI, the user will be able to download any results.
* this terminology difference between SGE and Xgrid is really starting to get to me. I bet DRMAA has its own set of terms.