Wagon: a generic pull-model job distribution service

introduction

Wagon presents a really simple mechanism for farming out work to a number of volunteers.

• firewall friendly HTTP
• C/C++ CGIs: fast and statically linkable
• atomic filesystem operations, no file locking
• command-line client and submission tools
• easily adapted to different job types

the basic model:

slightly more detailed:

1. operator builds a job (into a cdb) and submits it
2. a CGI picks it up and runs a verify program on it
3. if it passes the verify program, the job is queued
4. a few seconds later, a client polls another CGI and picks the job up
5. the client runs a job proxy which runs the real worker program in a scratch directory
6. if there are any results, the client collects them and sends them back
7. another CGI receives the results, and runs another verifier on them to check that they make sense
8. if the results make sense, they are deposited along with the completed job in a directory for pickup
9. the operator can pick the results up at their leisure, just using a HTTP GET, or ask a CGI to delete them

see? simple.

customization

as you may have guessed, in order to customize wagon to run your job, you need to write 3 programs: an input verifier, a result verifier, and a proxy. don't worry though: these can be very simple programs. an input verifier for a job type foo is just some program called foo-job-verify which returns 0 iff its single command line argument represents a valid job of type foo. a result verifier is the same, only it's called foo-result-verify and it verifies results rather than jobs. a proxy is a program which wraps another program's invocation interface: the proxy is provided with 2 command line arguments, and it must endeavour to turn the first into the second by way of the program it's wrapping. for a job of type foo, the proxy is called foo-proxy.

example

#!/bin/sh
cdbget job <$1 \
 | grep -q '^#.*\(B3LYP\|SCF\)'
	  

#!/bin/sh
IN=$1
OUT=$2
cdbget job <$IN >job.com
LINES=`cdbget lines <$IN`
nice -20 g94 job.com
tail -n $LINES job.log >$OUT
rm job.com job.log
	  

#!/bin/sh
cat $1 | grep -q 'Job cpu time'
	  

No, this doesn't prevent someone from submitting really wacky jobs or results, it's just a plausible sanity check. For input checking, I suggest you adopt a public key signature (such as sigs). You'll have to work out for yourself if there's any way to check the validity of results, for your particular job type. If there isn't, you're just going to have to put up with annoying people who think it's clever to submit bogus results. there is a more thourough tutorial inside the distribution, as well as online, here.

download

• wagon-1.0-1.i386.rpm
• wagon-1.0.tar.gz