Return to README CVS log

Up to [venge] / src / wagon

fixups.

Wagon is a simple, portable, lightweight, pull-model work distribution system
designed to farm out large scientific jobs over HTTP to a number of volunteer
computers participating in a large task.

it currently has built on FreeBSD 4.0 and Red Hat Linux 6.2.

***** IMPORTANT IMPORTANT IMPORTANT IMPORTANT *****
***** IMPORTANT IMPORTANT IMPORTANT IMPORTANT *****

if you install a wagon server, you MUST select one of the following three
options or else it will silently not work, do nothing, and annoy you
greatly. and you will email me and say "wagon does not work" and I will simply
ask you if you read the README file in the section marked with rediculous
IMPORTANT markings. and you will feel stupid and low and go throw yourself under
a truck, and your important scientific work will never get done. so choose one
of these, convince yourself that you know how to do it, read the associated man
pages, and do NOT complain if your server doesn't run because you were silly
enough to not read this and pay attention to it:

(1) apache runs CGIs as user "nobody" (or something), so you must make the
directory structure under the wagon CGIs writeable by user "nobody" (or whoever
apache runs CGIs as)

(2) apache runs CGIs as _you_, using some sort of nice setuid-cgi wrapper, so
you can just leave the directories as owned by you.

(3) apache runs CGIs as nobody, and you're not the sysadmin, so you must install
the wagon CGIs as setuid _you_ (4755), in order to give them access to the
directories you own. the usual warnings about setuid apply: it causes testicular
cancer and tax audits and whatnot, your life will end if you use it, but it will
get you around this problem.

***** IMPORTANT IMPORTANT IMPORTANT IMPORTANT *****
***** IMPORTANT IMPORTANT IMPORTANT IMPORTANT *****





USER PROGRAMS
~~~~~~~~~~~~~

wagon-build <name> <type> <filename> [key val]+

	builds a job named <name> of type <type> out of <filename>, optionally
	annotating the job with any extra key/val pairs provided. returns the
	name of the newly constructed job file.

	example:

	kat-moda:~ $ wagon-build 'water SCF test' g94 water.com
	398a5a86-kat-moda-474-1


wagon-push <url> <filename>+

	submits one or more files via HTTP upload to the specified URL.

	example:
	kat-moda:~ $ wagon-build 'water SCF test' g94 water.com
	398a5a86-kat-moda-474-1
	kat-moda:~ $ wagon-push http://jobhost/job-post 398a5a86-kat-moda-474-1


wagon-pull <url>

	fetches a job from <url>, reading first the filename, then a newline,
	then the contents of the job. writes out to the named file (if it's a
	safe name) in the current working directory.


wagon-client

	repeatedly runs "wagon-pull <url>/wagon-dispatch" in directory todo/,
	where <url> is loaded from cfg/url, pausing <pause> seconds between
	requests, where <pause> is read in from cfg/pause. if a job of type
	<type> appears in todo/, makes a unique directory <guid> in scr/ and
	runs "prx/<type>-proxy scr/<guid>/in scr/<guid>/out" in directory
	scr/<guid>, such that prx/<type>-proxy only needs to transform its
	first command line argument into its second. if scr/<guid>/out is
	produced, it is linked to fin/<jobname>, otherwise an error message is
	written to fin/<jobname>. any files found in fin/ cause wagon-client to
	run "wagon-push <url>/wagon-collect fin/<jobname>". all scratch
	directories in src/ are cleaned after each job terminates, and all todo/
	and fin/ files are unlinked (somewhat aggressively, to avoid infinite
	loops).

	example:

	$ mkdir todo scr cfg fin prx
	$ echo -n "http://jobhost" >cfg/url
	$ echo -n "10" >cfg/pause
	$ cat >prx/g94-proxy
	#!/bin/sh
	~/jobs/cdbget job <$1 >job.com
	nice -20 g94 job.com
	cp job.log $2
	^D
	$ ./wagon-client &



CGI PROGRAMS
~~~~~~~~~~~~

wagon-accept

	a CGI program which accepts incoming jobs, writing them to a new unique
	name in the tmp/ directory, then inspecting the job, determining its
	type, running jv/<type>-job-verify tmp/<job>. if the verification
	succeeds atomically linking <job> into the new/ directory. if a job
	makes it into new/, the job is queued for dispatch to the first
	available client.


wagon-dispatch

	a CGI program which links a job from the new/ directory to the run/
	directory, then feeds the job name followed by a newline and the job
	contents to the client, unlinking the entry in new/.


wagon-collect
	
	a CGI program which accepts incoming results to tmp/, inspects the
	corresponding entry in run/, determines the job type, and runs
	rv/<type>-result-verify tmp/<result> run/<job> on the uploaded
	results. if the results verify, it then proceeds to link the
	corresponding job from run/ to end/, and the results from tmp/ to
	res/. the job is then considered "complete".


wagon-view
	
	a CGI program which gives an HTML listing of the contents of a work
	distribution server, with links to view the job data and any existing
	results, as well as delete jobs.


wagon-delete
	
	a CGI program which erases jobs from a work distribution server. don't
	give out access to this CGI unless you want people nuking your jobs.



SUPPORTING PROGRAMS
~~~~~~~~~~~~~~~~~~~

cdbget <key> < <job>

	reads the value of the key <key> from the job file <job>. used to
	extract particular keys from a job, when writing verification scripts or
	work proxies.

	example:

	kat-moda:~ $ wagon-build g94 water.com
	398a5a86-kat-moda-474-1
	kat-moda:~ $ cdbget type <398a5a86-kat-moda-474-1
	g94