Deployment
Jobber is written in Go, and it should be possible to compile and run it on any Unix/Linux system.
Installation from a Package
The easiest way to install Jobber is via a binary package.
Compilation and Installation from Source
Instead of installing a binary package, you can certainly compile and install Jobber yourself.
A Jobber installation consists of four files:
jobber
: A command-line programjobberrunner
: A per-user daemon that runs jobsjobbermaster
: A daemon that runs as root and controls all thejobberrunner
instancesjobber.conf
: A YAML config file
To build, you first need a recent version of Go. Next:
git clone https://github.com/dshearer/jobber.git cd jobber git checkout v1.4.4 make
The results will be put in a directory named “bin”.
To install Jobber:
sudo make install
You can change where the four files are installed by first running
./configure
(do ./configure -h
for the syntax)
and then:
make clean make build sudo make install
Finally, do something to ensure that jobbermaster runs as a daemon (as root) when your computer boots. Note that jobbermaster does not daemonize itself, but that should not be a problem if you use systemd; otherwise, daemonize(1) could help.
Putting Jobber to Work
As with cron, each user can have its own set of jobs, which will be run under that user’s privileges. A user’s jobs are defined in a jobfile — a file named “.jobber” in the user’s home directory.
IMPORTANT: Jobfiles must be owned by the user who owns the home directory that contains them, and their permissions must not allow the owning group or any other users to write to them (i.e., the permission bits must exclude 022).
Here's an example for a user named “bob” (/home/bob/.jobber
).
bob
has three jobs:
DailyBackup
: Daily incremental backup of /home/bob/documentsOffsiteBackup
: Weekly offsite backup of /home/bob/documentsGetAirQuality
: Daily retrieval of local air quality
Results and errors from each of these jobs are handled a bit differently.
DailyBackup
really should not have errors; if an error nevertheless
happens, the job is stopped permanently and an email is sent to bob
.
OffsiteBackup
, on the other hand, uses an online service, which
may occasionally have trouble; in such cases, this job applies exponential backoff
several times before stopping permanently. When it fails, an email is sent to bob
and the stdout and stderr of the job are written to files in the directory
/home/bob/backup-results
.
Finally, GetAirQuality
also uses an online service, but it is not
as vital as OffsiteBackup
, so errors are simply ignored. On success,
however, details of the run (including stdout) to the shell script
/home/bob/text-air-quality.sh
, which (presumably) sends a text
message about the day’s air quality.
version: 1.4 prefs: logPath: /home/bob/jobber/jobber.log runLog: type: file path: /home/bob/jobber/runlog maxFileLen: 100m maxHistories: 2 jobs: DailyBackup: cmd: /usr/local/bin/backup /home/bob/documents time: 0 0 13 onError: Stop notifyOnFailure: - *systemEmailSink OffsiteBackup: cmd: | tar -C /home/bob -czf /home/bob/documents.tgz documents /usr/local/bin/backup-offsite /home/bob/documents.tgz time: 0 0 14 * * 1 onError: Backoff notifyOnFailure: - *systemEmailSink - *filesystemSink GetAirQuality: cmd: /home/bob/get-air-quality.sh time: 0 0 8 onError: Continue notifyOnSuccess: - *textAirQuality resultSinks: - &systemEmailSink type: system-email - &filesystemSink type: filesystem path: /home/bob/backup-results data: - stdout - stderr maxAgeDays: 10 - &textAirQuality type: program path: /home/bob/text-air-quality.sh
In general, each job must be given a script to run (cmd
) and a
string specifying when it should run it (time
). Optionally,
you can specify what happens to a job if the script returns an error
(onError
)
You can also specify what should be done with the results of a job.
This is done using “result sinks”,
which provide several handy ways of responding to the results of job runs.
Result sinks are defined in the resultSinks
section, and are
referenced within job definitions in the notifyOnSuccess
,
notifyOnError
, and notifyOnFailure
items.
More details on the jobfile structure are given in the following sections.
Specifying Commands
Jobs’ commands are specified with YAML strings containing sh
scripts. YAML’s syntax makes it easy to write multi-line scripts:
jobs: MyJob: cmd: | /home/bob/do-something.sh if [ $? -ne 0 ]; then /home/bob/handle-error.sh fi ...
These commands will be executed as the user that owns the jobfile, in a login session.
Time Strings
Field
time
specifies the schedule at which a job is run, in a manner similar to how
cron jobs’ schedules are specified: with a space-separated list of up
to six specifiers, each one of which constrains a different component of
time (second, minute, hour, etc.). Schematically:
sec min hour month_day month week_day
A job is scheduled thus: the next run time is the time that satisfies the second, minute, hour, and month specifiers and at least one of the month-day and weekday specifiers.
Important: If a specifier is one of the random specifiers
(beginning with
R
), a random value is chosen when the jobfile is loaded, and stays the same
until the jobfile is loaded again.
Each specifier can take one of the following forms: (“a”, “b”, “c”, and “n” are placeholders for arbitrary numerals.)
Specifier Form | What It Matches |
---|---|
* |
Any value |
a
|
The value a |
*/n |
Every a-th value — e.g., */25 in
the second specifier would match 0, 25, and 50, whereas in the month
specifier it would match 1 and 26.
|
a,b,c,... |
The values a, b, c, ...
— e.g., 10, 20, 30, 40 matches 10, 20, 30, and 40.
|
a-b |
Any of the values between and including a and b |
R
| A random value |
Ra-b
| A random value between and including a and b |
The specifiers have different permitted values for the placeholders in the specifier forms:
Specifier | Values for “a”, “b”, and “c” | Values for “n” |
---|---|---|
sec | 0 thru 59 |
1 thru 59 |
min | 0 thru 59 |
1 thru 59 |
hour | 0 thru 23 |
1 thru 23 |
month_day | 1 thru 31 |
1 thru 30 |
month | 1 thru 12 |
1 thru 11 |
week_day | 0 (Sunday) thru 6 (Saturday) |
1 thru 5 |
The default value for
time
is
* * * * * *
.
Error-handling
When you run jobber list
to list the jobs for a user, you get
an output like this:
NAME STATUS SEC/MIN/HR/MDAY/MTH/WDAY NEXT RUN TIME NOTIFY ON SUCCESS NOTIFY ON ERR NOTIFY ON FAIL ERR HANDLER DailyBackup Good 0 0 13 * * * Nov 22 13:00:00 +0000 UTC system-email Stop
In this example, the job’s status is Good
, which means that
runs of the job will be scheduled normally. A status of Failed
means that runs will no longer be scheduled. A status of
Backoff
means that runs will be scheduled in the following several
seconds according to an exponential backoff algorithm.
A job’s status is affected by job errors —
that is, runs of the job’s shell script that resulted in non-0 exit codes.
This is done according to the job’s error-handler, as specified
in the jobfile under the onError
key in the job’s definition.
Value | Effect when a job error happens |
---|---|
Stop |
Stop scheduling runs of this job. (The status is set to Failed ). |
Backoff |
Schedule runs of this job according to an exponential backoff
algorithm (setting the status to
Backoff ). If a later run of the job succeeds, jobber resumes
scheduling this job normally; but if the job continues to err,
Jobber will eventually stop scheduling it (setting the status to
Failed ). |
Continue |
(Default) Continue scheduling this job normally. (The status is always Good .) |
Result Sinks
Doing Stuff with Job Run Output
A run of a job results in one of three events:
Job success | The job (or, specifically, the shell script that the job runs) exited with status 0 |
Job error | The job (or, specifically, the shell script that the job runs) exited with a non-0 status |
Job failure | Jobber stopped scheduling runs of the job due to one or more job errors |
For a partiular job, you can assign result sinks
that can do various things with the job’s output in response to these
events, using the notifyOnSuccess
, notifyOnError
,
and notifyOnFailure
fields in the job’s definition:
jobs: GetAirQuality: cmd: /home/bob/get-air-quality.sh time: 0 0 8 onError: Backoff notifyOnSuccess: - type: program path: /home/bob/text-air-quality.sh notifyOnFailure: - type: system-email - type: filesystem path: /home/bob/jobber-failures data: - stdout - stderr maxAgeDays: 10
In this example, result sinks are used only for successes and failures.
The notifyOnSuccess
, notifyOnError
, and
notifyOnFailure
fields take lists of result sink definitions,
and each result sink definition has a required type
field as
well as other fields depending on the type.
It is common to want to use the same result sink in different jobs (or even
for different events from the same job). This is best done by moving the
sink’s definition to the resultSinks
section of the jobfile
and then using YAML anchors in the job definitions to reference the sink’s
definition:
jobs: GetAirQuality: cmd: /home/bob/get-air-quality.sh time: 0 0 8 onError: Backoff notifyOnSuccess: - type: program path: /home/bob/text-air-quality.sh notifyOnFailure: - type: system-email - *filesytemFailureSink DailyBackup: cmd: /usr/local/bin/backup /home/bob/documents time: 0 0 13 onError: Stop notifyOnFailure: - *filesytemFailureSink resultSinks: - &filesytemFailureSink type: filesystem path: /home/bob/jobber-failures data: - stdout - stderr maxAgeDays: 10
Here are the types of result sinks:
Type | Behavior | Params | ||||||
---|---|---|---|---|---|---|---|---|
system-email | Sends an email using sendmail to the job’s owner
containing details of the run (e.g., stdout output, stderr output, exit code). |
None | ||||||
stdout | Writes a run record (see below) to jobberrunner’s stdout. This is useful mostly when running Jobber in a Docker container. |
|
||||||
socket | Writes a run record (see below) to incoming connections to a TCP or
Unix socket. This result sink causes
jobberrunner to listen for connections on the specified socket.
NOTE: The owning user must have permission to listen to the specified socket or make a Unix socket at the specified path. |
|
||||||
program | Runs a program and writes a run record (see below) to the program’s stdin. |
|
||||||
filesystem | Writes the run&squo;s stdout output, stderr output, or both to the filesystem.
A path to a directory must be given. The stdout output and stderr output will be written
to separate files with timestamps as names, and these files will be
organized by job name. For example:
- /some/dir/ - JobOne/ - 1521318351.stdout - 1521318351.stderr - 1521318411.stdout - 1521318411.stderr - 1521318471.stdout - 1521318471.stderr |
|
Several types of result sinks publish details about a job’s run in the form of a run record, which is a JSON document. Here is an example:
{ "version": "1.4", "job": { "name": "DailyBackup", "command": "/usr/local/bin/backup /home/bob/documents", "time": "0 0 13 * * *", "status": "Good" }, "user": "bob", "startTime": 1543150800, "succeeded": true, "stdout": "Backing up...\nSuccess", "stderr": "" }
If the run’s stdout (or stderr) output contained non-UTF-8 bytes, then
the run record will have a stdoutBase64
(or stderrBase64
) field
instead of a stdout
(or stderr
) field, containing
the output encoded in Base64.
Keeping a Log of Job Runs
The jobber log
command prints the run log — i.e., a log
containing info about previous runs. By default, the run log only contains the 100 most recent
runs, and it is not written to disk, which means that it will be lost when the Jobber service stops
or restarts.
Here is an example of keeping the run log in memory:
prefs: runLog: type: memory maxLen: 500
With this, Jobber will keep log entries for no more than the 500 most recent runs of any of the user’s jobs.
Here is an example of keeping the run log on disk:
prefs: runLog: type: file path: /var/log/jobber-runs maxFileLen: 100m maxHistories: 2
With this, Jobber will write run log entries to /var/log/jobber-runs. /var/log/jobber-runs will grow to no more than 100 MB, and Jobber will rotate it when it gets too big. Jobber will keep up to two of these rotated “history” files (named “jobber-runs.1” and “jobber-runs.2”). When it is necessary to delete a history file, the oldest one will get the axe.
Loading Jobs
After you've created a user’s jobfile, log in as that user and do:
jobber reload
You can also reload all users’ jobfiles by logging in as root and doing:
jobber reload -a
Listing Jobs
You can list the jobs for a particular user by logging in as that user and doing
jobber list
This command also shows you the status of each job — that is, whether the job is being scheduled as normal, the exponential backoff algorithm is being applied, or the job has failed.
As with the
reload
command, you can do the same for all users by adding the
-a
option as root.
Listing Runs
You can see a list of recent runs of any jobs for a particular user by logging in as that user and doing
jobber log
As with the other commands, you can do the same for all users by adding the
-a
option as root.
Testing Jobs
If you'd like to test out a job, do
jobber test JOB_NAME
Jobber will immediately run that job, tell you whether it succeeded, and show you its output.
Pausing and Resuming Jobs
You can temporarily stop Jobber from running a job thus:
jobber pause JOB_NAME
And of course you can resume it:
jobber resume JOB_NAME
NOTE: If a job is paused when the Jobber service stops, then when the Jobber service starts again the job will be scheduled normally.
Printing a Job’s Command
You can print a job’s command (the shell script that it executes) thus:
jobber cat JOB_NAME