Deployment

Jobber is written in Go, and it should be possible to compile and run it on any Unix/Linux system.

Installation from a Package

The easiest way to install Jobber is via a binary package.

Compilation and Installation from Source

Instead of installing a binary package, you can certainly compile and install Jobber yourself.

A Jobber installation consists of four files:

  • jobber: A command-line program
  • jobberrunner: A per-user daemon that runs jobs
  • jobbermaster: A daemon that runs as root and controls all the jobberrunner instances
  • jobber.conf: A YAML config file

To build, you first need a recent version of Go. Next:

git clone https://github.com/dshearer/jobber.git
cd jobber
git checkout v1.4.4
make

The results will be put in a directory named “bin”.

To install Jobber:

sudo make install

You can change where the four files are installed by first running ./configure (do ./configure -h for the syntax) and then:

make clean
make build
sudo make install

Finally, do something to ensure that jobbermaster runs as a daemon (as root) when your computer boots. Note that jobbermaster does not daemonize itself, but that should not be a problem if you use systemd; otherwise, daemonize(1) could help.

Putting Jobber to Work

As with cron, each user can have its own set of jobs, which will be run under that user’s privileges. A user’s jobs are defined in a jobfile — a file named “.jobber” in the user’s home directory.

IMPORTANT: Jobfiles must be owned by the user who owns the home directory that contains them, and their permissions must not allow the owning group or any other users to write to them (i.e., the permission bits must exclude 022).

Here's an example for a user named “bob” (/home/bob/.jobber). bob has three jobs:

  1. DailyBackup: Daily incremental backup of /home/bob/documents
  2. OffsiteBackup: Weekly offsite backup of /home/bob/documents
  3. GetAirQuality: Daily retrieval of local air quality

Results and errors from each of these jobs are handled a bit differently. DailyBackup really should not have errors; if an error nevertheless happens, the job is stopped permanently and an email is sent to bob.

OffsiteBackup, on the other hand, uses an online service, which may occasionally have trouble; in such cases, this job applies exponential backoff several times before stopping permanently. When it fails, an email is sent to bob and the stdout and stderr of the job are written to files in the directory /home/bob/backup-results.

Finally, GetAirQuality also uses an online service, but it is not as vital as OffsiteBackup, so errors are simply ignored. On success, however, details of the run (including stdout) to the shell script /home/bob/text-air-quality.sh, which (presumably) sends a text message about the day’s air quality.

version: 1.4

prefs:
  logPath: /home/bob/jobber/jobber.log
  runLog:
    type: file
    path: /home/bob/jobber/runlog
    maxFileLen: 100m
    maxHistories: 2

jobs:
  DailyBackup:
    cmd: /usr/local/bin/backup /home/bob/documents
    time: 0 0 13
    onError: Stop
    notifyOnFailure:
      - *systemEmailSink

  OffsiteBackup:
    cmd: |
      tar -C /home/bob -czf /home/bob/documents.tgz documents
      /usr/local/bin/backup-offsite /home/bob/documents.tgz
    time: 0 0 14 * * 1
    onError: Backoff
    notifyOnFailure:
      - *systemEmailSink
      - *filesystemSink

  GetAirQuality:
    cmd: /home/bob/get-air-quality.sh
    time: 0 0 8
    onError: Continue
    notifyOnSuccess:
      - *textAirQuality

resultSinks:
  - &systemEmailSink
    type: system-email

  - &filesystemSink
    type: filesystem
    path: /home/bob/backup-results
    data:
      - stdout
      - stderr
    maxAgeDays: 10

  - &textAirQuality
    type: program
    path: /home/bob/text-air-quality.sh

In general, each job must be given a script to run (cmd) and a string specifying when it should run it (time). Optionally, you can specify what happens to a job if the script returns an error (onError)

You can also specify what should be done with the results of a job. This is done using “result sinks”, which provide several handy ways of responding to the results of job runs. Result sinks are defined in the resultSinks section, and are referenced within job definitions in the notifyOnSuccess, notifyOnError, and notifyOnFailure items.

More details on the jobfile structure are given in the following sections.

Specifying Commands

Jobs’ commands are specified with YAML strings containing sh scripts. YAML’s syntax makes it easy to write multi-line scripts:

jobs:
  MyJob:
    cmd: |
      /home/bob/do-something.sh
      if [ $? -ne 0 ]; then
        /home/bob/handle-error.sh
      fi
    ...

These commands will be executed as the user that owns the jobfile, in a login session.

Time Strings

Field time specifies the schedule at which a job is run, in a manner similar to how cron jobs’ schedules are specified: with a space-separated list of up to six specifiers, each one of which constrains a different component of time (second, minute, hour, etc.). Schematically:

sec min hour month_day month week_day

A job is scheduled thus: the next run time is the time that satisfies the second, minute, hour, and month specifiers and at least one of the month-day and weekday specifiers.

Important: If a specifier is one of the random specifiers (beginning with R ), a random value is chosen when the jobfile is loaded, and stays the same until the jobfile is loaded again.

Each specifier can take one of the following forms: (“a”, “b”, “c”, and “n” are placeholders for arbitrary numerals.)

Specifier Form What It Matches
* Any value
a The value a
*/n Every a-th value — e.g., */25 in the second specifier would match 0, 25, and 50, whereas in the month specifier it would match 1 and 26.
a,b,c,... The values a, b, c, ... — e.g., 10, 20, 30, 40 matches 10, 20, 30, and 40.
a-b Any of the values between and including a and b
R A random value
Ra-b A random value between and including a and b

The specifiers have different permitted values for the placeholders in the specifier forms:

Specifier Values for “a”, “b”, and “c” Values for “n”
sec 0 thru 59 1 thru 59
min 0 thru 59 1 thru 59
hour 0 thru 23 1 thru 23
month_day 1 thru 31 1 thru 30
month 1 thru 12 1 thru 11
week_day 0 (Sunday) thru 6 (Saturday) 1 thru 5

The default value for time is * * * * * * .

Error-handling

When you run jobber list to list the jobs for a user, you get an output like this:

NAME         STATUS  SEC/MIN/HR/MDAY/MTH/WDAY  NEXT RUN TIME              NOTIFY ON SUCCESS  NOTIFY ON ERR  NOTIFY ON FAIL  ERR HANDLER
DailyBackup  Good    0 0 13 * * *              Nov 22 13:00:00 +0000 UTC                                    system-email    Stop

In this example, the job’s status is Good, which means that runs of the job will be scheduled normally. A status of Failed means that runs will no longer be scheduled. A status of Backoff means that runs will be scheduled in the following several seconds according to an exponential backoff algorithm.

A job’s status is affected by job errors — that is, runs of the job’s shell script that resulted in non-0 exit codes. This is done according to the job’s error-handler, as specified in the jobfile under the onError key in the job’s definition.

Value Effect when a job error happens
Stop Stop scheduling runs of this job. (The status is set to Failed).
Backoff Schedule runs of this job according to an exponential backoff algorithm (setting the status to Backoff). If a later run of the job succeeds, jobber resumes scheduling this job normally; but if the job continues to err, Jobber will eventually stop scheduling it (setting the status to Failed).
Continue (Default) Continue scheduling this job normally. (The status is always Good.)

Result Sinks
Doing Stuff with Job Run Output

A run of a job results in one of three events:

Job success The job (or, specifically, the shell script that the job runs) exited with status 0
Job error The job (or, specifically, the shell script that the job runs) exited with a non-0 status
Job failure Jobber stopped scheduling runs of the job due to one or more job errors

For a partiular job, you can assign result sinks that can do various things with the job’s output in response to these events, using the notifyOnSuccess, notifyOnError, and notifyOnFailure fields in the job’s definition:

jobs:
  GetAirQuality:
    cmd: /home/bob/get-air-quality.sh
    time: 0 0 8
    onError: Backoff
    notifyOnSuccess:
      - type: program
        path: /home/bob/text-air-quality.sh
    notifyOnFailure:
      - type: system-email
      - type: filesystem
        path: /home/bob/jobber-failures
        data:
          - stdout
          - stderr
        maxAgeDays: 10

In this example, result sinks are used only for successes and failures. The notifyOnSuccess, notifyOnError, and notifyOnFailure fields take lists of result sink definitions, and each result sink definition has a required type field as well as other fields depending on the type.

It is common to want to use the same result sink in different jobs (or even for different events from the same job). This is best done by moving the sink’s definition to the resultSinks section of the jobfile and then using YAML anchors in the job definitions to reference the sink’s definition:

jobs:
  GetAirQuality:
    cmd: /home/bob/get-air-quality.sh
    time: 0 0 8
    onError: Backoff
    notifyOnSuccess:
      - type: program
        path: /home/bob/text-air-quality.sh
    notifyOnFailure:
      - type: system-email
      - *filesytemFailureSink

  DailyBackup:
    cmd: /usr/local/bin/backup /home/bob/documents
    time: 0 0 13
    onError: Stop
    notifyOnFailure:
      - *filesytemFailureSink

resultSinks:
  - &filesytemFailureSink
    type: filesystem
    path: /home/bob/jobber-failures
    data:
      - stdout
      - stderr
    maxAgeDays: 10

Here are the types of result sinks:

TypeBehaviorParams
system-email Sends an email using sendmail to the job’s owner containing details of the run (e.g., stdout output, stderr output, exit code). None
stdout Writes a run record (see below) to jobberrunner’s stdout. This is useful mostly when running Jobber in a Docker container.
data A list of strings each of which must be “stdout” or “stderr”. This specifies whether the run record written to jobberrunner’s stdout will contain the run’s stdout output, stderr output, or both.
socket Writes a run record (see below) to incoming connections to a TCP or Unix socket. This result sink causes jobberrunner to listen for connections on the specified socket.

NOTE: The owning user must have permission to listen to the specified socket or make a Unix socket at the specified path.
proto A string: “tcp”, “tcp4”, “tcp6”, or “unix”
address A string: either a port specification like “:1234” (specifying on which port to listen) or a filesystem path (specifying where to make the Unix socket).
data A list of strings each of which must be “stdout” or “stderr”. This specifies whether the run record written to the socket will contain the run’s stdout output, stderr output, or both.
program Runs a program and writes a run record (see below) to the program’s stdin.
path A filesystem path to the program.
runRecFormatVersion
filesystem Writes the run&squo;s stdout output, stderr output, or both to the filesystem. A path to a directory must be given. The stdout output and stderr output will be written to separate files with timestamps as names, and these files will be organized by job name. For example:
- /some/dir/
  - JobOne/
    - 1521318351.stdout
    - 1521318351.stderr
    - 1521318411.stdout
    - 1521318411.stderr
    - 1521318471.stdout
    - 1521318471.stderr
path A path to a directory.
data A list of strings each of which must be “stdout” or “stderr”. This specifies whether the stdout outout, stderr output, or both will be written.
maxAgeDays How many days to keep the output files.

Several types of result sinks publish details about a job’s run in the form of a run record, which is a JSON document. Here is an example:

{
  "version": "1.4",
  "job": {
    "name": "DailyBackup",
    "command": "/usr/local/bin/backup /home/bob/documents",
    "time": "0 0 13 * * *",
    "status": "Good"
  },
  "user": "bob",
  "startTime": 1543150800,
  "succeeded": true,
  "stdout": "Backing up...\nSuccess",
  "stderr": ""
}

If the run’s stdout (or stderr) output contained non-UTF-8 bytes, then the run record will have a stdoutBase64 (or stderrBase64) field instead of a stdout (or stderr) field, containing the output encoded in Base64.

Keeping a Log of Job Runs

The jobber log command prints the run log — i.e., a log containing info about previous runs. By default, the run log only contains the 100 most recent runs, and it is not written to disk, which means that it will be lost when the Jobber service stops or restarts.

Here is an example of keeping the run log in memory:

prefs:
  runLog:
    type: memory
    maxLen: 500

With this, Jobber will keep log entries for no more than the 500 most recent runs of any of the user’s jobs.

Here is an example of keeping the run log on disk:

prefs:
  runLog:
    type: file
    path: /var/log/jobber-runs
    maxFileLen: 100m
    maxHistories: 2

With this, Jobber will write run log entries to /var/log/jobber-runs. /var/log/jobber-runs will grow to no more than 100 MB, and Jobber will rotate it when it gets too big. Jobber will keep up to two of these rotated “history” files (named “jobber-runs.1” and “jobber-runs.2”). When it is necessary to delete a history file, the oldest one will get the axe.

Loading Jobs

After you've created a user’s jobfile, log in as that user and do:

jobber reload

You can also reload all users’ jobfiles by logging in as root and doing:

jobber reload -a

Listing Jobs

You can list the jobs for a particular user by logging in as that user and doing

jobber list

This command also shows you the status of each job — that is, whether the job is being scheduled as normal, the exponential backoff algorithm is being applied, or the job has failed.

As with the reload command, you can do the same for all users by adding the -a option as root.

Listing Runs

You can see a list of recent runs of any jobs for a particular user by logging in as that user and doing

jobber log

As with the other commands, you can do the same for all users by adding the -a option as root.

Testing Jobs

If you'd like to test out a job, do

jobber test JOB_NAME

Jobber will immediately run that job, tell you whether it succeeded, and show you its output.

Pausing and Resuming Jobs

You can temporarily stop Jobber from running a job thus:

jobber pause JOB_NAME

And of course you can resume it:

jobber resume JOB_NAME

NOTE: If a job is paused when the Jobber service stops, then when the Jobber service starts again the job will be scheduled normally.

Printing a Job’s Command

You can print a job’s command (the shell script that it executes) thus:

jobber cat JOB_NAME