Automation across the estate

Toft has one kind of automation: the job. A job is a command an app declares in its toft.toml, run as a process of its own with its own timeout and memory limit. Scheduled work is a job with a schedule; work an app starts for a person is a job too. Each start is a run.

Admin → Automation lists every schedule on the install, read-only, with its app, owner, next run and last outcome, and says where runs go. Jobs are defined only in toft.toml: neither you nor an app’s owner can create or edit one in the UI. An owner can run one now, pause its schedule until the next deploy, run a finished run again, and cancel.

Where a job runs

Jobs are on by default (background_jobs_enabled). The install decides where a run goes; an app’s own execution mode is not consulted.

Install	A run is	Memory limit
Isolated execution available	A pod in the cluster	Enforced by the cluster
No cluster	A process group on the platform’s machine, started by toft runner under the app’s OS account, with a scratch directory removed when it ends (not PM2)	Enforced by the kernel through a cgroup where the machine allows it; otherwise a sampled watchdog

Admin → Settings → Jobs shows both facts read-only: Where runs go (Jobs run on the cluster or Jobs run on this machine) and, on this machine, Memory limits (Enforced by the kernel or Sampled by a watchdog). Each run records the mode it ran in.

The kernel holds the limit when the platform can write to a cgroup v2 hierarchy: running as root in a container, as a systemd service with Delegate=yes, or under a parent you name in JOB_CGROUP_PARENT. The run is placed in its cgroup before it starts, so there is no moment it runs unlimited.

Without a cgroup the memory limit is a watchdog

The platform samples the run’s process tree every 250 ms and stops it past the limit, recording out_of_memory. Between samples a run can overshoot, and on a machine with no headroom the kernel may kill the platform instead. A run that outlives a platform restart continues unwatched until its deadline. For a guaranteed limit, give the platform a cgroup or use isolated execution.

JOB_RUN_DIR sets where a host run’s copy of the release is unpacked. Point it at local disk when APPS_DIR is on a network file system; see the environment reference.

Limits

You set the default and the ceilings; the developer declares what a job needs in toft.toml.

Setting	Controls	Range, default
default_job_timeout_seconds	Stop a job after — the timeout of a job that declares none	60–604,800 s, 3,600
max_job_timeout_seconds	Longest a job may run	60–604,800 s, 86,400
max_job_memory_mb	Most memory a job may have	128–262,144 MB, 8,192
job_concurrency_per_app	Concurrent runs per app — runs of one app executing at once; the rest wait	1–100
max_queued_runs_per_app	Runs waiting per app — a start past this is refused with queue_full until one finishes	1–10,000, 50

A new or changed declaration above a ceiling is refused at toft check and at deploy with limit_exceeds_ceiling, so a developer never believes they have more than they get. If you lower a ceiling below jobs already deployed, those jobs are capped rather than broken: each is recorded as limit_clamped, and the next check and deploy say so. Every surface shows a job’s effective limit with its source: declared, install default, app limit or clamped.

Batch apps

A batch app has only jobs — no web server. It is declared with a [batch] table in toft.toml, never inferred. Node, Python, Go and R only; [batch] beside [static] is refused.

Behavior	Batch app
Deploy	Installs dependencies, skips start and ready; the release switch is going live
Status	ready, never running — alerts written as “not running” will count batch apps
Its address	Lists each job’s schedule, next run and last outcome to anyone who can see the app
Restart / Stop	409 no_service; no Open, Start or Stop in the UI
Lock	Writes no stopped; cancels its runs

Schedules and time zones

• A schedule is UTC unless the job names a timezone beside schedule in toft.toml. A CRON_TZ= prefix inside a manifest schedule is refused.
• It fires once per wall-clock time across daylight saving changes. A time in the spring gap moves forward; a time repeated in autumn fires once.
• The zone database is compiled into the binary, so zones resolve the same on any image.
• A missed firing (the platform was down) is skipped, never fired late.
• overlap decides what a firing does while the last run is still going: queue (the default) waits, skip records a skipped run with its reason.
• A schedule does not keep an app awake.

A paused schedule stays paused until the app’s next deploy, which re-applies toft.toml.

The instant a run is for

A run can carry scheduled_for — the tick it belongs to, not when it was dispatched. The job receives it as TOFT_SCHEDULED_FOR, in the schedule’s zone; a rerun keeps its original’s. A manual run has none unless someone sets it with toft job run --for or Run for a past date in the Run now form.

How runs end

A run ends once, in one of six states, with its reason in words.

State	Means
succeeded	Exit 0.
failed	Another exit code, or the platform could not start it (stage platform).
timed_out	Stopped at its timeout.
out_of_memory	Stopped at its memory limit.
cancelled	Cancelled by a person, the app, or a lock.
skipped	A scheduled firing overlap = "skip" stood down.

Nothing is retried automatically. A failure is one run, shown once. Running it again is a deliberate act — Run again, toft job rerun, or the app starting it again — and the new run records which run it repeats.

Failed, skipped and overdue

Each is an app event, mailed to the app’s owners under the usual failure-notification rules, and resolved by that job’s own next success. A job’s notify key adds a message to its owners per run.

Event	Means	Raised by
job_run_failed	A run ended failed, timed_out or out_of_memory, whoever started it	The run
job_schedule_skipped	A scheduled firing was refused because the app’s queue was full	The scheduler
job_run_overdue	No success within the job’s expect_within	The jobs sweep
limit_clamped	A ceiling you lowered is below the job’s declaration	Dispatch
schedule_disabled	A schedule could not be read and was switched off	The scheduler

expect_within is a duration such as 26h or 7d, from 1 hour to 31 days.

• The window starts at the later of the last success and when the schedule was turned on.
• A paused job, or a locked app’s, is never overdue.
• One stuck job raises one event until it resolves.

Locked apps run no jobs

Whoever placed the lock (an admin or the dependency policy):

Run state	Effect
Scheduled or manual start	Refused
Queued before the lock	Cancelled at dispatch
Already running	Cancelled by the jobs sweep

What a job may reach

A job holds no app token. Its run token is accepted only on the send routes (/api/v1/app/notify/*, /deliver/*, /sends/*) and the store routes, so the SDK’s notifications and store calls work inside a job, through integrations and stores attached to its app.

Call	Result
Any other app route	403 job_send_only
After the run ends	401 run_not_in_flight

Jobs cannot start jobs.

Other rules

Rule	Detail
Who sees job history	Definitions, runs, logs and output need edit access to the app — the same audience as logs
Who starts a run	Owners and editors from Toft or the CLI; the app as itself; the app for a signed-in viewer, where the app decides who may
The status page	Served on the app’s own origin; a run started for a person is shown to that person and the app’s editors
Which deployment runs	The deployment the run was queued against; if retention removed it, the run is refused by name
Parameters	Typed, checked at the start, never interpolated into a command line
Claims	A run is claimed once; no double execution
Stranded runs	Reconciled; the Stuck work check finds the rest
App-started work	An app may start another entrypoint of itself, never another app, and reach only its own runs

Run and log retention is on Retention.

When automation stops

1. Admin → Checks: stuck_work and scheduled_jobs; the app’s events for overdue, skipped or clamped.
2. Admin → Automation: is the schedule there, not paused, and when did it last run?
3. Admin → Settings → Jobs: are jobs on, and is there somewhere to run them?
4. The app: is it locked?
5. The caps: is job_concurrency_per_app one, or the queue full?