# Deployments and rollbacks

## Statuses

| Status | Meaning |
| --- | --- |
| Queued | Waiting for a server to pick it up. |
| Building | A server is building the image from your code. |
| Built | The image is ready; the new version is starting (shown as **Starting** or **Switching**). |
| Live | It runs and takes the requests. |
| Failed | The build failed, or the new version never became ready: the log and the error say why. |
| Superseded | A newer deployment replaced it before it went live, or after. |

Click a deployment in the **Deployments** tab for its log: the build, then each step of the switch.

## Deploys without downtime

A new version of a public service doesn't replace the running one at once:

1. The new version starts **next to** the running one. Visitors still reach the running one.
2. Alf waits until the new version is ready: its port answers, and its **health check path** if you set one.
3. New requests go to the new version. Requests still open on the old one finish (up to 60 seconds, WebSockets included).
4. The old version stops and is removed.

If the new version isn't ready within 3 minutes (or exits), it's removed, the deployment **fails**, and the old version keeps running as if nothing happened. Nothing is retried by itself: fix the cause and deploy again.

```diagram deploy
1. The new version is built or pulled while the old one keeps serving; if the build fails, the deployment fails and nothing else changes. 2. Its release command runs, if it has one: once, in a container of the new version, in the project's network; if it fails or runs over 30 minutes, the deployment fails and the new version never starts. 3. The new version starts next to the old one, and visitors still reach the old one. 4. It's ready when its port answers, and its health check path if it has one; if not within 3 minutes, or it exits, it's removed, the deployment fails and the old one carries on. 5. New requests go to the new version, which is live; the old one finishes its open requests (up to 60 seconds, WebSockets too) and is removed.
```

A few details:

- **Health check path** (service settings, e.g. `/health`): a path that answers with a 2xx or 3xx status once the app can take requests. Set one if your app opens its port before it's ready (migrations, warm-up); without one, the open port counts. It's waited for every time the app starts, not only on a deploy: when a request wakes it from sleep, and when another replica starts under load, nothing is sent to it until the path answers.
- **Replicas**: when the running version has several containers under load, the new version starts as many before the switch, so capacity doesn't drop.
- **Several servers**: one server goes first. The others keep the old version until the new one is live there, so a version that doesn't start never reaches them.
- **Sleeping services**: the new version is started once to check it, then takes over.
- **Settings**: changing the port, command, memory or CPU limit, or public/private is a deployment too ("settings changed") and rolls out the same way.

## Release commands

A service's **release command** runs once per deploy, before the new version takes over: what database migrations are for (`npx prisma migrate deploy`, `python manage.py migrate`, `bin/rails db:migrate`, `php artisan migrate --force`). Set it in the service's settings (**Command, variables and deploys → Release command**) or as `release_command` in [alf.toml](/docs/alf-toml); [`alf launch`](/docs/launch) sets it when it recognises the code's migrations.

1. The new version's image is built (or pulled), while the old version keeps serving.
2. The command runs in a container of the **new** version, with its variables and secrets and in the project's network, so it reaches the database. It runs like `alf exec`: through the image's entrypoint.
3. When it ends without an error, the new version starts and takes over as usual. Its output is in the deployment's log.
4. When it fails (or runs longer than 30 minutes), the deployment **fails** with what it printed, the new version never starts, and the old one keeps serving. Nothing is tried again until the next deploy.

Good to know:

- It runs once per deploy, on the server the deploy goes to first, not once per server or per container. A rollback runs it again (migrations that are applied already are skipped by every migration tool).
- It runs without the service's volumes: the running version still has them. An app keeping SQLite on a volume migrates in its start command instead.
- Migrations should keep working with the version that's still serving while they run, as with any deploy without downtime: add before you remove.
- A service with a release command can't have [candidates](#candidates-try-a-version-before-it-s-live): a candidate would migrate the live database before anyone decided to promote it. Try such a change in a [preview environment](/docs/previews), which has its own database.
- Unlike a start command that migrates, it doesn't run when a sleeping app wakes, so waking stays fast.

## Candidates: try a version before it's live

A candidate is the new version started **beside** the live one and held there:

- in the same project network, so it reaches the same database and services;
- with the same variables and secrets, or with the staged ones if changes are waiting (they go live when it's promoted);
- at its own address, the service's address with `--next` (`web-shop-acme--next.smartalfred.com`).

Visitors and the project's other services keep reaching the live version. When the candidate is healthy it shows as **Ready to promote**:

- **Open** it and check.
- **Promote** makes it the live version: the usual switch, without downtime.
- **Discard** removes it. The live version was never touched.

Start one from the arrow next to **Deploy**, from **Deploy new code** (tick *Hold it as a candidate*), with `--candidate` on the [deploy command](/docs/deploying-code), or have an agent do it (`deploy`, `upload_code` or `push_code` with `candidate`).

Good to know:

- A candidate talks to the **real** database. If the new version changes the schema on start, that happens for real. For that kind of change use a [preview environment](/docs/previews), which is a separate copy of the project.
- One candidate per service; a new one replaces the one waiting. Left alone, it's removed after a day.
- Only services that can run twice at once can have one: public services without a volume.
- Previews and candidates are different things: a preview copies the whole project, with its own network, addresses, empty volumes, databases and buckets; a candidate is one service's next version inside the real project.

## When a deploy fails

A failed deployment says what kind of failure it was and shows what the app last printed:

| It says | Meaning | Usual fix |
| --- | --- | --- |
| **The app stopped** | The process exited before it was ready (with its exit code; "out of memory" if it used more than its limit). | Read its last output: a missing variable or secret, a database it can't reach, a wrong start command. When the output says a variable is missing, the failure names it and the `alf env set` that sets it. |
| **Nothing on its port** | It runs, but nothing accepts connections on the service's port. If it listens somewhere else, that port is named. | Set the service's **Port** to the one named, or make the app listen on `$PORT` and on `0.0.0.0`. |
| **Health check** | The port is open, but the health check path answered with an error, or not at all. | Fix the path, or change or clear **Health check path** in Settings. |
| **Release command** | The [release command](#release-commands) failed, so the new version never started; the old one still serves. | Read its output: often a database it can't reach yet, or a migration that doesn't apply. |
| **Couldn't start** / **Build** | The container or image couldn't be made. | The message and the build log say why; for the usual build failures (a file not sent, a lockfile out of step, a missing script, a runtime too old, a missing base image) the failure says what fixes it. |

A service that has never been live and fails its first deploy because its app listens on another port gets that port and deploys again by itself, once: the failed deployment says "Fixed by itself" with the port, and the new one follows it. It's only done when the app listens on exactly one other port. A port changed that way is the service's from then on, as if you had set it: if alf.toml says another, the next `alf deploy` asks which to keep and can write the new one into the file (without a terminal, as in CI, the new one stays).

The same goes for a service's very first start: a new service is started once, right after it's created, to see that it comes up. A service that sleeps when idle goes to sleep again afterwards.

### When the old version stops first

Some services' old container stops before the new one starts (a gap of a few seconds):

- services with a [volume](/docs/volumes): two containers on one data directory would corrupt it;
- [databases](/docs/databases): they have one place in the project's network;
- private services, unless you choose otherwise: two workers at once might do the same job twice.

A private app is called through its server's front, so it can switch without downtime like a public one: the new version starts beside the old one and takes the calls once it's ready. Choose **Without downtime** under **Settings → When a new version is deployed** (or `zero_downtime = true` in alf.toml) for an API the project calls; leave it for a worker.

You can choose to stop first for any service: **Settings → When a new version is deployed → Stop the old version first**.

## Deploying again

**Deploy** at the top of the service's page deploys now: image services pull their image again, services built from code rebuild the code pushed last. New code arrives with the [deploy command](/docs/deploying-code), from your computer or CI. [Variable and secret changes](/docs/variables-and-secrets) waiting to be deployed go along.

## Rolling back

Earlier deployments that were live have **Roll back**: it deploys the same image or commit again, with today's variables and secrets. Pushed code can be rolled back while it's kept (the newest three pushes of a service).

## Working inside a container

**Service → Terminal** opens a shell in a container, for a migration, a database console or a look
around. Two kinds:

- **A container of the service's image**, with its variables, secrets, volumes and project
  network. Anything you break here is thrown away when you close it.
- **Its own running container** (tick *Its own running container*): what's actually serving
  traffic. It's held awake while the terminal is open, and what you change is live.

Either way the container is metered like any other container of the service while it's open, and
it closes itself after fifteen minutes without a command.

Your coding agent has the same thing without the terminal: `exec` runs a command and gives back
the exit code and output, and `open_session` keeps a container between commands. Both need a token
that may run commands. Everything run in a session is written down.

## Running something on a schedule

**Service → Scheduled** runs a command over and over: a nightly clean-up, an hourly digest, a
report on the first of the month. Each run gets the service's image, variables, secrets, volumes
and project network, in a container thrown away afterwards — the same as a one-off command, with
a time attached.

The schedule is written the way cron has always been written, in **UTC**:

```
0 3 * * *     every day at 03:00
*/15 * * * *  every fifteen minutes
0 9 * * 1     every Monday at 09:00
@daily        the same as 0 0 * * *
```

One thing that catches people out, and that cron has always done: if you give **both** a day of
the month and a day of the week, it runs on **either**. `0 0 1 * 1` is the first of the month *and*
every Monday, not the first when it falls on a Monday.

**Run now** runs it without waiting, which is how to see whether it works. A failed run is kept
with its output, and you're told through your notification channels. A run still going when the
next is due is skipped rather than started twice, and a service that can't be deployed to — an
unpaid invoice, a spending limit — is passed over until that's sorted out.