# Metrics and logs

## Metrics

A service's **Overview** charts, per minute and for the last hour, 6 hours, day or week:

- Requests (answered, client errors, server errors) and requests per second: what came in at its address and custom domains. Calls from the project's own services are counted too, and shown apart (**From inside the project**); Alf's own health checks aren't, though the app may log them.
- Response time (median and 95th percentile)
- CPU and memory: what the app holds against its size. Each container runs in a small VM of its own; what the VM itself takes (its kernel and caches) isn't the app's and isn't counted.
- Network in and out
- Time awake, wake-ups and wake time
- Replicas

Nobody watches a chart at three in the morning: [alerts](/docs/alerts) say when these numbers go bad. The top of the Overview says what watches the service.

Pick a server to see one copy only, or **Table** for the numbers. Metrics are kept for 7 days.

## Logs

The **Logs** tab shows the container's recent output (standard output and error) from one of its servers. **Follow** keeps it fresh. A sleeping service has no running container: its last output is still shown until the container is replaced.

Under load a service runs [more containers](/docs/scaling#under-load-more-containers), and each writes its own log. The **Container** menu at the top picks which one to follow; **History** keeps what all of them wrote, each line marked with the container it came from.

## Errors

The **Errors** view of the Logs tab reads the kept log (below) for what went wrong: each error once, with its stack trace, how many times it happened and when it was first and last seen, the most recent first. The same error is counted as one however its numbers, ids and values change, so "user 81 not found" and "user 7 not found" are one line with a count. A line starts an error when it says so (`Error`, `TypeError`, `panic`, `Traceback`, `FATAL`, `level=error` and the like), and the indented lines that follow it straight after are its trace.

```bash
alf errors web              # the last 24 hours
alf errors web --since 2h --trace 30
```

Your coding agent has the same as `search_logs` with `errors: true`.

## Requests

`alf requests web -f` shows each request a service answers as it comes: when, the status, the method, how long it took, the path (without its query string, which may hold tokens) and the size of the answer. `--status 5xx` (or `4xx`, `404`, `error`) and `--path /api` narrow it down. `--with-logs` puts under each 5xx what the app printed while it answered it, from the kept log. Its servers keep the newest few hundred in memory, so it's for what's happening now; nothing about who asked is kept, except which of the project's own services it was for a call from inside (`← worker`). `--from inside` (or `internet`, or a service's name) narrows to those. A gRPC call shows its status (`grpc 5`), and one that failed counts as the error it is, so `--status error` finds it. What came from inside and wasn't HTTP (a database's protocol, TLS, a port of the app's own) has no requests to show; `alf requests cache --connections` shows those connections instead: when each ended, which service made it, the port, how long it was open, the bytes each way, and whether it closed, was reset, was refused (nothing listened on the port) or couldn't wake the service. `alf top` has every service's numbers for the last few minutes side by side.

## Build logs

Each deployment in the **Deployments** tab has its build log, streamed while it builds.

## Looking back

**Live** is the container's own output as it happens. **History** is what the servers have shipped
over the last **three days** — which outlives the container that wrote it, so a crash this morning
or what a replaced version said is still there.

Search it for the text of a line, narrow it to a stretch of time, or to only what went to stderr.
Your coding agent has the same thing as `search_logs`, and `log_span` tells it how far back the
log goes before it asks for a time that isn't kept any more.

Lines arrive with each server's report, every fifteen seconds or so, so the newest second or two
is in **Live** before it is in **History**. A service that writes an enormous amount has its lines
capped per report, and the gap says so in the log itself rather than disappearing quietly.