Containers & boot

If every other layer failed and something hostile ran inside a Steelmoth container, the container itself is built so there is very little it could do.

The pages so far govern agents writing code. This page is about the running services — the API, the workers, the daemons — and two questions: how locked-down is the box each one runs in, and what guarantees they come up correctly after a reboot.

How each container is hardened

The production compose file (compose.steelmoth.private.yml) applies the same set of restrictions to every service:

Setting	What it prevents
read_only: true	The container’s filesystem cannot be written at all. Malicious or buggy code can’t drop files, edit binaries, or persist anything.
tmpfs /tmp — noexec,nosuid, 64 MB	The one writable scratch space is memory-only, capped in size, and nothing placed there can be executed. The classic “download a payload to /tmp and run it” move fails.
no-new-privileges	No process inside can ever gain more privileges than it started with — setuid binaries and similar escalation tricks are dead ends.
cap_drop: ALL	Every special kernel capability is removed. The process can’t change networking, mount filesystems, trace other processes, or any of the other privileged operations.
Docker secrets (/run/secrets, root-owned, mode 600, tmpfs)	The vault token reaches the container as a memory-backed file readable only at startup by root — see secret handling.
Entrypoint drops to the steelmoth user via setpriv	Root exists only for the instant needed to read the secret; the actual service runs as an unprivileged user that can’t read /run/secrets anymore.
Split internal / egress networks	Services that don’t need the internet aren’t on the network that can reach it. The database and internal services are unreachable from outside their own network.

The comment that guards the comment

The compose file carries a written warning at the broker-related blocks citing HARD RULE 10, ADR 0105 and INV-024 — so an agent editing the file is told, at the exact spot, that removing those lines is a rule violation. And because written warnings aren’t enough on their own, the canonical-containers guard separately blocks any new service from appearing in compose files without an explicit allowlist entry or a [allow-new-service] commit token.

Boot: the stack comes up in order, or not at all

A reboot is the moment most ad-hoc setups quietly break. Steelmoth boots through one systemd unit, ops/systemd/steelmoth-private.service, which runs scripts/steelmoth-private-boot.sh:

  systemd starts steelmoth-private.service   (timeout: 8 minutes)
        │
        ├─ 1. wait until Docker itself is ready
        ├─ 2. remove stale containers whose vault-token mount
        │     no longer exists (see secret handling)
        ├─ 3. start postgres, wait until it answers
        ├─ 4. run database migrations
        └─ 5. start the services, each with a fresh ephemeral token

The ordering matters: migrations before services means no service ever runs against a database schema it doesn’t expect, and the stale-container cleanup in step 2 is what makes reboots survive the deliberately non-restartable token design.

The hourly self-documentation timer

A second pair of units — steelmoth-system-wiki-update.service and its hourly .timer — runs scripts/system-wiki-hourly-update.sh, which refreshes the repo’s internal system wiki from the live system’s state. It is a small thing, but it means the written description of the system is regenerated from reality every hour instead of rotting, and the timer’s journal is itself an audit trail of when documentation last matched the machine.

See also

• Secret handling — why token-using containers can’t be restarted, only recreated.
• CI guards — the canonical-containers check.
• Hard rules — HARD RULE 10.
• Decision records — ADR 0105.
• Deployment — deploy scripts, private-compose topology, and the edge nginx vhosts.