Deployment

The root operational entrypoints are run.sh, status.sh, and stop.sh. Dashboard and container work has stricter rules than normal local code changes: builds, recreates, restarts, and duplicate deployable service starts require explicit operator approval in the current thread.

AGENT_REPO_GUIDE.md and CLAUDE.md now also require canonical-root discipline for deployable private-compose mutations: run steelmoth_private build/recreate/start commands from /home/mareekkk/steelmoth. Active implementation work belongs on task branches in WIP worktrees (default /home/mareekkk/steelmoth-wip), while canonical main remains the deploy source of truth.

For normal staging restarts, the repo guide points to scripts/compose-up.sh as the no-build path. Runtime-token handling is ephemeral by default: when no non-empty STEELMOTH_RUNTIME_AGENT_VAULT_TOKEN_FILE exists, the compose start path execs scripts/with-ephemeral-runtime-token.sh, which prepares a temporary token source from an existing mounted Docker secret, systemd credential, or one-shot stdin and deletes the source after compose returns. If a stopped runtime container still points at a deleted temp source, the startup script removes that stopped container before compose recreates it. Explicit rebuilds use scripts/compose-rebuild-changed.sh or scripts/compose-rebuild.sh. Dashboard deployment uses scripts/compose-dashboard-deploy.sh. That script checks dashboard package version drift against compose env, optionally builds the dashboard image, force-recreates the private dashboard service, recreates the standalone public dashboard container, waits for both health endpoints, and compares the private, public, and public-domain CSS fingerprints. When recreating the public dashboard container, it can override public auth routing/cookie env with STEELMOTH_PUBLIC_DASHBOARD_AUTH_URL and STEELMOTH_PUBLIC_DASHBOARD_AUTH_COOKIE_DOMAIN.

The three canonical deploy scripts — scripts/compose-runtime-deploy.sh, scripts/compose-dashboard-deploy.sh, and scripts/compose-provisioner-deploy.sh — run a capability-parity preflight (INV-028, steelmoth_runtime.capability_parity) before touching any container: if the operator brain has a capability that the owlet client template, the client dashboard flags, the /integrations card↔capability map, or the gsuite client OAuth wiring were never given, the deploy refuses with a message naming the missing wiring. Operator-only powers are exempt via owlet_profile; deliberate deferrals live in reviewed ledgers inside the comparator. The guard fails only on confirmed drift — unreadable inputs warn and continue — and STEELMOTH_SKIP_CAPABILITY_PARITY=1 is the loud emergency skip. The same comparator runs strict in CI (invariant-fitness-tests), so drift is normally caught at PR time. prepare_public_broker_env() in the dashboard deploy script force-sets ALL STEELMOTH_DASHBOARD_CLIENT_*_ENABLED integration flags the dashboard code consumes (24 as of 2026-07-12) onto the public container — previously only 5, which hid the connect form for 19 integrations from enrolled clients.

The public dashboard container resolves Auth.js secrets at startup through the prontera broker. The deploy script mounts a one-shot Agent Vault token volume; the dashboard entrypoint removes the token file immediately, attempts to mint a temporary session, and falls back to using the same proxy token in memory only for read-only broker secret fetches when Agent Vault refuses session minting. This keeps the runtime identity read-only after the Agent Vault 0.22 cutover while still loading AUTH_SECRET and AUTHENTIK_CLIENT_SECRET before the Next.js server handles login actions.

The default private startup path now includes both notifier and reminder-daemon profiles via scripts/compose-up.sh, and the default service list includes steelmoth-reminder-daemon, so reminder polling stays on during normal stack bring-up.

Worker-profile startup also brings up steelmoth-morpheus-init, which prepares the shared Morpheus workspace volume. It prefers the configured GitHub remote, falls back to the read-only host Git checkout when available, and can bootstrap a local Git worktree from the host snapshot when the remote has not been seeded yet. The helper keeps running as a periodic fetch loop after setup.

Public docs exposure is represented by ops/nginx/docs.steelmoth.com.conf. That file is a host-level nginx config that proxies docs.steelmoth.com to dashboard wiki routes using the shared dashboard proxy snippet. It is not a container deployment and does not by itself create DNS records, certificates, or a running dashboard/API backend.

The current Steelmoth cutover also has host-level nginx templates for the new public domains. ops/nginx/steelmoth.ai.conf proxies steelmoth.ai / www.steelmoth.ai to the marketing website and proxies app.steelmoth.ai to the standalone public dashboard container on 127.0.0.1:3012. The live container is named stm-dashboard-app and needs both the egress network for prontera Agent Vault DNS and the internal Steelmoth network for runtime API access. ops/nginx/steelmoth.app.conf is a redirect alias only and sends steelmoth.app / www.steelmoth.app traffic to app.steelmoth.ai. Prontera terminates public Authentik login at auth.steelmoth.ai with ops/nginx/auth.steelmoth.ai.conf; that config blocks /if/admin/ and /api/v3/admin/ on the public hostname and redirects auth.steelmoth.app to the .ai auth origin. Authentik admin access is the separate Tailnet-only ops/nginx/auth.tailnet.conf vhost on https://prontera.sole-augmented.ts.net:4443/if/admin/.

Private compose container_name overrides now use the stm-* prefix for the Steelmoth stack. The default running private baseline is stm-data-postgres, stm-data-redis, stm-memory-memlink, stm-runtime-api, stm-worker-sidecars, stm-worker-notifications, and stm-worker-reminders; optional profile containers follow the same prefix. Do not remove or rename stopped Steelmoth-era stm-* containers while working on this Steelmoth stack.

Private compose now also defines steelmoth-llm-usage-proxy, a loopback Anthropic-format proxy on port 4250 by default. It reuses the runtime image and serves steelmoth_runtime.llm_usage.proxy_app:app to record external-tool token usage without storing caller credentials. Because it is a deployable service, starting, rebuilding, or recreating it still requires current operator approval under the container mutation rules.

The observability stack (ADR 0226, phased in 2026-07-24) added a metrics plane without changing the canonical running baseline. steelmoth_runtime.metrics mounts /metrics plus a low-cardinality RED middleware on the brain (web_chat/app.py) and the LLM usage proxy; the Telegram dispatcher serves an inline /metrics (its purity invariant forbids importing the shared module); /metrics is auth-exempt for scraping (added to the auth public-paths alongside the health endpoints, memlink_scope_auth._PUBLIC_PATHS) and carries no secrets/PII by label discipline (ADR 0226 §4). Five off-the-shelf exporters (node, cAdvisor, Postgres, Redis, blackbox) live in compose.steelmoth.private.yml behind the observability profile — which NO canonical deploy script activates (compose-up/compose-rebuild use the private/worker/browser-pool profiles), so the running stack’s convergence is unchanged. Their scrape ports bind to the host tailnet IP only: the juno observability VPS (tag:observability) scrapes, the public internet cannot. The Postgres exporter uses the read-only monitor role, and its DSN comes from an env file kept outside the repo (PG_EXPORTER_DSN default-empty so unrelated compose invocations still interpolate). Phase 2 double-binds the runtime app port on the tailnet IP alongside the loopback binding so juno can scrape the brain’s /metrics and blackbox-probe the health endpoints; every loopback consumer is untouched. Phase 3 added the juno-side Grafana dashboards and Prometheus alert rules plus POST /v1/observability/alertmanager (web_chat/routes_observability.py), which turns a firing Alertmanager alert into an operator Inbox item. That webhook’s bearer token is a mounted file (/run/secrets/alert_webhook_token via ALERT_WEBHOOK_TOKEN_FILE, kept outside the repo — never an env var, because the raw-secret boot guard refuses those); an absent token leaves the webhook disabled with a loud startup log, never open. Phase 4 added the dashboard’s own /api/metrics (dashboard/app/api/metrics/route.ts) and the memlink /metrics sidecar (memlink_hybrid/infrastructure/metrics_server.py). Prometheus, Grafana, and Alertmanager themselves run on juno from observability-assets/ (a prontera-assets-style documentation-plus-config tree), not on this VPS. Phase 5 (the auditability layer, ADR 0227) is design-only as of 2026-07-25.

The same private compose file now standardizes the default operator identity for private runtime-facing services: owner defaults resolve to owner_mark_canarybuilds_com, the primary Authentik email defaults to mark@canarybuilds.com, the token dashboard slice is enabled, and mail-broker Gmail/admin flags are enabled through canonical STEELMOTH_* env keys. These are deployment defaults; live behavior still depends on the environment used to start the running containers.

The Memlink service env in private/shared compose explicitly enables the deployed memory ingestion and retrieval gates: LLM extraction, embeddings, relation extraction, retrieval orchestration, and the retrieval router. Live behavior still depends on recreating the running Memlink container from the updated compose config.

Private compose now also sets MOTH_HYBRID_TELEGRAM_AGENT_VAULT_BROKER_BASE_URL for steelmoth-worker-moth and steelmoth-notifier, defaulting to https://prontera.sole-augmented.ts.net/telegram. In broker mode, runtime config falls back to ${AGENT_VAULT_ADDR}/telegram only when this env is missing. In the current compose file, steelmoth-api enables broker mode but does not set the explicit base URL, so that container still relies on the fallback unless the env is provided externally.

The prontera Telegram Bot API broker resolves the bot token inside Agent Vault. For the Mark client rollout it first tries client-mark/CLIENT_MARK_TELEGRAM_BOT_TOKEN, the key written by the dashboard Telegram connector, then falls back to the existing client-mark/TELEGRAM_TOKEN so current Telegram delivery remains live before the operator completes the new connector flow. The broker still validates callers against the client-mark vault; steelmoth containers never receive the bot token.

Maintenance Notes

Update this page when canonical deploy scripts or hard deploy rules change.
Update this page when tracked edge configs or public docs publish paths change.
This page is documentation only; always re-read the exact script before running operational commands.

Known Unknowns

This page does not assert which containers are currently running.