Mail broker

The mail broker is the provider-neutral boundary for mail account sync, metadata search/count/read, and future approved send flows. PRD 0023 and ADR 0076 define v1 as read-only by default and metadata-first: bounded account sync, metadata search/count, message metadata lookup, explicit body fetch support where allowed, sync status, and query audit evidence.

V1 explicitly does not make full mailbox backfill, attachments, draft creation, or send available by default. The runtime code contains a send-approved scaffold, but client.py gates it behind explicit send flags, approval markers, and outbound checks.

Runtime Shape

Layer	Source	Responsibility
Domain	mail_broker/domain.py	Opaque refs, account/message/thread metadata, sync cursors, query predicates, query audit records, redaction.
Query parser	mail_broker/query.py	Gmail-style filter tokens compiled into repository-ready metadata predicates.
Port	mail_broker/ports.py	Durable metadata repository protocol, including predicate-driven list/count.
Storage	mail_broker/postgres_repository.py	steelmoth schema persistence for accounts, cursors, messages, threads, audits, pending approval summaries, and predicate SQL.
Gmail bridge	mail_broker/runtime_gmail_bridge.py	Bounded metadata sync, metadata-backed list/count/read projections, unsupported-filter audit recording.
Runtime client	mail_broker/client.py	API used by runtime Gmail query paths.
Dashboard route	web_chat/routes_mail_broker.py	Admin dashboard health, flags, account connect/disconnect, audit, and approval views.

routes_mail_broker.py now resolves dashboard health/readiness flags from live environment values first (canonical STEELMOTH_* keys, then legacy MOTH_HYBRID_* aliases), then falls back to RuntimeConfig. This keeps mail-broker dashboard and connect-start behavior aligned with admin settings PATCH updates without requiring a runtime restart.

Account provisioning, connect-start, service-account self-link, and callback completion use the canonical mail principal when primary_authentik_email is configured. The requested identity and primary address must derive from the canonical Authentik email; callback completion also checks that the account ref and stored owner match the canonical runtime owner. This keeps dashboard Gmail connection state tied to the same operator identity as runtime auth and /v1/me.

The OAuth handoff path still exists as POST /v1/mail-broker/accounts/connect/start plus callback completion. Connect-start requires an explicit broker redirect or handoff URL from configuration instead of silently falling back to the dashboard callback URL, stores a pending_connect_v1:* marker, and callback completion requires a broker-issued Gmail vault reference rather than a synthetic authentik:* reference.

POST /v1/mail-broker/accounts/connect/service-account is the current dashboard-driven connect path. It requires the mail broker and Gmail provider flags, validates that the tenant has Gmail service-account credentials available through the Agent Vault alias helpers, and upserts the canonical Gmail account as active with agent-vault:gmail:moth-google-sa-json#service_account as the stored credential reference. The endpoint is idempotent for the same canonical identity and preserves any existing last_synced_at; if service-account credentials are absent, it returns FAILED_PRECONDITION and names the OAuth handoff fallback as not implemented yet.

Before broader deterministic planning, runtime now short-circuits clear mailbox list/search requests through _resolve_email_list_intent(). When google_gmail is enabled and the prompt clearly asks to search/find/show/list inbox/mail/email content, the runtime emits a deterministic ToolPlan (tool_name=google_gmail, action=list) and executes the broker Gmail list path directly.

Runtime Gmail list/read calls keep the chat credential identity separate from the broker metadata identity. _handle_google_gmail_tool() still uses gsuite.pick_identity() to choose the Gmail credential token such as mark or moth, but passes _canonical_broker_identity_for() into RuntimeMailBrokerClient.gmail_list() and gmail_read(). The client and maybe_sync_metadata() derive account_ref from broker_identity while the actual Gmail API calls and _build_mail_account() continue to receive the credential identity. For mark with configured primary_authentik_email, the runtime derives the broker key with mail_identity_from_authentik_email(); for moth or missing canonical email configuration, it leaves the chat identity unchanged. This keeps chat-side broker rows aligned with accounts created by connect-start and service-account linking, avoiding a split between mark and the canonical mail identity such as mark_canarybuilds_com.

For google_gmail.read, runtime now resolves the read target before invoking gmail_read(). _resolve_gmail_read_token() keeps direct reads for token-shaped queries, but for filter-shaped or empty queries it runs gmail_list(fetch_cap=1) using derive_gmail_query() intent and then reads the first returned row token (id or provider_message_id). If that list step returns zero rows, runtime returns the broker-unavailable read message and does not call gmail_read() with an empty token.

For the read action specifically, runtime requests the full message body rather than the metadata snippet: runtime.py passes metadata_only=False into gmail_read() at the read call site (list and sync keep their own metadata-only call sites). The bridge then fetches the live body via the injected fetch_body_fn (gsuite.gmail.read_message), which prefers the text/plain part and falls back to an HTML->text conversion for HTML-only messages (gsuite/gmail.py _extract_body/_extract_html_body). Body and snippet are run through mail_broker/text_clean.py (sanitize_body_text) to strip zero-width / soft-hyphen preheader filler before display, so HTML-only marketing mail no longer renders as an empty or garbled preview, and To: populates from the live fetch. This is the explicit, auditable body fetch ADR 0076 contemplates; each fetch is recorded via _record_read_audit, and body text is never logged.

derive_gmail_query() also sets GmailQueryIntent.requires_match for find/about/ specific phrasing (e.g. “tell me more about Karan at Cartesia”). When requires_match is set but the compiled predicate carries no discriminating filter (no content:/ from:/subject:), runtime returns a “no matching emails found” reply (_gmail_no_match_message) instead of surfacing the newest inbox message — closing the wrong-email fallback (the reported “Sanity newsletter” answer to a “Karan at Cartesia” query). A generic “show my inbox” (requires_match false) still lists, and a count query (is:unread) is unaffected. See FAILURE_MODES mode 402 for the residual false-negative risk when a target is unresolvable at snippet level.

RuntimeMailBrokerClient.gmail_list() and .gmail_read() now accept an optional on_step callback and emit bounded step updates for sync_check, syncing, searching, and fetching with running/ok/error status. runtime.py forwards those callbacks from tool execution so dashboard web-chat turns can render granular Gmail progress through tool_step_progress timeline events while broker sync/list/read work is still running.

Gmail tool handling is now broker-only for list, read, and send. The legacy direct gsuite.gmail fallback branch was retired in v2.11.3; when the broker is disabled, the broker client cannot initialize, or the broker returns no payload for a list/read request, runtime returns a short user-visible broker-unavailable message instead of silently falling back to direct Gmail API reads. The legacy fallback flag remains visible in admin health/status contracts, but it no longer re-enables the runtime Gmail direct path. (Amended 2026-07-25: on the operator brain this paragraph now describes only the broker lane — the default direct lane reads Gmail live with no mirror on the read path; see the next section. Send stays broker/approval-gated in both lanes.)

Operator direct-fetch live read lane (default since 2026-07-19)

On the operator brain, chat Gmail list/read/count and the gmail_arrival automation seam read Gmail LIVE under the operator identity each turn — the metadata mirror is no longer on the operator read path (STEELMOTH_GMAIL_READ_LANE, default "direct", surfaced as config.gmail_read_lane). The lane routes only on the operator path — after the broker-off client block (client-gateway reads are byte-identical) and after the positional-referent short-circuit (so “open the most recent one” still serves the cached filtered row instead of listing all mail). Behavior changes under direct:

• A live-fetch failure surfaces an honest “couldn’t reach Gmail — try again” instead of the silent “0 messages” a failed mirror read produced (the 2026-07-17 incident). Counts come from the live message-ID list (page-bounded, an honest “first 500+” cap), never a mirror COUNT(*).
• The gmail_arrival trigger seam detects new mail live (gsuite.gmail.list_messages with an after:<watermark> query, reusing the client-gateway adapt_gmail_row row shape so rule matching is byte-identical) — new-mail automations keep firing with the sync stopped.
• maybe_sync_metadata NO-OPs (Phase 4b, _direct_fetch_lane_active): no new email is copied into the mirror tables — the mirror is frozen, with existing rows kept for search/triage until live replacements land. The AI email tagger is gated off with it (it classifies mirror rows; under the live lane there are no new ones).

The legacy broker/mirror path stays in the tree as the revert: STEELMOTH_GMAIL_READ_LANE=broker restores mirror reads with no code change, and STEELMOTH_GMAIL_READ_LANE_FALLBACK_ENABLED (canonical + MOTH_HYBRID_GMAIL_READ_LANE_FALLBACK_ENABLED, default off) re-enables falling through to the broker list after a live-fetch failure — off by default so failures stay visible. Everything above the lane (the credential-vs-broker identity split, broker_identity derivation, the write path, audits) is unchanged. Regression coverage: tests/test_gmail_direct_fetch_lane.py; the pre-existing broker suites pin themselves to the broker lane in tests/conftest.py.

Broker-backed list/count compiles supported Gmail-style filters before hitting Postgres. derive_gmail_query() still produces the planner/user search string, then parse_metadata_query() accepts is:unread, is:starred, has:attachment, from:<value>, to:<value>, cc:<value>, subject:<text>, content:<text>, newer_than:Nd, older_than:Nd, and after:/before: absolute dates in either YYYY-MM-DD or YYYY/MM/DD (including unpadded month/day forms). in:inbox is accepted as a no-op because broker sync is inbox-scoped today. from:<value> compiles to a case-insensitive substring match over both from_address and from_display, and parser cleanup strips Gmail-style quoting/grouping noise from from: and subject: values before predicate compilation. content:<text> compiles to a case-insensitive substring over subject, snippet, from_display, and from_address (snippet-level — no full-body fetch, no new column); content text drives SEARCH only, never write-target selection (INV-007). Multi-word content values are quoted at emission (content:"karan at cartesia") and parsed through a quote-aware tokenizer (_tokenize_query) so the phrase survives as one token. These tokens become a MetadataQueryPredicate passed through the repository port into SQL against steelmoth.mail_messages. Unsupported tokens are not ignored: list_messages() records a rejected query audit containing the unsupported tokens and returns them to _handle_google_gmail_tool(), which tells the user which filters were rejected.

Metadata sync now enriches broker rows for starred and attachment filters. In addition to the bounded newer_than:<N>d window and unread probe, the Gmail bridge fetches bounded is:starred and has:attachment windows and marks stored message metadata from those provider id sets. The direct Gmail metadata fetcher also carries provider labelIds; a row with STARRED in those labels is stored as starred even if the auxiliary starred query is empty.

Recipients (To/Cc) are persisted in the steelmoth.mail_message_recipients junction table (migration 0029), the recipient sibling of mail_message_labels. The Gmail metadata fetcher now carries the Cc header in addition to To (metadataHeaders + the single-read full fetch). At sync, the bridge parses both headers with email.utils.getaddresses (multi-address aware) into MailRecipient rows, skipping any entry whose address is empty (the address column is NOT NULL with no default — mirrors the label skip guard), assigning a stable recipient_index per recipient_type. upsert_messages() replaces recipients with the same DELETE-then-INSERT … ON CONFLICT DO NOTHING per- message transaction as labels (empty tuples clear the rows). Inbound Gmail metadata never exposes Bcc, so sync writes only 'to' / 'cc'. get_message() and list_messages() read the recipient rows back (the list path uses one set- based message_ref = ANY(...) SELECT to avoid N+1) and the read/list projections render To/Cc headers; on a single full-body read the live fetch’s To still wins. The to:<value> / cc:<value> search operators compile to an EXISTS subquery over mail_message_recipients (mirroring labels_include, ORing the recipient address against the display name like from:); they drive SEARCH only, never write-target selection (INV-007), and To/Cc are render-only PII that is never logged (INV-006).

On the dashboard side, Google connect is a POST form action. GET redirects to /integrations?status=bad_request and does not mutate runtime broker state. POST /api/integrations/google/connect now calls the service-account self-link endpoint directly, then redirects to /integrations?status=connected; it maps 403 responses to denied, service-account 412s to service_account_unavailable, other 412s to broker_required, and unexpected errors to exchange_failed. The OAuth callback route remains in place for the future handoff path and still requires a real vault_reference.

The dashboard integration boundary forwards request headers into resolveMailBrokerSessionIdentity() and applies the shared internal runtime hop helper when calling runtime broker routes. x-forwarded-for is carried as metadata only; trusted x-authentik-* claims require the server-only internal hop credential when that runtime boundary is configured. The Settings sidebar section exposes /workspace/mail-broker and /integrations behind the existing mail_broker feature flag. Mail Broker health/control/connect/audit destinations are no longer separate sidebar sub-anchors; they remain reachable from the Mail Broker page itself.

Migration 0036_mail_broker_remove_orphan_pending_connect deletes one stale steelmoth.mail_accounts row keyed on the old "mark" identity derivation only when its auth_subject_id is an expired pending-connect marker. Its rollback is intentionally limited to removing the migration ledger row; it does not recreate stale pending-connect data.

Migration 0037_mail_broker_query_filter_indexes adds pg_trgm support plus GIN indexes for case-insensitive from_address and subject substring predicates, and partial scope-aware received-at indexes for is:starred and has:attachment. The down migration drops those four indexes and leaves the extension installed.

Automatic email filtering (RFC 0127 / ADR 0243 / ADR 0244)

Merged to main 2026-08-06 (PRs #1569, #1570, #1572, #1574, #1575; task plan tp_e8d5a40026664440961a). The governance pair ADR 0243 + RFC 0127 lands with the independent docs PR #1566 (still open at ingest time). Every flag defaults OFF and dry-run defaults ON — nothing runs in production until the operator enables it (ADR 0243 must flip to Accepted before EMAIL_AUTO_FILTER_DRY_RUN=0). The pipeline connects the existing deterministic tagger (email_tagger.derive_tags) to the existing reversible executors (gmail_write_methods) through one pure core, two triggers, and an audit trail it also learns from:

Piece	Source	Responsibility
Decision core	mail_broker/auto_filter.py	Pure decide_actions/execute_decision. Code-level invariants: two-tier confidence (AI-tier can never auto-execute — it raises), importance guard (starred/IMPORTANT never auto-filed), suppressions, label-absent → skip+audit (never auto-create).
Operator sweep	mail_broker/auto_filter_sweep.py	Periodic live-fetch trigger (newer_than:Nd label:INBOX, never the frozen mirror) wired into the trigger-evaluator thread’s operator branch behind email_auto_filter_enabled; throttled by email_auto_filter_sweep_seconds (default 300). Optional residue AI pass (email_auto_filter_ai_residue_enabled, default off) reuses _classify_ai_tags_batch_async; output is batch-only. Applies the rung gate (below) when email_auto_filter_trust_ladder_enabled is on.
Audit read-back	mail_broker/auto_filter_trust.py, derive_suppressions_and_rules (re-exported by the sweep)	The correction loop over the operator’s own RLS-forced mail_policy_events rows: undo rows (reason_code starting undo:) suppress that message id forever + demote the sender to ask-first; ≥ email_auto_filter_sender_rule_min_filings (default 3) consecutive same-action clean filings promote a sender rule (one undo in the window kills it); sender_reallowed/rule_revoked (decision="review" rows) cancel a demotion / block promotion; unchanged dry-run outcomes are never re-audited.
Rung gate	mail_broker/auto_filter_trust_gate.py (RFC 0128 Stage 2 PR 2 / ADR 0246)	Lane-level gate BOTH decision lanes call behind email_auto_filter_trust_ladder_enabled (default off): with the flag on, a deterministic/sender_rule decision auto-executes only when its (sender, category) pair sits at rung ≥ quiet-auto; below-rung decisions redirect to the EXISTING batch-approval lane (downgrade-only — the ai tier and the two-tier ValueError floor are untouched). Executed filings carry the trust_rung audit marker; rung transitions write rung_transition review rows exactly once (derivation emits a payload only when the derived rung differs from the newest recorded row). Enabled ⇒ the per-lane read-back widens to the trust window (30d); off ⇒ byte-identical v2 (golden-fixture E2E pin).
Digest + undo	mail_broker/auto_filter_digest.py, mail_broker/auto_filter_undo.py	Deterministic plain-English report (AWST timestamps; filings with WHY, per-reason undo rate, sender rules/demotions) delivered via the create_run background-run factory behind email_auto_filter_digest_enabled (default off, cadence email_auto_filter_digest_seconds default 86400). undo_filing applies the exact inverse label delta, Gmail write FIRST and the undo:* audit row SECOND (a failed undo never writes a correction the mailbox doesn’t reflect).
Client push	web_chat/routes_gmail_pubsub.py, mail_broker/auto_filter_push.py, mail_broker/auto_filter_batch.py	POST /webhooks/gmail/pubsub per ADR 0244: fail-closed flag+token gate (404), constant-time token verify (reuses routes_telegram_webhook._verify_secret_token), the envelope’s emailAddress only LOOKS UP the server-side watch registration (never trusted for identity, INV-007), historyId monotonicity + audit dedupe replay guards, unknown addresses ack-and-drop. Batch-approval is the client default (ONE google_gmail.batch_file chip per push, 24h TTL, redeemed through the existing per-id honest path); auto-file requires client_email_auto_filter_mode="auto", and the AI tier stays batch-only for clients too. Applies the same rung gate as the sweep when email_auto_filter_trust_ladder_enabled is on (below-rung decisions join the batch chip).
Watch lifecycle	mail_broker/gmail_watch.py + gsuite/gmail.py watch_user/stop_watch	users().watch() register/renew (24h watch_needs_renewal boundary) / best-effort stop; renewal runs on the evaluator’s client branch behind gmail_pubsub_watch_renewal_enabled.

Audit table. steelmoth.mail_policy_events (pre-existing, forced-RLS via runtime_scope_matches) gained its FIRST writer and reader (ports.record_mail_policy_event / list_mail_policy_events / get_mail_policy_event in postgres_repository.py) — no schema change. Every decision records its WHY in metadata_json (action, label_id, gmail_message_id, dry_run, pre_labels, matched_tags, signals, classifier ∈ {deterministic, ai, sender_rule}) plus the sweep-layer enrichment sender_address + label_name (required by the read-back; operator-visible audit only — logs and signals stay sender-DOMAIN-only).

Watch registrations. Migration 0162_gmail_watch_registrations (steelmoth.mail_gmail_watch_registrations, applied live 2026-08-06 as steelmoth_migrator) persists per-account watch state: scoped composite PK FK’d to mail_accounts ON DELETE CASCADE, unique lower(email_address) routing index, forced RLS, the monotonic history_id replay baseline, and the watch expiration the renewal sweeper re-registers ahead of.

Enablement gates (operator-only, in order): Google Cloud Pub/Sub topic + push subscription + verify token (mounted file, path carrier STEELMOTH_RUNTIME_GMAIL_PUBSUB_VERIFICATION_FILE); ADR 0243 → Accepted; EMAIL_AUTO_FILTER_ENABLED=1 with dry-run first (review the deduped mail_policy_events rows), then EMAIL_AUTO_FILTER_DRY_RUN=0; client-gateway flags (GMAIL_PUBSUB_WEBHOOK_ENABLED, CLIENT_EMAIL_AUTO_FILTER_ENABLED, GMAIL_PUBSUB_WATCH_RENEWAL_ENABLED) for the push path.

Write Outcome Honesty

The write path must never report a false “done”: when the operator approves an action and some or all of it fails, the user must see that in words, and a draft must never read as a sent message. Two layers cooperate.

Target resolution (the three lanes). How a user’s description of an email becomes the exact email acted on (ADR 0157 / INV-022 — the id set is always server-produced and re-validated before any broker call):

Lane	Resolves	Source of the id set
(a) Recent-list	positional / “those” / “all of these” follow-ups	the list the server already showed this turn (planner/gmail_referent.py)
(b) Content-narrow	“the Wise email” against the just-shown rows	substring match scoped strictly to the cached recent-list rows
(c) Search-then-confirm	“all GitHub notifications” when nothing is listed	a server gmail_list search (query from operator text only), promoted into the recent-list, then the existing count-confirm chip

In every lane the description only steers a read-only search query; the resolved id is re-checked by _proposal_ids_in_recent_list before the broker call, so a hallucinated or prompt-injected id can never pass. Lanes (b) and (c) are gated on the ADR 0157 decision; lane (a) is already shipped.

Per-id outcome contract (pinned by tests, not re-built). The executors (mail_broker/gmail_write_methods.py) and the redeem path (web_chat/routes_write_action.py) already honor these; the regression pins in tests/mail_broker/test_gmail_write_outcome_honesty.py and tests/web_chat/test_redeem_outcome_reflects_broker.py lock them so no future edit can reintroduce a false “done”:

• C1 — unwired executor is FAILED. When an executor callable is None, _unavailable_response returns FAILED with every requested id in failed_ids/error_per_id (“UNAVAILABLE”) and succeeded_ids empty (gmail_write_methods.py _unavailable_response). A missing executor is never reported as a silent success.
• C2 — a per-id provider exception is a failure, never a success. In _run_per_id a raise lands the id in failed_ids and error_per_id (the exception kind only — never the raw message, INV-006/INV-013) and never in succeeded_ids; a partial batch keeps the clean ids succeeded and the failed id out.
• C3 — the mirror sync cannot flip a real outcome. apply_label_changes (mail_broker/label_state.py) is a best-effort post-write mirror sync; Gmail is source of truth. A raise inside it is swallowed (logged mirror_drift_after_write) and does not alter the bulk counts — a success stays a success, a partial stays partial.
• C4 — the redeem response mirrors the broker counts. RedeemApprovalResponse.items_failed equals the broker’s failed count and error_per_id carries the per-id reason (_summarise_bulk → RedeemApprovalResponse). A partial or total broker failure can never render as a clean “approved”.
• C5 — drafted is distinct from sent. _summarise_send flags drafted in the per-item result for a DRAFTED write, so a draft never reads as “sent” (the surfacing layer renders “Drafted (pending approval)” vs “Sent”).

A broker exception on the redeem endpoint maps to BAD_GATEWAY (502), never a silent success — so a broker-down redeem is reported as a failure the operator can see.

Safety Rules

• Store provider-neutral opaque refs and metadata, not raw RFC822 or attachment bytes.
• Use redaction helpers for email addresses and provider identifiers in admin summaries and audit views.
• Keep provider credentials behind the broker/Agent Vault boundary; dashboard and normal runtime callers must not receive raw secrets.
• Keep runtime Gmail list/read/send broker-only; do not reintroduce direct provider reads as a silent fallback path.
• Sanitize displayed body/snippet through mail_broker/text_clean.py; render mail body as plain text (never re-parsed as HTML) and never log body text (FAILURE_MODES mode 80).
• Treat To/Cc recipients as email-address PII: render them only into projection payloads, never log them, and use redact_email_address for any admin/audit surface (INV-006). Recipient text drives SEARCH only — it must never select a write target or mailbox identity (INV-007).

Maintenance Notes

• Update this page when mail broker flags, metadata tables, repository methods, metadata filter parsing, runtime Gmail routing, or dashboard mail routes change.
• Schema changes need migration evidence and repository tests before wiki prose is updated.
• Re-check ADR 0076 before enabling any write path.

Known Unknowns

• This page does not prove a specific mail account is connected or synced in the live environment.
• maybe_sync_metadata() still upserts a mail account before syncing when no account row exists for the resolved account_ref; this page does not decide whether that auto-create behavior should remain or become a read-only no-account failure.