nixfleet architecture: declarative, signed, sovereign

Design principle. The control plane is a caching router for signed declarative intent. It holds no secrets, forges no trust, and can be rebuilt from empty state without data loss.

Every structural decision below serves that inversion of trust. In today’s nixfleet, the control plane is the source of truth - compromise it, and the fleet follows wherever it points. In this design, truth lives in git and in signing keys; the control plane only moves already-signed artifacts around. Destroying the control plane is an outage, not a breach. Rebuilding it from the flake and the signed artifacts in storage gives you back the same fleet.

This document consolidates the v0.2 design: the spine, the RFCs, the Rust/Nix boundary, the content-addressing generalization, and the supporting homelab infrastructure into a single architecture with a single build order.

1. Components

Each component below has a defined role, a defined owner, and a defined trust property. Components only interact through versioned, typed boundaries.

1.1 The flake (source of truth)

Git-tracked, hosted on a self-run Forgejo instance on the coordinator. Contains:

• nixosConfigurations.<host> - per-host NixOS modules.
• fleet flake output - produced by mkFleet { ... } per RFC-0001; describes hosts, tags, channels, rollout policies, edges, disruption budgets.
• age.secrets.<name> - secrets encrypted per-recipient at rest, declared alongside the fleet.
• nixfleet.compliance.controls.<name> - typed controls with static evaluate and runtime probe projections.

Trust role: primary trust root for intent. A commit that passes review IS the desired state. No other place in the system can claim “the fleet should be X” without a corresponding commit.

Framework Nix surface (mkFleet, mkHost, hostSpec, scopes)

The framework exposes two builders: nixfleet.lib.mkFleet (declarative fleet topology — hosts + channels + rollout policies; the typical operator path) and nixfleet.lib.mkHost (the underlying per-host primitive, also usable directly for one-off setups). The mkFleet wrapper iterates hosts and calls mkHost per host with hostName, platform, and fleetResolved pre-bound; built configurations surface as fleet.nixosConfigurations. Operators using mkHost directly pass fleetResolved themselves (or omit it on one-off rigs where probe topology isn’t load-bearing).

mkHost takes a typed hostSpec identity record plus a list of consumer modules and returns a nixosSystem or darwinSystem; it does not impose a fleet/org/role DSL above hostSpec. An auto-discovered set of service modules under modules/scopes/ self-activate via services.<name>.enable options gated by lib.mkIf. Adding a new scope requires no mkHost change; inactive scopes cost zero at evaluation. Roles, when used, are scope bundles defined in consuming fleets that set enable defaults with lib.mkDefault; the framework itself has no “role” concept.

hostSpec carries identity and locale data only — hostname, primary user, home directory, timezone, locale, platform marker, root access keys. Behaviour belongs to scopes.

The agent and control plane are themselves NixOS service modules (services.nixfleet-agent, services.nixfleet-control-plane), not opinionated profiles. Host operators stay in charge of firewall, persistence, and TLS posture; framework concerns stay in the services.* namespace, with secrets wired through the consumer’s chosen backend (agenix, sops, vault). Fleet repos extend hostSpec with their own opinionated capability flags (isGraphical, isDev, theme) by declaring additional options in plain NixOS modules passed via nixosArgs.modules on a host — the NixOS module system merges option declarations, so consumer extensions compose with framework-defined options without modifying the framework.

1.2 Continuous integration (the intent-signing oracle)

Runs on the coordinator (Hercules CI agent, or Forgejo Actions with a self-hosted runner). On every commit to a watched branch:

1. Evaluates the flake; builds every host’s closure.
2. Runs static compliance gates (type = static controls evaluated against each config). Failure aborts the pipeline; no release is produced.
3. Pushes closures to attic, which signs them with its ed25519 private key.
4. Produces fleet.resolved.json (RFC-0001 §4.1 projection) and signs it with the CI release key.
5. Updates channel pointers (stable, edge-slow, …) to the new git ref, committing the signed artifact set.

Trust role: converts reviewed-and-merged commits into signed releases. CI key lives in an HSM, ideally on the coordinator with a TPM-backed keyslot. Rotation is a documented procedure, not an incident response.

1.3 Attic binary cache

Runs on the coordinator. Stores every closure CI produces, content-addressed by sha256, signed with its own ed25519 key. Clients verify signatures against a pinned public key embedded in their NixOS config.

Trust role: self-verifying content store. A compromised attic host cannot forge closures: the signing key is the trust root, not the host. An attacker who steals attic’s disk learns what closures have been built; they cannot inject malicious ones into any host.

1.4 Control plane (the router)

Rust/Axum service, SQLite for operational state, mTLS for all incoming connections. What it does:

• Polls the git forge for channel-ref updates (or receives webhooks).
• Fetches the signed fleet.resolved.json for each channel rev; verifies the CI signature; if it doesn’t verify, refuses to reconcile.
• Runs the reconciler (RFC-0002 §4 decision procedure) on each tick.
• Serves agent check-ins (RFC-0003): tells each host its current target closure hash, current rollout membership, expected probes.
• Records observed state (last check-in, current generation, probe results) as a cache of what agents have reported.

What it does not do:

• Hold any secret material (all secrets are agenix-encrypted in the flake).
• Sign anything that a host is asked to trust (closures -> attic; intent -> CI; probe outputs -> hosts).
• Store anything that cannot be recomputed from git + attic + agent check-ins.

Trust role: router. Compromise yields at worst a denial of service (refuse to propagate updates) or a replay attack (point hosts at stale-but-valid closures). Cannot inject code, cannot read secrets, cannot forge compliance evidence.

Destroying the control plane and rebuilding from scratch: re-pull fleet.resolved from git, re-fetch channel refs, let agents check in on their next poll cycle. Operational state reconstructs within one reconcile tick per channel.

Scaling envelope

The CP’s SQLite handle is wrapped in tokio::sync::Mutex<rusqlite::Connection>. WAL mode is enabled, so reads proceed while a write is in flight at the file level, but every operation that goes through the mutex serializes on the mutex itself. The current factoring is sized for fleets of O(100) hosts checking in at the configured polling cadence (default 60s with jitter); past ~150 hosts, dispatch bursts and report ingestion start to contend on the mutex and p99 dispatch latency can rise above one polling cycle. The bound is conservative, not load-tested, and intentionally invisible to operators today beyond the host-count log emitted on snapshot prime.

The path past the bound is a connection pool (deadpool-sqlite - same rusqlite::Connection surface, tokio-native async fn get()), scoped to when measurable contention appears: fleet size > 150, p99 dispatch_for_host exceeding the polling cycle in steady state, or operator-visible queueing in the journal. Migration is a wrapper swap plus an await per use site - same SQL, same schema, same behaviour, multi-connection on the inside. The mutex is the v0.2 commitment; the pool is the v0.3 trigger.

1.5 Agent (the actuator)

Rust daemon running on every managed host. Single-binary, minimal dependencies. What it does:

• Polls the control plane over mTLS at the channel’s declared cadence.
• On a new target: fetches the closure from attic (not from the control plane), verifies attic’s signature, verifies the hash.
• Decrypts host-scoped secrets from the flake using the host’s private ed25519 (SSH host key).
• Runs nixos-rebuild switch. Opens the magic-rollback confirm window.
• On post-activation boot: phones home with bootId + probe results. On silence past the window: auto-rollback.
• Reports current generation + probe outcomes at next check-in.

Self-switch resilience. When the new generation changes the agent itself, switch-to-configuration switch must complete after systemd stops the agent’s own cgroup. The agent’s apply path is fire-and-forget: the switch is queued in a detached transient systemd unit (systemd-run --unit=nixfleet-switch) before activation begins, so systemd stopping the agent does not kill the in-flight activation. The agent does not wait on the child; it polls /run/current-system until the symlink matches the desired generation, with a bounded timeout. If the agent is killed mid-poll, the new agent re-runs at startup and reconciles state by reading the active generation. The same mechanism handles rollback. The carve-out: switch inhibitors (dbus, systemd, kernel, init swaps) trip an inline pre-check that downgrades to nix-env --set only and posts ActivationDeferred, leaving the new generation to activate on next reboot - see ./contracts.md §I.7.

What it does not do:

• Accept arbitrary commands from the control plane. The vocabulary is only “your target is closure sha256-X”. Not “run this shell snippet”, ever.
• Trust the control plane’s closure recommendation without signature verification against attic’s pinned key.
• Hold long-lived credentials beyond its mTLS client cert (short-lived, auto-rotating) and its SSH host key (machine-lifetime).

Trust role: local decision-maker. The agent is the last line of defense against a compromised control plane. If signatures don’t verify, it refuses. If the magic-rollback window closes silently, it reverts. Every decision is made with information the agent can independently verify.

1.6 Compliance framework (enforceable evidence)

nixfleet-compliance repo. Controls declared as typed units with two projections:

• evaluate :: config -> { passed, evidence } - pure, runs at CI time. Violations fail static gate; no release produced.
• probe :: { command, expectedShape, schemaVersion } - descriptor consumed by the agent post-activation. Output is canonicalized and signed by the host’s key, producing non-repudiable evidence.

Every control belongs to one or more frameworks (ANSSI-BP-028, NIS2, DORA, ISO 27001). A channel’s compliance.frameworks list enforces the union of controls.

Trust role: turns NixOS configuration into auditable, content-addressed evidence. The chain: host key signs probe output -> closure hash pins what was running -> git commit pins what was intended. An auditor verifies the whole chain without trusting the control plane, the CI runner, or the operator.

1.7 Secrets (zero-knowledge ferrying)

agenix-style: secrets encrypted per-recipient in git. Recipients are host SSH pubkeys, declared in fleet.nix under secrets.<name>.recipients. Ciphertext ships as part of the closure or as separate content-addressed blobs. Decryption happens on the target host, using its private SSH host key, into tmpfs only.

Trust role: eliminates the control plane from the secret path entirely. A fully-public flake repo combined with good host key hygiene gives you the same secrecy guarantees as a locked-down vault. Rotation = re-encrypt + commit + redeploy.

1.8 Test fabric (microvm.nix)

In-flake fixture. Each scenario declares N microvms (cloud-hypervisor, shared Nix store via virtiofs), a stub control plane, and an expected action plan. Exercises: clean rollout, canary rollback on probe failure, agent offline during rollout, host key rotation, cert revocation, compromised-control-plane simulation (swap signing key, verify hosts refuse).

Runs in nix flake check on PR for small scenarios (10 hosts); nightly for larger (50).

Trust role: the only honest way to know the protocol is correct. Every state machine in RFC-0002 must have fixture coverage. No transition lands without a test that exercises it. The reconciler is a pure function (§2 below); there’s no excuse for not testing it exhaustively.

2. The Nix / Rust boundary

Nix owns evaluation. mkFleet, selector algebra, compliance control declarations, secret recipient lists. Produces signed artifacts at CI time. Never called at runtime.

Rust owns execution. Reconciler, state machines, agent protocol, activation, probe running, CLI. Takes signed artifacts as input; never evaluates Nix.

Boundaries. Three typed, versioned contracts:

1. fleet.resolved.json - Nix -> Rust, via CI, signed.
2. Compliance probe descriptors - Nix -> Rust, embedded in closures, schema-versioned.
3. Agent/control-plane wire protocol - Rust ↔ Rust, versioned in header.

Crossing a boundary always means a version check and a signature verification (where applicable). Nothing is trusted by proximity.

3. The main flow

The happy path, one commit from push to all hosts converged:

1. operator ─── git push ──────────────▶ Forgejo
                                             │
2. Forgejo ─── webhook ────────────────▶ CI
                                             │
3. CI evaluates flake -> builds closures per host
   CI runs static compliance gate
   CI pushes closures -> attic (signs)
   CI produces fleet.resolved.json (signs)
   CI updates channel pointer, commits
                                             │
4. control plane polls/receives ◀───── git ref change
   verifies fleet.resolved signature
   reconciler emits action plan for new rollout
                                             │
5. agent (workstation, canary wave) long-polls ─▶ control plane
   control plane responds on /v1/agent/dispatch:
     target = sha256-X, rollout R, wave 0
                                             │
6. agent fetches sha256-X from attic
   verifies attic signature, verifies hash
   decrypts host-scoped secrets locally
   activates -> confirm window opens
                                             │
7. agent boots new generation
   runs runtime probes, signs outputs with host key
   posts ActivationCompleted to /v1/agent/events
   control plane reduces the event into rollout state
                                             │
8. soak elapses -> wave 0 promoted -> wave 1 begins
   attic-01 receives dispatch; same sequence
                                             │
9. wave 1 converges -> rollout Converged
   channel's lastRolledRef updated to new rev

Nothing in this flow requires trusting the control plane with anything it shouldn’t have. The control plane knows: which hosts exist, which closure hash each should run, which rollouts are in flight, what check-ins have happened. It does not know: what’s in the closures, what’s in the secrets, whether the probe outputs were forged (it can verify via host keys, but it could not fabricate them).

4. The trust flow

Independent of the operational flow, trace where trust originates and where it’s verified. This is the diagram that should stay true forever:

trust origins (signing keys, offline, rotatable):

   ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐
   │  CI release key │   │  attic cache key│   │  org root key   │
   │  (signs fleet.  │   │  (signs closures│   │  (signs bootstrap│
   │   resolved)     │   │                 │   │   tokens)       │
   └────────┬────────┘   └────────┬────────┘   └────────┬────────┘
            │                     │                     │
            │                     │                     │
trust per-host (derived, short-lived):
            │                     │                     │
            │            ┌────────┴────────┐            │
            │            │  host SSH key   │            │
            │            │  (signs probe   │            │
            │            │   outputs,      │            │
            │            │   decrypts      │            │
            │            │   secrets)      │            │
            │            └────────┬────────┘            │
            │                     │                     │
            │            ┌────────┴────────┐            │
            │            │  agent mTLS cert│            │
            │            │  (short-lived,  │            │
            │            │   derived from  │            │
            │            │   host key at   │            │
            │            │   enrollment)   │◀───────────┘
            │            └─────────────────┘
            │
verification happens everywhere (runtime, cheap):

   agents verify attic signatures on every closure fetch.
   agents verify CI signatures on every fleet.resolved fetch (if fetched directly).
   control plane verifies CI signatures before reconciling new revisions.
   control plane verifies agent mTLS certs on every check-in.
   auditors verify host-key signatures on probe outputs post-hoc.

Four keys. Everything else is derived. Compromise of any derived credential has a bounded blast radius because the roots are separate.

5. The failure cases

The design earns its keep when things go wrong. Walking through the scenarios:

Control plane host is compromised (attacker has root on the VM hosting Axum/SQLite). Attacker cannot: read secrets, forge closures, inject malicious code. Can: refuse to serve updates (DoS), serve stale-but-valid targets (replay). Mitigation: agents refuse to accept targets older than a configurable freshness window signed by CI.

Attic cache host is compromised. Attacker cannot forge closures (signing key is the trust root). Can: delete closures (hosts fall back to building locally if builders are present, else stall). Can: learn what closures exist (metadata leak). Disk loss is recoverable from CI artifacts.

CI runner is compromised. Serious - attacker can sign releases. Mitigation: CI key in HSM, CI runner in restricted environment, signing operation requires hardware confirmation. Detection: anomalous release signatures (signed outside normal CI run time) trip alerts. Recovery: revoke CI key, re-sign from clean environment, all agents refuse old-key artifacts.

Host is compromised (root on the target machine). Attacker can: read secrets decrypted for that host, forge probe outputs signed with that host’s key. Cannot: affect other hosts, modify the control plane’s view of the fleet. Detection: probe outputs from a compromised host might show inconsistencies that trigger runtime gates. Mitigation: TPM-backed host keys make key extraction hard; short-lived agent mTLS certs limit persistence.

Operator is compromised / malicious. If they have git commit access: can push any config. Mitigation: protected branches, mandatory review, CI static compliance gate catches obviously-bad configs (SSH password auth, disabled firewall, etc.) before release. Post-hoc: git history is the audit log.

Network partition mid-rollout. Agents cache last known desired state, continue operating. Magic rollback handles post-activation failures locally. Rollout pauses until partition heals; disruption budgets prevent cascade.

6. CP-resident state by recovery profile

Every SQLite table the CP keeps falls into one of two recovery classes. The classification is load-bearing for done-criterion #1 of §7: rebuilding the CP from empty state must restore the fleet’s desired-state guarantees within one reconcile cycle, not just “approximately reach steady state eventually”.

• Soft state — recoverable from agent inputs on the next checkin cycle, or acceptable as a one-window operational regression:

  • token_replay — bootstrap nonces with 24h TTL. Loss extends the replay window by up to one TTL. Bounded; no breach.
  • pending_confirms — in-flight activation deadlines. Loss could force the agent into an unnecessary local rollback when its confirm POST hits a 410. Mitigated by orphan-confirm recovery: when the agent’s reported closure_hash matches the verified target, the handler synthesises a confirmed row and returns 204 instead of 410.
  • host_rollout_state — per-host soak markers. Loss restarts soak windows from zero. Mitigated by agent-attested last_confirmed_at: the agent persists the moment of its most recent successful confirm and echoes it on every checkin; the CP repopulates last_healthy_since from the attestation, clamped to min(now, attested).
  • host_reports — SQLite-backed. Hydrated at boot; outstanding ComplianceFailure / RuntimeGateError events survive CP restarts so the wave-promotion gate stays armed across the unlock window. Soft only because individual late-arriving reports retry on the next checkin.

• Hard state — must come from signed artifacts pre-existing in git or from operator-declared trust roots:

  • cert_revocations — agent-cert revocation list. Loss is a security regression — previously-revoked certs become valid again. Mitigated by the signed revocations.json sidecar: operator commits revocations to the fleet repo, CI signs the artifact with the same ciReleaseKey that signs fleet.resolved.json, the CP fetches + verifies + replays on every reconcile tick. Recovery from empty is “one tick later, table populated from the signed artifact.”
  • trust.json — the trust roots themselves. Sourced from the flake at build time; rebuildable as long as the flake survives. A deferred TPM-bound issuance CA is tracked as future work.

The principle: the CP holds nothing whose loss creates a security regression on rebuild, and nothing whose loss creates more than a one-window operational regression. Orphan-confirm recovery, the last_confirmed_at attestation, and the signed revocations sidecar are what make it true.

7. When is it actually done

Four falsifiable statements. If any is false, the design hasn’t landed:

1. Destroying the control plane’s database and rebuilding from empty state results in full fleet visibility within one reconcile cycle, with zero operator intervention beyond restarting the service. Strict reading: every CP-resident table either repopulates from agent inputs (soft state — token_replay, pending_confirms, host_rollout_state) or from a signed artifact in git (hard state — cert_revocations via the signed revocations.json sidecar, trust.json via the flake). See §6 for the per-table classification.
2. An auditor can be handed a host’s hostname + a date range, and - without access to the control plane - produce a cryptographically-verifiable statement of “on this date, this host ran closure sha256-X, which was built from commit Y, and passed compliance controls Z₁..Zₙ with signed probe outputs matching the declared schemas”.
3. The control plane’s disk contents, stolen in their entirety, yield zero plaintext secret material.
4. A deliberately-corrupted closure pushed to attic (bypassing CI) is rejected by every agent; a deliberately-modified fleet.resolved served by the control plane is rejected by the control plane’s own signature verification.

If all four hold, the slogan is true. If not, find the gap and close it before calling the framework done.

8. Source tree map

nixfleet/
├── flake.nix                      <- entry point, inputs, flake-parts wiring
├── Cargo.toml                     <- Rust workspace root
├── crane-workspace.nix            <- Nix wrapper around crane for Rust builds
│
├── README.md, CHANGELOG.md, etc.  <- consumer-facing docs (root meta-files)
├── SECURITY.md, CONTRIBUTING.md, CODE_OF_CONDUCT.md, LICENSE-*
│
├── contracts/                     <- schemas. Top-level so import-tree skips
│   ├── host-spec.nix              │  them. They declare options; impls
│   ├── persistence.nix            │  satisfy them. NO mechanism here.
│   └── trust.nix                  ↓
│
├── impls/                         <- pluggable contract implementations,
│   ├── persistence/impermanence.nix
│   ├── keyslots/tpm/
│   ├── gitops/forgejo.nix
│   └── secrets/default.nix        ↑  exposed as flake.scopes.<family>.<impl>
│
├── lib/                           <- public API (mkHost, mkFleet, ...)
│   ├── default.nix                │  wired entry: imports flake inputs
│   ├── mk-fleet.nix               │  pure entry: just nixpkgs lib
│   ├── mk-host.nix                │
│   └── mk-vm-apps.nix             ↓
│
├── modules/                       <- flake-parts modules (auto-imported by
│   ├── flake-module.nix           │  import-tree, except _-prefixed files)
│   ├── apps.nix                   │  These declare flake outputs:
│   ├── formatter.nix              │    flake.lib, .scopes, .nixosModules
│   ├── options-doc.nix            │    perSystem.apps, .packages, .checks
│   ├── rust-packages.nix          │    .devShells, .formatter
│   │
│   ├── core/                      <- minimal NixOS/Darwin glue
│   │   ├── _nixos.nix             │  hostSpec -> standard options,
│   │   └── _darwin.nix            ↓  flake-mode nix prereqs.
│   │
│   ├── scopes/nixfleet/           <- framework runtime services
│   │   ├── _agent.nix             │  systemd unit for the agent
│   │   ├── _agent-darwin.nix      │  launchd unit for the agent (macOS)
│   │   ├── _control-plane.nix     │  systemd unit for the CP
│   │   ├── _cache.nix             │  binary-cache client wiring
│   │   ├── _microvm-host.nix      │  microvm host (bridge, NAT, dnsmasq)
│   │   ├── _operator.nix          │  workstation tools (mint-token, etc.)
│   │   └── _trust-json.nix        ↓  shared helper: build trust.json
│   │
│   └── tests/                     <- flake-parts entries that register
│       ├── eval.nix               │  the checks that the test fabric runs
│       ├── harness.nix            │
│       ├── _agent-v2-trust.nix    │
│       ├── _cp-v2-trust.nix       │
│       └── _trust-options.nix     ↓
│
├── crates/                        <- the Rust workspace
│   ├── nixfleet-proto/            <- shared types (boundary contracts)
│   ├── nixfleet-canonicalize/     <- JCS canonicalizer (lib + bin)
│   ├── nixfleet-reconciler/       <- pure decision engine (lib only)
│   ├── nixfleet-agent/            <- per-host actuator daemon
│   ├── nixfleet-control-plane/    <- Axum HTTP server + reconcile loop
│   ├── nixfleet-cli/              <- operator workstation tools
│   ├── nixfleet-release/          <- CI release pipeline orchestrator
│   └── nixfleet-verify-artifact/  <- offline verifier for auditors
│
├── tests/                         <- test code, fixtures, harness
│   ├── fixtures/                  │  Static QEMU references
│   ├── harness/                   │  microvm.nix scenarios
│   └── lib/mk-fleet/              ↓  positive + negative eval fixtures
│
└── docs/                          <- human-readable docs
    ├── README.md                  │  navigation index
    ├── design/                    │  this file + contracts.md + source-layout.md
    ├── reference/                 │  harness.md + per-crate overviews
    ├── operations/                │  disaster-recovery + operator-cookbook + troubleshooting
    ├── rfcs/                      │  RFC-0001 ... RFC-0012
    └── mdbook/                    ↓  composed book; `{{#include}}` wrappers

Convention: _*.nix is skipped by import-tree. Files like _agent.nix are imported explicitly by lib/mk-host.nix. This is why agent/CP modules end up in every host’s module list while test modules under modules/tests/ only register via their non-prefixed siblings.

9. The Nix layer

10.1 Flake wiring

flake.nix is the entry point. Three jobs:

1. Declares inputs - nixpkgs, darwin, home-manager, flake-parts, import-tree, disko, microvm, crane, lanzaboote, treefmt-nix, nixos-anywhere, nixos-hardware, impermanence.
2. Picks the system matrix - x86_64-linux, aarch64-linux, aarch64-darwin, x86_64-darwin.
3. Calls flake-parts.lib.mkFlake with ./modules/ auto-imported by import-tree.

outputs = inputs:
  inputs.flake-parts.lib.mkFlake { inherit inputs; } (
    (inputs.import-tree ./modules)
    // { systems = [ "x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin" ]; }
  );

import-tree walks modules/, skips _*.nix, returns an attrset of flake-parts modules; mkFlake merges them. This decomposition is why outputs (apps, packages, checks, devShells, lib, scopes) live in five small files (flake-module.nix, apps.nix, formatter.nix, options-doc.nix, rust-packages.nix) rather than one monolith.

nixpkgs is pinned to nixos-unstable; the framework re-pins consumers via follows, so a fleet’s effective nixpkgs = the framework’s. impermanence is required only by fleets that import flake.scopes.persistence.impermanence; inert otherwise.

10.2 Public API (lib/)

Four exports: mkHost, mkFleet, mkVmApps, plus mergeFleets and withSignature. Wiring in lib/default.nix:

{ inputs, lib }: let
  mkFleetImpl = import ./mk-fleet.nix { inherit lib; };
in {
  mkHost     = import ./mk-host.nix    { inherit inputs lib; };
  mkVmApps   = import ./mk-vm-apps.nix { inherit inputs; };
  inherit (mkFleetImpl) mkFleet mergeFleets withSignature;
}

mkFleet is pure (just needs lib), so the canonicalize binary and eval-only tests can import lib/mk-fleet.nix directly without dragging in flake inputs. mkHost and mkVmApps need inputs because they build actual systems / spawn QEMU.

mkHost - the primary API (lib/mk-host.nix)

One function. Returns a NixOS or Darwin system, ready for nixos-rebuild / darwin-rebuild.

mkHost {
  hostName     = "my-server";          # required
  platform     = "x86_64-linux";       # selects nixosSystem vs darwinSystem
  stateVersion = "24.11";              # NixOS only
  hostSpec     = { userName = "deploy"; rootSshKeys = [ "ssh-ed25519 ..." ]; };
  modules      = [ ... ];                # consumer modules
  isVm         = false;                # if true, inject test fixtures
  extraInputs  = {};                   # consumer inputs to make visible
}

Internally:

1. Picks nixpkgs.lib.nixosSystem or darwin.lib.darwinSystem based on platform.
2. Auto-injects framework modules: contracts/host-spec.nix, contracts/persistence.nix, modules/core/_nixos.nix or _darwin.nix, all six modules/scopes/nixfleet/_*.nix. (Darwin gets only the agent-darwin and core-darwin modules.)
3. Sets hostSpec defaults (mkDefault-wrapped so consumer overrides win).
4. Forces hostSpec.hostName = hostName exactly (never overrideable).
5. Merges consumer’s modules last.

Every framework service module is auto-injected but disabled by default. Zero cost unless the host opts in (services.nixfleet-agent.enable = true; etc.). The framework deliberately exposes one builder; no fleet/org/role taxonomy.

mkFleet - the fleet topology (lib/mk-fleet.nix)

Consumes a fleet description and produces fleet.resolved - the canonical projection that CI signs and the control plane consumes. Five major parts:

1. hosts - atomic units. Each declares system, configuration, tags, channel.
2. tags - flat, non-hierarchical groupings.
3. channels - release trains. Each pins rolloutPolicy, freshnessWindow, signingIntervalMinutes, reconcileIntervalMinutes, compliance.frameworks.
4. rolloutPolicies - named strategies. Each declares waves (selector + soakMinutes), a healthGate, an onHealthFailure action.
5. edges + disruptionBudgets - DAG ordering and concurrent-change limits.

Selector algebra: tags, tagsAny, hosts, channel, all, not, and. No wildcards; resolves at eval time.

mkFleet runs invariant checks - every host’s channel exists, every channel’s policy exists, edges form a DAG, freshnessWindow ≥ 2 × signingIntervalMinutes, every selector resolves to ≥1 host. Compliance failures in enforce mode block the build before signing. Output is fleet.resolved with null placeholders for signedAt, ciCommit, closureHash - filled by nixfleet-release at CI time.

mergeFleets strict-merges multiple fleet inputs (collisions throw); withSignature stamps meta after CI builds.

mkVmApps - local VM lifecycle (lib/mk-vm-apps.nix)

Returns five flake apps: build-vm, start-vm, stop-vm, clean-vm, test-vm. Linux-only. The 37-line composer is thin; platform abstraction lives in lib/vm-platform.nix, shared bash in lib/vm-helpers.sh, per-app scripts in lib/vm-scripts/. State under ~/.local/share/nixfleet/vms/.

Flake-output modules (modules/*.nix)

• modules/flake-module.nix - exports flake.lib, flake.nixosModules.nixfleet-core, flake.scopes.<family>.<impl>.
• modules/apps.nix - declares perSystem apps. Most importantly, validate - the single test-suite entry (nix run .#validate -- --all runs format, eval, host builds, Rust tests, VM scenarios). Also exposes the agent / CP / cli / canonicalize / verify-artifact / release binaries.
• modules/formatter.nix - nix fmt via treefmt-nix (Alejandra + shfmt + deadnix).
• modules/options-doc.nix - generates the Markdown options reference.
• modules/rust-packages.nix - wires crane to build the workspace, exports docs-site, declares devShells.default.

10.3 Contracts

Pure schemas under contracts/. They declare options; they implement nothing. Kept top-level (not under modules/) so import-tree doesn’t treat them as flake-parts modules and leak assertions into flake-level scope. The cross-reference for every boundary-crossing artifact is ./contracts.md.

hostSpec - universal identity (contracts/host-spec.nix)

Every host has one. Identity (hostname, primary user, home dir), locale (timezone, locale, keyboard layout), access (root password file, root SSH keys), networking hints, secrets-backend hints, platform marker (isDarwin). The agent reads hostSpec.userName; persistence reads it for ownership; core reads hostSpec.hostName and stamps it into networking.hostName.

hostSpec carries identity only; behaviour is via scope enable options. Fleets extend hostSpec with their own options via plain NixOS modules.

persistence - what survives reboots (contracts/persistence.nix)

options.nixfleet.persistence = {
  enable      = lib.mkEnableOption "system-level persistence";
  persistRoot = lib.mkOption { type = str; default = "/persist"; };
  directories = lib.mkOption { type = listOf (either str (attrsOf anything)); default = []; };
  files       = lib.mkOption { type = listOf (either str (attrsOf anything)); default = []; };
};

Baseline contributions (/etc/nixos, /etc/NetworkManager/system-connections, /var/lib/systemd, /var/lib/nixos, /var/log, /etc/machine-id) are added regardless of impl. Other modules contribute their own paths (agent -> /var/lib/nixfleet, CP -> /var/lib/nixfleet-cp, secrets -> /etc/ssh/ssh_host_ed25519_key). The active impl reads the merged list.

trust - the four roots (contracts/trust.nix)

The most security-critical contract:

options.nixfleet.trust = {
  ciReleaseKey = mkOption { type = ciReleaseKeySlotType; ... };  # typed (algorithm + public)
  cacheKeys    = mkOption { type = listOf str; ... };            # opaque, for nix's trusted-public-keys
  orgRootKey   = mkOption { type = keySlotType; ... };           # bare-string ed25519 (pinned)
};

Three roots declared in the flake; the fourth root - the per-host SSH key - is intrinsic to each host (generated by stock OpenSSH on first boot). Each KeySlot has current, previous, rejectBefore. The ciReleaseKey slot is typed to support both ed25519 and ecdsa-p256 (TPMs commonly support P-256 but not ed25519). The orgRootKey is pinned to ed25519 - bootstrap-token signing only, never reaches the CP. cacheKeys is forwarded verbatim to nix.settings.trusted-public-keys. Serialised to JSON at build time (see _trust-json.nix below) and read at runtime.

10.4 Pluggable impls (flake.scopes.*)

The kernel/opinion split: framework declares contracts and ships one impl per family. Sibling impls are alternatives. Registered in modules/flake-module.nix:

flake.scopes = {
  persistence.impermanence = ../impls/persistence/impermanence.nix;
  keyslots.tpm             = ../impls/keyslots/tpm;
  gitops.forgejo           = import ../impls/gitops/forgejo.nix;
  gitops.gitea             = import ../impls/gitops/forgejo.nix;  # API identical
  secrets                  = ../impls/secrets;
};

• persistence.impermanence (impls/persistence/impermanence.nix) - btrfs-rootwipe-on-boot. initrd moves @root to old_roots/<timestamp>, creates fresh empty @root; upstream impermanence then bind-mounts paths from /persist/... back. Old snapshots pruned at default 30-day retention. Two impl-specific options: rootDevice, oldRootsRetentionDays.

• keyslots.tpm (impls/keyslots/tpm/) - first-boot TPM key generation, idempotent re-export after impermanence wipe. tpm2_createprimary + tpm2_evictcontrol to a persistent handle (default 0x81010001); exports public key to /var/lib/nixfleet-tpm-keyslot/; installs a tpm-sign shell wrapper. Configurable: handle, algorithm (default ecdsa-p256), exportPubkeyDir, signWrapperName. Does not handle disk encryption.

• gitops.forgejo / .gitea (impls/gitops/forgejo.nix) - pure data, a URL builder. Returns { artifactUrl; signatureUrl } for a Forgejo or Gitea host. Wire into services.nixfleet-control-plane.channelRefsSource.

• secrets (impls/secrets/default.nix) - backend-agnostic identity-path manager. Declares where decryption identities live (identityPaths.{hostKey, userKey, extra}); ensures the SSH host key exists at first boot; adds those paths to the persistence contract; computes resolvedIdentityPaths (read-only introspection hook). Does NOT wrap agenix / sops / vault - your fleet wires those itself.

Consumer pattern (mkFleet wraps the per-host mkHost call):

# fleet-repo/flake.nix
let fleet = nixfleet.lib.mkFleet {
  hosts.web-01 = {
    system = "x86_64-linux";
    channel = "stable";
    tags = [];
    nixosArgs = {
      hostSpec = { userName = "deploy"; rootSshKeys = [ "ssh-ed25519 ..." ]; };
      modules = [
        nixfleet.scopes.persistence.impermanence
        nixfleet.scopes.secrets
        nixfleet.scopes.keyslots.tpm
        ./hardware/web-01.nix
        ({ ... }: {
          services.nixfleet-agent = { enable = true; controlPlane.url = "https://cp.example.com:8080"; };
        })
      ];
    };
  };
  channels.stable = {
    rolloutPolicy = "all-at-once";
    signingIntervalMinutes = 60;
    freshnessWindow = 1440;
  };
  rolloutPolicies.all-at-once = {
    strategy = "all-at-once";
    waves = [{ selector.all = true; soakMinutes = 0; }];
  };
};
in { nixosConfigurations = fleet.nixosConfigurations; }

10.5 Runtime service modules (modules/scopes/nixfleet/)

All underscore-prefixed (skipped by import-tree) and explicitly imported by lib/mk-host.nix. Each defaults to enable = false.

_agent.nix - Linux agent service

Key options: enable, controlPlaneUrl, machineId, pollInterval (60s default), trustFile (materialised from nixfleet.trust), tls.{caCert, clientCert, clientKey}, bootstrapTokenFile, stateDir (/var/lib/nixfleet-agent), complianceGate.mode, package (escape hatch for harness/vendor). Activation: materialises trust.json via environment.etc; installs Type=simple, Restart=always, RestartSec=30, NoNewPrivileges=true; contributes /var/lib/nixfleet to nixfleet.persistence.directories.

_agent-darwin.nix - macOS agent

Same schema plus sshHostKeyFile (default /etc/ssh/ssh_host_ed25519_key) and tags (passed via NIXFLEET_TAGS env). Differences: launchd instead of systemd (KeepAlive, RunAtLoad, ThrottleInterval=10); 15-second sleep in ExecStart to defend two boot races (NTP not synced -> rustls cert “not yet valid”; agenix not yet decrypted -> cert files missing); launchctl kickstart -k in postActivation forces clean restart even on unchanged plist; environment.etc.<...>.text instead of .source because Darwin’s flake-source symlinks are unreliable.

_control-plane.nix - CP service

Richest module. Key options:

Long-running systemd service (Type=simple) with ProtectSystem=strict, PrivateTmp=true, etc. The CP does not use a systemd timer - it has its own internal 30-second reconcile loop. systemd.tmpfiles.rules auto-bootstraps observed.json to an empty skeleton on first deploy.

_cache.nix - binary-cache client

Trivial: declares services.nixfleet-cache.{cacheUrl, publicKey}; appends to nix.settings.substituters and nix.settings.trusted-public-keys. Format-agnostic.

_microvm-host.nix - microVM host wiring

Bridges, NAT, dnsmasq DHCP. Default bridge nixfleet-br0, 10.42.0.1/24. The microVMs themselves are defined by your fleet via upstream microvm.vms.

_operator.nix - workstation tools

Adds nixfleet-cli (nixfleet, with subcommands mint-token, derive-pubkey, mint-operator-cert) to environment.systemPackages. Optional orgRootKeyFile exposed via NIXFLEET_OPERATOR_ORG_ROOT_KEY. Crucially: the org root private key is encrypted to the operator user only; the CP never decrypts it (it only verifies token signatures with the public half declared in config.nixfleet.trust.orgRootKey.current).

_trust-json.nix - shared trust serialiser

Helper imported by _agent.nix, _control-plane.nix, _agent-darwin.nix. Builds the JSON payload for /etc/nixfleet/{agent,cp}/trust.json. schemaVersion = 1 is required per RFC-0010 §1.5 - binaries refuse to start on unknown versions.

Core glue (modules/core/)

_nixos.nix: flake-only nix.nixPath, experimental-features, hostName/timeZone/locale/keyMap/xkb from hostSpec, root SSH keys + hashed password file, imports contracts/trust.nix. _darwin.nix is even smaller - system.stateVersion, system.primaryUser, disables verifyNixPath, marks hostSpec.isDarwin = true. Core was deliberately trimmed to mechanism-only; everything else lives in scopes.

10. The Rust layer

11.1 Crate map

Eight crates. Three boundary (types, canonicalisation, decision engine); five binaries. Dependency direction: proto -> canonicalize -> reconciler -> consumers. No cross-deps among consumers.

                ┌─────────────────────────────────────────────┐
                │              nixfleet-proto                 │
                │  (boundary types: FleetResolved, wire,      │
                │   trust, revocations, rollout manifest)     │
                └────────────────────┬────────────────────────┘
                                     │
                  ┌──────────────────┼─────────────────┐
                  ▼                  ▼                 ▼
       ┌────────────────────┐   ┌────────────┐   ┌──────────────────┐
       │ nixfleet-          │   │ used by    │   │ used by          │
       │ canonicalize       │   │ everyone   │   │ everyone         │
       │ (JCS, RFC 8785)    │   └────────────┘   └──────────────────┘
       └─────────┬──────────┘
                 │
                 ▼
       ┌────────────────────┐
       │ nixfleet-          │
       │ reconciler         │
       │ (verify_artifact,  │
       │  reconcile fn,     │
       │  evidence verify)  │
       └─┬──────────────────┘
         │
   ┌─────┴──────┬──────────────┬──────────────┬──────────────┐
   ▼            ▼              ▼              ▼              ▼
┌──────┐   ┌────────┐   ┌──────────┐   ┌──────────┐   ┌────────────────┐
│agent │   │ control│   │ release  │   │  cli     │   │verify-artifact │
└──────┘   └────────┘   └──────────┘   └──────────┘   └────────────────┘
 per-host    Axum +       CI build      operator     offline auditor
 actuator    SQLite       pipeline      tools         tool

11.2 Boundary crates

nixfleet-proto - shared types

Canonical definitions for every artifact and message. Modules:

• fleet_resolved.rs - FleetResolved, Host, Channel, RolloutPolicy, Wave, DisruptionBudget, Edge, Meta, Compliance, HealthGate, OnHealthFailure enum.
• agent_wire.rs - CheckinRequest/Response, EvaluatedTarget, ConfirmRequest, ReportRequest, ReportEvent. Constant PROTOCOL_MAJOR_VERSION = 1 (header X-Nixfleet-Protocol).
• enroll_wire.rs - BootstrapToken, TokenClaims, EnrollRequest/Response, RenewRequest/Response.
• revocations.rs - Revocations, RevocationEntry.
• rollout_manifest.rs - RolloutManifest, HostWave, fleetResolvedHash (anchor against mix-and-match).
• trust.rs - TrustConfig, KeySlot, TrustedPubkey.
• compliance.rs + evidence_signing.rs - typed signed payloads for every evidence event.

Conventions: optional fields use Option<T> with #[serde(default)] but no skip_serializing_if - null is present, important for JCS byte stability across Nix -> Rust round-trips. No #[serde(deny_unknown_fields)] - contracts evolve additively. Object key sorting + deterministic number formatting is the canonicalize crate’s job, not serde’s.

nixfleet-canonicalize - JCS

Library + tiny binary. The library is one function:

#![allow(unused)]
fn main() {
pub fn canonicalize(input: &str) -> Result<String> {
    let value: serde_json::Value = serde_json::from_str(input)?;
    serde_jcs::to_string(&value)
}
}

Every signer and every verifier feeds artifacts through this. Pinned serde_jcs 0.2, single source of truth. The binary is cat-style for use in CI sign hooks and tests.

nixfleet-state-machine - pure per-host reducer

Per-host rollout reducer with no I/O, no clock reads, no allocations of side-effects (effects are returned as data). Single entry point: step(state, event, now, policy) -> Result<(state, Vec<Effect>), TransitionError>. The same crate runs on both sides — the agent’s runtime drives local state from worker output (Local* events); the CP-side mirror synthesises the same state from inbound wire AgentEvents (Remote* events). Dependency list is part of the safety contract (no tokio / reqwest / rusqlite); CI verifies via cargo tree. Spec lives in RFC-0005 §3 (the 6-state machine) and RFC-0006 §3 (functional-core / imperative-shell pattern); RFC-0008 adds a parallel rollout-level reducer in src/rollout/.

nixfleet-reconciler - pure decision engine

The brain of the control plane, but as a pure library. No I/O, no state, no side effects. Two main exports:

#![allow(unused)]
fn main() {
pub fn verify_artifact(
    artifact_bytes: &[u8],
    signature_bytes: &[u8],
    trusted_keys: &[&TrustedPubkey],
    now: DateTime<Utc>,
    freshness_window: Duration,
    reject_before: Option<DateTime<Utc>>,
) -> Result<FleetResolved, VerifyError>
}

Steps: parse -> re-canonicalise (assert byte-for-byte match) -> verify signature against each trusted key (ed25519 or ecdsa-p256, algorithm tag from meta.signatureAlgorithm) -> freshness check (now - meta.signedAt < freshness_window) -> reject_before check (compromise switch) -> schemaVersion == 1. Returns parsed FleetResolved or detailed VerifyError (10 variants). Same path is used for Revocations and RolloutManifest via the SignedSidecar trait. Rollout manifests get an extra step: recompute SHA-256(canonical(manifest)) and assert it equals the advertised rolloutId (content addressing).

#![allow(unused)]
fn main() {
pub fn reconcile(
    fleet: &FleetResolved,
    observed: &Observed,
    now: DateTime<Utc>,
) -> Vec<Action>
}

Inputs: verified fleet, Observed snapshot (channel refs, host states, active rollouts, compliance failures), current time. Output: a list of Actions (OpenRollout, DispatchHost, PromoteWave, ConvergeRollout, HaltRollout, SoakHost, ChannelUnknown, Skip, WaveBlocked).

Internal modules: host_state.rs (per-host shape; HostRolloutState itself now lives in nixfleet-state-machine per RFC-0005/0009 — reconciler + CP-side runtime consume that crate’s reducer), rollout_state.rs (RolloutState + advance_rollout()), budgets.rs (disruption budget enforcement - currently scaffolded), edges.rs (DAG ordering - reserved for future), verify.rs (verify_artifact, verify_rollout_manifest, verify_revocations, SignedSidecar trait, compute_canonical_hash), evidence.rs (verify_canonical_payload for host-signed compliance evidence using OpenSSH ed25519 pubkeys), manifest.rs (project_manifest, compute_rollout_id_for_channel).

11.3 Runtime binaries

nixfleet-agent - per-host actuator

Long-running daemon. Flags set by the NixOS module: --control-plane-url, --machine-id, --poll-interval, --trust-file, --ca-cert, --client-cert, --client-key, --bootstrap-token-file, --state-dir, --compliance-mode.

Main loop: load trust -> enrol if no cert + bootstrap token present -> build mTLS client -> run_boot_recovery() (handles fire-and-forget self-switch convergence) -> loop every poll_interval: POST /v1/agent/checkin; if response.target set, fetch + verify rollout manifest, pre-realise (nix-store --realise <closure> with cache_keys signature verify), activate (systemd-run --unit=nixfleet-switch -- switch-to-configuration switch on Linux, setsid -c on Darwin - both detached so they survive agent self-restart during NixOS reload), poll /run/current-system every 2s up to 300s, post-verify basename == expected, run compliance gate if enabled, POST /v1/agent/confirm, clear last_dispatched. On failure: POST /v1/agent/report with signed evidence. If cert TTL <50%: POST /v1/agent/renew.

Key modules: comms.rs (mTLS reqwest, 10s connect, 30s per-request), activation.rs (three-stage validation, fire-and-forget launch, lock coordination via /run/nixos/switch-to-configuration.lock, ActivationOutcome enum), enrollment.rs (CSR generation + enrol + 50% TTL renew), checkin_state.rs (last_confirmed_at + last_dispatched), compliance.rs (Pass / Failures / Skipped / GateError; auto mode -> Permissive if collector present, Disabled if absent), evidence_signer.rs (loads /etc/ssh/ssh_host_ed25519_key, JCS-canonicalises, ed25519-signs, base64), freshness.rs, manifest_cache.rs (content-address verification), recovery.rs (run_boot_recovery()), host_facts/ (Linux reads boot_id from /proc/sys/kernel/random/boot_id; Darwin uses hardware UUID).

What it never does: accept arbitrary commands (vocabulary is target = sha256-X); trust a CP-recommended closure without cache-key verification; hold long-lived credentials beyond 30-day mTLS cert + machine-lifetime SSH host key.

nixfleet-control-plane - Axum + SQLite + reconcile loop

Long-running HTTPS server. Two subcommands: serve and tick (one-shot, for tests).

Routes (under /v1/ with protocol-version middleware):

GET  /healthz                          -> { ok, version, last_tick_at }
GET  /v1/whoami                        -> { cn, issuedAt }
POST /v1/enroll                        -> 30-day cert from bootstrap token
POST /v1/agent/renew                   -> re-issue cert from existing mTLS identity
POST /v1/agent/checkin                 -> { target?, revocations? }
POST /v1/agent/confirm                 -> marks host_dispatch_state row confirmed
POST /v1/agent/report                  -> ingests telemetry events
GET  /v1/agent/closure/{hash}          -> proxies to binary cache (optional)
GET  /v1/channels/{name}               -> channel metadata
GET  /v1/hosts                         -> { hostname: { online, current_generation } }
GET  /v1/rollouts/{rolloutId}          -> manifest JSON (mTLS-gated)
GET  /v1/rollouts/{rolloutId}/sig      -> manifest signature bytes

mTLS enforced at TLS handshake when --client-ca set. Agent routes authenticate solely via verified client cert (CN matches request hostname). No admin routes in the open kernel - fine-grained operator RBAC is intentionally out of scope and belongs in a sibling commercial-extensions repository.

State:

• In-memory (RwLock): host_checkins: HashMap<hostname, HostCheckinRecord>, channel_refs: HashMap<channel, git_ref>, rollout manifest cache, last_tick_at.
• SQLite (/var/lib/nixfleet-cp/state.db, refinery-managed migrations):
  • token_replay (24h TTL) - soft state.
  • cert_revocations - hard state, replayed from signed revocations.json sidecar every reconcile tick.
  • host_dispatch_state (hostname PK, rollout_id, channel, wave, target_closure_hash, target_channel_ref, dispatched_at, confirm_deadline, confirmed_at, state ∈ {pending, confirmed, rolled-back, cancelled}) - operational, one row per host.
  • dispatch_history (id PK, hostname, rollout_id, channel, wave, target_closure_hash, target_channel_ref, dispatched_at, terminal_state ∈ {converged, rolled-back, cancelled}, terminal_at) - audit log; one row per dispatch event.
  • host_rollout_state (rollout_id, hostname, host_state, last_healthy_since, updated_at) - soak-window tracking, repopulated from agent-attested last_confirmed_at on rebuild.
  • host_reports (event_id, hostname, received_at, event_kind, rollout, signature_status, report_json) - telemetry.
• Filesystem: artifact_path, signature_path, observed_path.

Reconcile loop (every 30s) reads inputs, calls verify_artifact(), projects Observed from in-memory checkins + SQLite, calls reconcile(), processes the resulting Vec<Action> against SQLite (UPSERT host_dispatch_state + INSERT dispatch_history on dispatch, update host_rollout_state, etc.).

Background tasks: reconcile_loop (30s), channel_refs_poll (60s - full verify_artifact on fetched bytes, update in-memory map), revocations_poll (60s - same trust pipeline; replay into cert_revocations table on every tick), rollback_check_loop (10s - scan state='pending' AND confirm_deadline < now, mark rolled-back, stamp dispatch_history), prune_timer (delete old token_replay, archive old host_reports). All share a tokio::sync::CancellationToken plumbed from main; signal::ctrl_c() triggers axum_server::Handle::graceful_shutdown (25s drain) followed by cancellation fan-out; drain_background_tasks gathers JoinHandles with a 30s deadline.

On-demand HTTP source - rollouts_source: fetches a rollout manifest lazily when GET /v1/rollouts/<rolloutId> misses --rollouts-dir. URL templates with literal {rolloutId} token. Trust posture: the CP only checks RolloutId::new(manifest.channel, manifest.channel_ref) == rolloutId (the RFC-0008 §6.3 canonical-id discriminator). It does not verify the signature. The agent verifies the signature against ciReleaseKey on receipt. Even when forwarding a signed manifest, the CP never pretends to attest to it.

nixfleet-cli - operator workstation tools

An umbrella binary with operator subcommands. nixfleet mint-token reads the org root private key (32 raw bytes / hex / PEM PKCS#8), generates a nonce, builds TokenClaims, JCS-canonicalises, ed25519-signs, outputs the bootstrap-token JSON. nixfleet derive-pubkey reads a private key file and emits the base64 ed25519 pubkey - used once when bootstrapping the org root key. nixfleet mint-operator-cert mints a clientAuth-EKU X.509 cert signed by the offline fleet root for operator mTLS access.

There is no big “fleet management” CLI in the open kernel - operations happen through git commits and CI, not CLI commands.

nixfleet-release - CI release pipeline orchestrator

Most complex binary. Orchestrates build -> inject closureHash -> stamp meta -> canonicalise -> sign -> release:

1. Enumerate hosts (auto = all; auto:exclude=foo,bar; or explicit list).
2. Build closures per host.
3. Per-closure push (optional --push-cmd hook; env: NIXFLEET_HOST, NIXFLEET_PATH, NIXFLEET_CLOSURE_HASH).
4. Evaluate .#fleet.resolved.
5. Inject closureHash per built host.
6. Stamp meta (signedAt = now, ciCommit, signatureAlgorithm).
7. Canonicalise via nixfleet-canonicalize.
8. Sign via --sign-cmd hook (env: NIXFLEET_INPUT, NIXFLEET_OUTPUT).
9. Smoke verify (re-parse, canonical round-trip, structural check).
10. Project per-channel rollout manifests (rolloutId = SHA-256(canonical(manifest))); sign each.
11. Atomic write of releases/fleet.resolved.json{,.sig}, revocations.json{,.sig}, rollouts/<rolloutId>.json{,.sig}.
12. Optional git ops (stage, commit, push).

The hook contract is what makes signing pluggable: framework doesn’t care how you sign (TPM, HSM, YubiKey, KMS, software ed25519); it cares only that the hook reads canonical bytes from $NIXFLEET_INPUT and writes raw signature to $NIXFLEET_OUTPUT.

nixfleet-verify-artifact - offline auditor

Three subcommands (pure verification, no network): artifact (verify a fleet.resolved), rollout-manifest (verify a rollout manifest, asserts rolloutId equals the canonical {channel}@{channel_ref} per RFC-0008 §6.3), probe (verify a host-signed probe payload against an OpenSSH host pubkey). Given just signed artifacts plus trust roots, an auditor can verify the chain without ever touching the control plane.

11. Testing fabric

Three tiers, fastest-first.

Tier C - eval-only (~5-15s, every PR)

• nix fmt -- --ci - Alejandra + shfmt + deadnix.
• nix flake check --no-build - eval every output across the system matrix.
• mkFleet-eval-tests - 14 fixtures (7 positive + 7 negative) under tests/lib/mk-fleet/. Positive fixtures must produce expected .resolved.json golden files; negative fixtures must throw expected eval errors.
• _agent-v2-trust.nix, _cp-v2-trust.nix, _trust-options.nix - eval-only assertions on agent/CP module wire shape (ExecStart flags, trust.json schemaVersion = 1, etc.).

Tier B - Rust unit/integration (~15-30s, pre-push subset, full in CI)

• cargo nextest workspace-wide (currently ~560 tests). Concentration: nixfleet-control-plane (Axum endpoint integration with in-process mTLS, SQLite transactions, mTLS CN matching, V001-V006 migration tests, graceful-shutdown drain), nixfleet-reconciler (state-machine transitions, signature round-trips, cycle detection), nixfleet-proto (round-trip serialisation, trust config), nixfleet-canonicalize (JCS golden vectors, RFC 8785 Appendix E), nixfleet-release (sign-smoke roundtrip + adversarial verify), nixfleet-verify-artifact, nixfleet-agent (boot-recovery convergence + per-variant DispatchHandler unit tests).
• cargo clippy with -D warnings.

Tier A - microvm scenarios (minutes, nightly / on-demand)

Full integration via runNixOSTest hosting microvm.nix guests under one host VM (much faster than per-node QEMU). Linux x86_64 only (microvm.nix needs nested KVM). Scenarios under tests/harness/scenarios/, registered in modules/tests/harness.nix. Memory budget max(4096, 3072 + N×256); fits fleet-50 in 16 GB.

Real-binary harness nodes (tests/harness/nodes/cp-real.nix + agent-real.nix) consume services.nixfleet-control-plane.enable = true / services.nixfleet-agent.enable = true directly - the scenario surface is the operator surface. Stub nodes (cp.nix, agent.nix, cp-signed.nix, agent-verify.nix) keep their curl+jq scaffolding because they exercise routes the real CP doesn’t expose (e.g. GET / for fleet-N substrate scaling, GET /canonical.json{,.sig} for the offline-auditor contract).

CI workflows: .github/workflows/ci.yml - format job + validate job (nix run .#validate, default fast mode: format + flake eval + mkFleet-eval-tests + host builds for every nixosConfiguration). Pre-commit hook: format + real-SSH-key detector. Pre-push hook: format + mkFleet-eval-tests + cargo nextest run --workspace.

12. Glossary

13. How to read this codebase

1. Start with flake.nix - five lines of meaningful logic. Open lib/default.nix next, then lib/mk-host.nix. That’s the API surface.
2. Open contracts/host-spec.nix, contracts/persistence.nix, contracts/trust.nix - read each fully. Maybe 80 lines combined. They define the entire vocabulary.
3. Pick one runtime module (modules/scopes/nixfleet/_agent.nix is a good one) and read it with the corresponding crate’s src/main.rs open in the other window. See how the NixOS module’s ExecStart flags map to the crate’s CLI.
4. Read crates/nixfleet-proto/src/agent_wire.rs and crates/nixfleet-reconciler/src/verify.rs. The boundary contracts and the verification logic. Most of the design pressure sits here.
5. RFCs come last: RFC-0001 / 0002 / 0003 in order.

Verification is cheap:

nix flake check --no-build                            # full eval, ~5s
nix run .#validate                                    # default fast mode
nix run .#validate -- --rust                          # add cargo nextest + clippy
nix run .#validate -- --vm                            # add microvm scenarios (Linux only)
nix build .#nixosConfigurations.<host>.config.system.build.toplevel   # one host's closure

One-sentence summary

Git is truth; CI is the notary; attic is the content store; the control plane is a router; agents are the last line of defense; and every boundary artifact carries its own proof. Everything else is implementation.