Implementation notes

This is maintainer reference, not a user guide (see the documentation index for that) and not a behavior spec (see openspec/specs/ for that — the requirements below are already covered there). It's the gotchas and cross-cutting rationale worth knowing before you touch the code, that don't fit either of those.

Gotchas

These are mistakes already made here; each was silent rather than loud, which is what makes them worth writing down.

The palette's two colour groups are not swappable. The brand colours (brandAccent, brandInk, brandInkDim) carry the same values as the web repo's --accent, --ink and --ink-2 tokens, written as hex and downsampled by lipgloss, and the accent is used only where nothing about a node is being reported — the title bar's product name and the selected panel's border. The state colours (ansiGreen and friends) are raw ANSI and report what an engine is doing: the resource bars and the health glyph draw from them. spinnerFrames is one cycle for the whole tool, shared by fleet deploy's progress lines and the dashboard's in-flight tiles.

The catalogue is embedded at build time. providers.yaml is //go:embed-ed, so a previously-built spinloop binary keeps applying the old catalogue no matter what the file says. Rebuild before testing any catalogue change, or you will "verify" a fix that is not in the binary you ran. (--providers/SPINLOOP_PROVIDERS reads a file at run time and sidesteps this.)

A preset section is not the whole preset. Preset.Select returns only the named [section]; the [*] defaults live separately in Preset.Global. Anything that consumes a section's Params directly — rather than going through Args/Command, which layer both — silently drops whatever the user put in [*]. That is usually the settings they consider obvious enough to write once, like ngl and jinja, so the failure surfaces later as a model running on CPU or refusing tool calls.

Match preset flags by canonical name. Preset keys have short aliases (ngl is n-gpu-layers, c is ctx-size, hf is hf-repo, a is alias). Comparing raw keys against a list of flag names therefore matches only the spelling you happened to think of. Use preset.CanonicalKey, which is exported for exactly this; Flags already dedupes layers by canonical name, last layer winning.

A preset dialect is not interchangeable. internal/preset parses any INI the same way, so an oMLX preset fed through llama.cpp's dialect parses cleanly and produces a wrong command: the alias table rewrites m to --model and c to --ctx-size, and the boolean table drops a key = 0 entirely. Nothing errors — the server just receives flags it does not accept, or silently loses a setting. The dialect always comes from the engine PROVIDER names, never from the file.

A busy engine does not answer its own metrics endpoint. llama.cpp serves /metrics from the same queue it serves inference from, so a scrape taken while a prompt is being processed waits for that prompt to finish — tens of seconds on a long context. /v1/metrics used to scrape inline, so the handler blocked for as long as the engine had work, and spinloop metrics (5s client timeout, against a handler whose scrape timeout was also 5s) could not win that race: the view went blank exactly when there was something worth watching, reporting the node as unreachable. The counters now come from the background sampler's last reading (engineSample), which the daemon takes every 15s regardless of who is asking — so the handler never waits on the engine, and staleness is bounded by the sample interval. Three things to preserve if you touch this: the cached scrape error is still reported, because silently omitting the token block is what once hid a scraper pointed at the wrong port; the sample is forgotten on start, so one engine's counters are never reported against the next; and the sampler retries at catchUpInterval (1s) until a reading lands, dropping to the full interval only afterwards — without that, a freshly started engine reports no counters for up to 15s, which is exactly the window someone watching a node they just started is looking at.

An exported-but-empty variable is a gap, not a choice. setEnvIfAbsent keys on the variable being present, so an OPENAI_BASE_URL= in the environment counts as set and suppresses the value routing meant to supply — leaving the agent pointed at nothing. setEnvIfBlank is the one to use for an address or a key, matching what harnessEnv already does for the remote endpoint's values; the distinction only shows up when something exports an empty string, which shells do more often than you would think.

A pre-warm that races the engine's faults cannot win. The cloud's model load is I/O-bound and the engine arrives first: it maps its weights and faults pages in as it copies them to the GPU, so from the first second of a load the volume serves its per-page faults, and a provisioned gp3 root hands out at most 4,000 IOPS × ~68 KB ≈ 260 MB/s whatever else wants the disk. A sequential pre-warm reader that starts with the engine spends its first seconds ahead of the faults, then goes flat — the daemon's read_bytes did exactly that on a live instance (~1.5 GB of real EBS reads in the first seconds, then nothing for the whole load), because the engine's faults consume the budget and its readahead turns every later "read" into a cache hit. And the shape's 32 GB of RAM cannot hold a ~30 GB model in the page cache at all, so even a finished pre-warm would only shift the faults to a different second. Live-checked 2026-08-23: pre-warm on cost double-reads and saved no time (the ~30 GB model loaded in ~115 s either way). The feature was removed after that check; the provisioned gp3 throughput and IOPS stay, because the S3 sync is the one reader whose limit is the volume's.

contextsize.Parse is decimal. 128k is 128000, not 131072 — a CONTEXT written that way is not the power-of-two window it looks like. It also overrides a preset's ctx-size (both in serve and in remote deploy), so the Spinloop, not the preset, decides the window whenever it states one.

up dispatches by directory and reuses both branches. cmd/spinloop/up.go routes a working-directory fleet.yaml to the fleet start path — runFleetDrive over the named nodes, or over every node when none are given, since a bare fleet start lists and does nothing — and everything else to runServe's own body, so up and serve resolve and word things identically by construction. The completion slot is the only CWD-dependent one: upSlot offers the fleet's node names where ./fleet.yaml parses, the Spinloop slot elsewhere, and nothing where a fleet file is present but unreadable — __complete never errors, whatever the directory holds.

A freshly issued access key is not instantly resolvable. STS lags the iam:CreateAccessKey response by seconds — on one account the store's verification needed six attempts before the key resolved, at roughly 8–10 s propagation. The verify probe in cmd/spinloop/remote_auth.go (verifyNewKey) retries only when the error message contains InvalidClientTokenId, on an exponential backoff from 1 s to a 16 s cap (a ~31 s window over six attempts), and the stderr line says which retry is running. Two things to preserve: the match is a string match because the pinned service/sts SDK version has no typed error for that code — if the SDK is upgraded, switch to the typed check — and the retry is exclusive to that code. Any other verification failure, a key that resolves to a different account or lacks a permission, fails at once and deletes the key it created; waiting it out would only delay the deletion. verifyProbeBackoff is a seam the tests zero.

An IAM user's inline policies are capped at 2,048 characters in aggregate. The control plane's seven functions each take a grantInvokeUrl pair — two actions, the auth-type conditions, the function's ARN — and with the log-reading, stack-discovery, pricing and self-service statements the document far exceeds that; the first deploy of the RemoteCliUser inline policy failed with ServiceLimitExceeded, which CDK does not warn about ahead of time. It is now a stack-owned AWS::IAM::ManagedPolicy (RemoteCliPolicy, 6,144 cap; the deployed document measures ~2 KB, so the grant list has room to grow). Keep it managed rather than re-inlining it, and keep the iam self-service ARN built from the AWS::Partition/AWS::AccountId pseudo parameters instead of the user's Arn: the policy attaches to that user, so referencing the user from inside it is a dependency cycle.

The file credential store's index is non-secret by design. OS keystores offer no way to list entries, so the file store — used where no keystore is reachable or SPINLOOP_REMOTE_KEYSTORE=file — keeps a plain-text index of the stored regions beside the 0600 per-region files under <config>/keystore/. The report (spinloop remote auth) reads the index, so a file added, removed or renamed by hand is reported wrong until the index matches; and a corrupt index is reported, not silently reset, because a report that misleads about what is stored misleads about which access keys exist on the AWS side.

The two local model caches are separate. A model llama-server downloaded sits in llama.cpp's cache ($LLAMA_CACHE, else the platform's user cache directory) as flat filenames; one fetched with the hub's tools sits in the Hugging Face hub cache ($HF_HUB_CACHE, else $HF_HOME/hub, else ~/.cache/huggingface/hub) as models--<owner>--<name>/snapshots/<sha>/ of symlinks into a content-addressed blob store. Neither tool looks in the other's, so a model already on the machine is "already on the machine" in only one of the two senses. spinloop hf therefore resolves both roots up front (hf.ResolveRoots) and checks both before touching the network; a cache-aware lookup that consults only one side re-downloads what is already there. The hub-cache shape has two more traps: a snapshot entry whose symlink dangles is an interrupted or half-finished download and must count as absent, and refs/<revision> holds a commit sha, so a revision name is only a snapshot once it has been resolved through refs/.

opencode run takes its working directory from the PWD variable, not its own cwd. opencode run with no directory flag sets the session's directory — and with it the directory every tool the agent runs — from process.env.PWD where that variable is set, falling back to the process's cwd only where it is absent (as of v1.18.30). The dispatcher gave the child the item's directory with cmd.Dir, but the environment it inherited carried the PWD of wherever the orchestrator was started, so the session and the agent's tools ran in the orchestrator's directory while the process's cwd sat in the item's: an item's dir: was silently ignored, and nothing failed. startChild therefore rewrites the child's PWD to the item's directory (resolved with filepath.Abs) before the exec, on top of cmd.Dir carrying it. The variable then holds the unresolved path — the /var/... form on macOS — which is why the stub agent's record takes the physical directory with pwd -P and the variable's own value with printenv PWD.

internal/daemon must not depend on a cloud package. What an engine should serve is inference.DeployConfig, in internal/inference — a leaf that imports only the standard library. Both the daemon and the cloud control plane speak it, so neither has to import the other to be described: a daemon is handed one over its control API, and internal/remote persists one against an environment. The dependency used to run the other way, internal/daemon importing internal/remote for the type and for a config-directory helper that only forwarded to internal/config.Dir, which made the package that knows nothing about AWS depend on the package that is nothing but AWS. Keep new shared vocabulary in internal/inference only where it describes an engine's workload and needs nothing of ours to express; anything cloud-shaped — IsInstanceType and the rest of the EC2 vocabulary — stays in internal/remote.

Dashboard (fleet_dashboard.go and friends)

A few Bubble Tea/lipgloss specifics that are easy to break by "simplifying":

• The tea.Program holds the model by pointer: Bubble Tea never reads a value model's Init back, so the first round's mutations (its deadline spend — real cloud calls, for remote environments) would be silently discarded.
• The tick reschedules itself whenever it fires; a one-shot tea.Tick without the reschedule leaves the board still after the second round. A second, faster chain (dashSpinTickMsg) runs only while an action is in flight, so the spinner and the elapsed time beside a verb advance; it stops on the first tick that finds nothing in flight.
• Every reading carries the time its own call returned (fleet.NodeResult.At, set in the fan-out), and the board draws a reading only when it was taken later than the one already on screen. Reads run concurrently and take differing times, so a reading can land after one taken later than it — including a round issued before an action finished and landing after it, which would otherwise repaint the node's pre-action state.
• A start reports a fleet.StartPhase — what it is doing, when that began, when the next attempt is due — rather than a line of text, and fleet.RenderPhase(phase, now) is the only place it becomes text. A wait therefore counts down and a boot counts up on every repaint, and a situation the start has moved on from cannot be left on the tile: each phase replaces the one before it. spinloop remote start renders the same phases to stderr, so the tile and the CLI cannot word one situation differently.
• One function, dashNodeView, produces both a panel's lines and its health tier, from the reading, the action, the current time, and how old a reading of that node may be. Nothing in it reads a clock, so every pairing of a start's phase against a reading can be enumerated in a test.
• A tile's first line is a header bar drawn in raw ANSI — the body is one plain string under a single lipgloss style, so per-character colour cannot be lipgloss's. The board's own title bar (dashTitleBar) uses lipgloss instead, and the two share one surface index (barSurface) because they are set through different mechanisms and would otherwise drift.
• A grid row joins the corresponding lines of the tiles it places, not the tile blocks — joining whole blocks glues the second tile's top border to the first tile's bottom border and shifts its body down a line.
• A tile's content is exactly the lines metrics prints for the node in the board's current format (renderStatBars/renderStatGauges/renderTokenLines are shared, not reimplemented), so the panel and metrics can never disagree on a number. The format is board-wide, in dashModel.gauge, toggled by g, opening in gauge; the tile draws the sparkline at dashBarLineW, chosen so a full row (label, glyphs, trailing percentage) fits the tile's width exactly.

Behavior (panel contents, refresh cadence, start/stop/abort semantics, the detail view) is specified in openspec/specs/fleet-client/spec.md.

The metrics history

• The bar format's data lives in the daemon, not the client: systemHistory (internal/daemon/history.go) is appended on each sampler tick while an engine runs, survives a stop, and clears on the next start alongside the counter baseline's sample.forget(). Every client — one-shot, watch, dashboard, the cloud relay — draws the same window from the one read.
• The history samples' JSON field names are one letter each (t/c/m/g, i/u/m) because the reply crosses SSM on the cloud relay, and SSM command output truncates at 4KB. The window is 10 minutes at the 15s cadence (40 samples) and is capped at historyLimit samples regardless of cadence — the catch-up ticks run at 1s, and an engine with no scrape target never leaves that cadence. If the window or the sample shape grows, cap the size in the daemon, not the client: a truncated reply is corrupt JSON, and parseDaemonMetrics turns that into "daemon unreachable" for the whole metrics call.

Adapter schema references

• opencode config schema: https://opencode.ai/docs/config/. The catalogue follows it: amazon-bedrock is the Bedrock provider id, custom providers (ollama, llamacpp, openai-compatible) carry an npm package plus options.baseURL. The key is written as opencode's {env:VAR} substitution rather than the resolved secret, so no secret lands on disk; spinloop harness open passes the keys it can resolve to the agent it launches, which is what makes a config spinloop wrote usable without exporting anything by hand.
• Pi custom-models schema: https://github.com/earendil-works/pi (packages/coding-agent/docs/models.md). api is one of openai-completions/openai-responses/anthropic-messages/google-generative-ai; apiKey supports $ENV_VAR interpolation. Not every provider maps to Pi — those without a pi: block (e.g. amazon-bedrock) error under the pi harness.
• lucinate connections store: https://github.com/lucinate-ai/lucinate. An OpenAI-compatible connection is {id, name, type: "openai", url, defaultModel}; the key comes from lucinate's secrets store or, when unset, the LUCINATE_OPENAI_API_KEY env var (which is how spinloop configures it — no secret on disk). Only providers with a lucinate: marker map; the rest (e.g. amazon-bedrock, the Vertex providers) error under the lucinate harness.