- Generated by
1.16.1
|
ra8-firmware 0.1.0
Bare-metal firmware for the Renesas RA8 family (RA8D2 / RA8P1)
|
Every machine this project runs on: what it is, what it does, how it relates to the others, and how to rebuild it from nothing.
This is the document to read when you have forgotten how any of it works. It is deliberately narrative rather than a role-by-role reference – the roles document themselves, and infra/README.md is the per-role index.
One command orients you:
The machines themselves are declared in infra/fleet.yml – one block per host, and everything downstream is derived from it. This document is the narrative; CI_FLEET.md is the runbook for changing the fleet: adding a host, retuning one, giving one quiet hours, removing one, and how instance counts are derived rather than guessed.
k3s-pve and the dev box are both guests on the same 10-core / 16-thread i5-12600K, and together they are allocated 28 vCPU on a 16-thread part.
That is roughly 1.75x oversubscribed before anything runs. In practice it is far worse under load: the k3s node alone has been measured at load average 68-110 against its own 16 vCPU while agents were simultaneously running gates in the dev box guest beside it.
This is the reason CI feels slow, and almost every "the runner timed out" investigation eventually lands back here. Two consequences follow, and both are already encoded in the roles rather than left as folklore:
If pve1's headroom ever genuinely improves, the number to re-check first is total vCPU commitment on the physical host – not any single guest's setting.
Proxmox. ssh pve, user pve-admin, passwordless sudo. Bridges: vmbr0 192.168.1.50/24 (LAN) and vmbr1 10.10.10.2/29 (a 2.5 Gb link to TrueNAS, MTU 9000). Storage is local (dir) plus local-lvm (lvmthin, ~1.8 TB).
Not codified. The guest definitions exist only as live config. This is the one remaining hole in the rebuild story – see section 5.
16 vCPU, 64 GB, 500 GB, Ubuntu 24.04, two PCIe devices passed through. Reachable as ssh k3s-pve from any control node: infra/fleet.yml declares its tailnet address (100.64.0.1, user ubuntu) and just infra::ssh_config generates the alias from it.
Runs:
12 vCPU, 24 GB, 360 GB, Debian 12, unprivileged LXC with nesting=1 (needed so just ci can run podman inside it). ssh dev.
This is where every agent runs gates. Its whole toolchain is now codified (just infra::apply dev), including two tools built from source because no Debian suite carries them at the pinned version:
Also: the pinned Arm GNU Toolchain and Unicorn, the pinned lint set, /etc/profile.d/ra8-ci.sh, the shared ccache at /var/cache/ccache-ra8, ~/ra8-ws agent workspaces, and two systemd user units – the shared CI status poller and the workspace reaper.
Never work directly in ~/ra8-firmware on this box. Use just workspace::new <name>; improvised checkouts have clobbered other work.
Plain (non-ARC) self-hosted runners in Docker containers, deployed by the ci_runner_docker role. Measured roughly 2x faster per job than a pve1 pod, which is the oversubscription in section 1 stated from the other direction.
Everything it writes lives on a pool dataset, so docker rm is non-destructive and a TrueNAS SCALE upgrade is survivable. The role refuses, by assertion, to put CI I/O on the appliance's known-degraded 100T pool.
It is the one host class with a real one-command teardown:
Windows desktop running the win-ci-* runner instances under WSL2, carrying the ra8-ci and ra8-win labels. Reachable only through the bench Pi, which infra/fleet.yml declares as its jump: – so any control node reaches it, not just the Mac:
The most powerful CPU in the estate. Because it answers ra8-ci, it absorbs load that would otherwise land on pve1.
Raspberry Pi 5, Ubuntu 24.04, ssh star. Everything physical hangs off it:
just infra::apply star provisions all of it. One package cannot be fetched unattended – Digilent WaveForms sits behind a click-through licence gate – so the role fails with download instructions rather than skipping, because a bench reporting success without libdwf would be a lie.
The GitHub workflow does not run on this Pi. Its dedicated dev-box listener connects as the role-owned ra8-hil account with a single pinned SSH identity. That account has no access to star's mode-0600 TAPO/OpenBao configuration and receives only the narrow privileged ip, uhubctl, and USB-authorisation operations required by the existing HIL scripts.
The bench lock is also provisioned by that role. Its SSH liveness bound is ClientAliveInterval 15 with ClientAliveCountMax 4 (about 60 seconds after a client disappears without closing its socket), and the advisory overrun reaper runs every five minutes. Verify the live deployment without touching firmware:
Provisioning must preserve the same lock contract as every HIL recipe: run just infra::check star first, confirm just hil::status reports FREE, then apply the role. The health-check path resets and halts the board; do not run a full apply while another actor holds the rig.
Two hard-switch segments share the FortiGate. The odd lan segment (10.0.40.0/24) is denied WAN access by policy 2; the even lan-even segment (10.0.41.0/24) NATs through wan1. The FortiGate supplies routing, DHCP, switching, and PoE; the MR18 bridges ra8-bench only into the odd segment.
Not Ansible – it is driven over the console cable from the bench Pi by infra/network/fg_bringup.py, which reads every credential from OpenBao. See infra/network/README.md.
The vault runs on the k3s node, reachable at the LAN NodePort :32200 (BAO_ADDR). Consumers authenticate with a read-only AppRole and a ~/.config/hil/openbao.env (mode 0600).
Names only, values never:
| Path | What reads it |
|---|---|
| ra8/ci-runner-pat | both CI runner roles, to register runners with GitHub |
| secret/hil/tapo | HIL smart-plug power control (just hil::tapo) |
| secret/ra8d2/bench-network | the FortiGate/AP bring-up – admin creds, PSKs, and the chassis serial |
The FortiGate chassis serial is treated as a credential, not an asset tag: FortiOS derives the console recovery password from it mechanically as bcpb<serial>. It lives in the vault with everything else and appears nowhere in this tree.
bao operator init emits the Shamir unseal keys and the root token once. They live in ~/.openbao/init.json on the k3s node at mode 0600, outside every checkout, and their loss is unrecoverable – that is what a Shamir seal means, and no backup of the vault's data substitutes for it.
Deployment is codified; initialising and unsealing are deliberately not, because both produce secrets and a playbook that handles a root token can log one. The openbao role deploys the vault, reports its seal state, and stops. The operator steps are scripts/secrets/README.md.
A Shamir-sealed OpenBao comes up sealed after every restart, by design. If something that reads a credential suddenly cannot, check just infra::status first – then /bin/bash -p scripts/secrets/openbao_unseal.sh.
Order matters, because each step is the next one's prerequisite.
| # | Rebuild | Command | Notes |
|---|---|---|---|
| 1 | the Proxmox host | – | manual, not codified (section 5) |
| 2 | VM 300 + CT 107 | – | manual, not codified (section 5) |
| 3 | k3s + helm + vault | just infra::apply k3s-pve k3s-node | then init + unseal by hand |
| 4 | the ARC runner pool | just infra::apply k3s-pve ci-runner | needs 3 |
| 5 | the dev box | just infra::apply dev | slow: two source builds |
| 6 | extra runner hosts | just infra::apply truenas / just infra::apply win-ci | NAS, gaming PC |
| 7 | the HIL bench | just infra::apply star | needs the board attached |
| 8 | the bench LAN | just infra::fortigate_bootstrap | from ssh star; guarded confirmation |
Where do you run these from? Any machine with ansible and a key the hosts accept. It used to be nowhere: every host was addressed by an ~/.ssh/config alias that existed on the Mac, which had no ansible, while the dev box had ansible and could resolve none of them (#526). infra/fleet.yml now carries each machine's real address, fleet.py builds every command from it, and just infra::ssh_config generates the friendly aliases – so becoming a control node is just setup-ansible; Ansible core comes from uv.lock and Galaxy collections come from infra/ansible/requirements.yml. just infra::doctor says which half you are missing.
The first run against a host takes far longer than later ones. just infra::apply dev compiles gcc and cppcheck from source; re-runs skip both once the pinned versions are present and cost a handful of version probes.
The dev box is also the trusted control node for ordinary-runner convergence. A system timer executes a root-owned, operator-promoted repository snapshot every six hours, with k3s-pve first and the persistent Docker/WSL consumers afterward. No GitHub Actions job receives the fleet SSH authority. Inspect it with just infra::reconcile-status; run the same decision path read-only with just infra::reconcile.
Being honest about this is the point of the section.
just infra::status used to show a pool of k3s-runner* runners registered and online – pre-ARC leftovers from before the repository was renamed, carrying the bare self-hosted,Linux,X64 labels.
They were load-bearing, not dead, which is why "confirm before removing" was the right instinct: docs-publish, fuzz-nightly and osv-scan were still scheduled against exactly those labels, so deregistering the pool would have left those workflows with no runner that could serve them. That was the unfinished half of the ARC migration, which moved the core workflows to ra8-ci but left the stragglers behind "until their tools are confirmed in the image" and never revisited the condition.
The condition was checked against a live pod, the stragglers were moved to ra8-ci, and only then was the pool retired – registrations deleted, units stopped and removed with their drop-in directories, and the runner installs reclaimed from disk. The per-workflow dependency evidence is in infra/README.md under "The legacy `k3s-runner-*` pool is retired".
The k3s node also hosts the owner's personal homelab: the media stack (jellyfin, *arr, immich, audiobookshelf), authentik, a hugo blog, ntfy, paperless, vaultwarden, minecraft, headscale, pihole, grafana/prometheus/loki.
None of it is this project's concern. It is listed here only so nobody later assumes it was forgotten, or "helpfully" folds it into these roles. It shares hardware with CI and nothing else – though it is worth remembering that it also shares that oversubscribed CPU, and its steady-state draw is part of the arithmetic in ci_runner's defaults.
1.16.1