ra8-firmware 0.1.0
Bare-metal firmware for the Renesas RA8 family (RA8D2 / RA8P1)
Loading...
Searching...
No Matches
check_fleet_declaration.py
Go to the documentation of this file.
1#!/usr/bin/env python3
2# SPDX-License-Identifier: MIT
3# Copyright (c) 2026 Brighton Sikarskie
4"""Gate: ``infra/fleet.yml`` describes a fleet that could actually be built.
5
6The declaration is the single registry of what machines this project runs on
7and how much of each one CI may use, so an error in it is an error in the
8estate. This checks three things a green Ansible run would not:
9
101. **The declaration is internally sound.** Every rule in
11 :func:`fleet_model.validate` -- classes and plays that exist, an address any
12 machine could reach the host at, capacity that fits the declared budget,
13 per-instance floors, a parseable quiet-hours window, and an instance count
14 that is either the sizing formula's or comes with a written reason. A number
15 nobody can re-derive is folklore, and a host addressed by an ssh alias is
16 reachable only from whichever laptop defines it (#526).
17
182. **Nothing tunes a host twice.** A committed ``host_vars`` file may not
19 re-declare a variable the declaration owns. Extra-vars beat ``host_vars``,
20 so a duplicate would not change behaviour -- it would leave a number in the
21 tree that looks authoritative, that somebody will edit, and that will have
22 no effect.
23
243. **The derived variables land somewhere real.** Every ``fleet_capacity_*``
25 and ``dev_slice_*`` name the mapping emits must exist in that role's
26 defaults. A mapping keyed on a spelling no role reads is the same defect as
27 a checker rule keyed on a string no macro produces: it matches nothing and
28 reports success forever.
29
304. **The names both halves must agree on do agree.** The model predicts the dev
31 slice's unit name (``fleet.py`` passes it to the capacity script) and the
32 role creates it. Two spellings would give the host a quiet-hours window that
33 freezes a slice nothing ever made -- a schedule that stands nothing down.
34
355. **No command the tooling builds needs an ssh alias.** Rule 1 checks the
36 INPUT; this checks the derivation, by walking the real ssh argv and the real
37 inventory line for every host and failing on any destination or ProxyJump
38 hop that is a bare label. A future ``-J <fleet name>`` would pass every
39 input rule and still only work on a machine that happened to define that
40 name -- which is the whole of #526, one layer down.
41
426. **Native HIL labels cannot drift.** A declared native listener names its
43 workflow, and every job in that workflow must request exactly
44 ``self-hosted`` plus the listener's declared custom labels. A workflow
45 cannot acquire a HIL label without being owned by one declaration.
46
477. **The cache-only HIL repair stays cache-only.** Its standalone playbook,
48 private inventory driver and isolated Justfile must match one exact
49 execution document. The path and identity are literals, and inventory
50 variables may not override the corresponding full-role safety defaults.
51
52``--selftest`` runs first in the gate and asserts each rule fires on a
53deliberately broken declaration and stays quiet on a legal one. Without it,
54"0 problems" is indistinguishable from "checked nothing".
55"""
56
57from __future__ import annotations
58
59import argparse
60import re
61import sys
62import tempfile
63from copy import deepcopy
64from pathlib import Path
65from typing import Any
66
67import yaml
68
69REPO_ROOT = Path(__file__).resolve().parents[2]
70sys.path.insert(0, str(REPO_ROOT))
71sys.path.insert(0, str(REPO_ROOT / "scripts" / "dev"))
72
73import fleet_model as fm # noqa: E402 -- sibling tool, path set immediately above
74import fleet_reach as fr # noqa: E402 -- ditto; the reachability half of the model
75import hil_cache_repair_rules as hctr # noqa: E402 -- checker helper beside this script
76
77# Roles whose variables the mapping derives, keyed by the prefix it emits them
78# under. Every emitted name must exist in the role's defaults, or the role
79# would never read it and whatever it configures would silently not happen.
80DERIVED_ROLES = {
81 "fleet_capacity_": "fleet_capacity",
82 "dev_slice_": "dev_slice",
83 "dev_box_hil_runner_": "dev_box",
84 "hil_bench_": "hil_bench",
85}
86
87HIL_SERVICE_TEMPLATE = "infra/ansible/roles/dev_box/templates/ra8-hil-runner.service.j2"
88HIL_SERVICE_REQUIRED = frozenset(
89 {
90 "[Service]",
91 "User={{ dev_box_hil_runner_user }}",
92 "Group={{ dev_box_hil_runner_group }}",
93 "WorkingDirectory={{ dev_box_hil_runner_root }}",
94 "EnvironmentFile={{ dev_box_hil_runner_env_file }}",
95 'Environment="HOME={{ dev_box_hil_runner_home }}"',
96 "ExecStart={{ dev_box_hil_runner_root }}/runsvc.sh",
97 "NoNewPrivileges=true",
98 "PrivateDevices=true",
99 "PrivateTmp=true",
100 "ProtectHome=true",
101 "ProtectSystem=full",
102 "ProtectControlGroups=true",
103 "ProtectKernelModules=true",
104 "ProtectKernelTunables=true",
105 "RestrictSUIDSGID=true",
106 }
107)
108HIL_SERVICE_SINGLETONS = (
109 "User=",
110 "Group=",
111 "WorkingDirectory=",
112 "EnvironmentFile=",
113 "ExecStart=",
114)
115JINJA_VARIABLE = re.compile(r"{{\s*([A-Za-z_][A-Za-z0-9_]*)\s*}}")
116LINT_PROVIDER_INPUTS = (HIL_SERVICE_TEMPLATE,)
117
118
119def _role_defaults(role: str) -> dict[str, Any]:
120 """Read one role's declared defaults.
121
122 Args:
123 role: Role directory name under ``infra/ansible/roles``.
124
125 Returns:
126 The parsed ``defaults/main.yml`` mapping.
127 """
128 path = fm.ANSIBLE_DIR / "roles" / role / "defaults" / "main.yml"
129 return yaml.safe_load(path.read_text(encoding="utf-8")) or {}
130
131
132def _check_derived_vars() -> list[str]:
133 """Every variable the mapping emits exists in the role that consumes it.
134
135 Returns:
136 One message per name the role would never read.
137 """
138 data = fm.load()
139 emitted: set[str] = set()
140 for name, host in data["hosts"].items():
141 emitted |= set(fm.role_vars(data, name, host))
142 problems = []
143 for prefix, role in DERIVED_ROLES.items():
144 declared = set(_role_defaults(role))
145 problems += [
146 f"fleet.py emits '{key}', which is in no {role} default -- the role "
147 "would never read it, so whatever it configures would silently not happen"
148 for key in sorted({k for k in emitted if k.startswith(prefix)} - declared)
149 ]
150 return problems
151
152
153def _check_shared_constants() -> list[str]:
154 """The names the model and a role BOTH have to know are the same name.
155
156 ``fleet_model`` predicts the dev slice's unit name so ``fleet.py`` can pass
157 it to the capacity script, and the ``dev_slice`` role creates it. Two
158 spellings would produce a quiet-hours timer that freezes a slice nothing
159 ever made -- a window that silently stands nothing down.
160
161 Returns:
162 One message per disagreement.
163 """
164 role_unit = _role_defaults("dev_slice").get("dev_slice_unit")
165 if role_unit == fm.DEV_SLICE_UNIT:
166 return []
167 return [
168 f"fleet_model.DEV_SLICE_UNIT is '{fm.DEV_SLICE_UNIT}' but the dev_slice role "
169 f"creates '{role_unit}'. fleet.py passes the first to the capacity script and "
170 "the role creates the second, so quiet hours would freeze a slice that does "
171 "not exist and dev work would keep the machine through the owner's window."
172 ]
173
174
175def _check_hil_service_template(
176 repo_root: Path = REPO_ROOT, declared_vars: set[str] | None = None
177) -> list[str]:
178 """Validate the exact privileged systemd/Jinja input owned by this gate."""
179 path = repo_root / HIL_SERVICE_TEMPLATE
180 try:
181 text = path.read_text(encoding="utf-8")
182 except (OSError, UnicodeError) as exc:
183 return [f"{HIL_SERVICE_TEMPLATE}: cannot read template: {exc}"]
184 lines = [line.strip() for line in text.splitlines() if line.strip()]
185 problems = [
186 f"{HIL_SERVICE_TEMPLATE}: missing required service contract {line!r}"
187 for line in sorted(HIL_SERVICE_REQUIRED - set(lines))
188 ]
189 problems.extend(
190 f"{HIL_SERVICE_TEMPLATE}: {prefix} must occur exactly once"
191 for prefix in HIL_SERVICE_SINGLETONS
192 if sum(line.startswith(prefix) for line in lines) != 1
193 )
194 declared = declared_vars if declared_vars is not None else set(_role_defaults("dev_box"))
195 unknown = sorted(set(JINJA_VARIABLE.findall(text)) - declared)
196 if unknown:
197 problems.append(f"{HIL_SERVICE_TEMPLATE}: undeclared Jinja variable(s): {unknown!r}")
198 if "TAPO" in text.upper():
199 problems.append(f"{HIL_SERVICE_TEMPLATE}: unrelated TAPO credentials must never be exposed")
200 return problems
201
202
203def _workflow_jobs(path: Path) -> tuple[dict[str, Any], str | None]:
204 """Load the job mapping from one GitHub Actions workflow.
205
206 Args:
207 path: Workflow YAML path.
208
209 Returns:
210 ``(jobs, error)`` with exactly one side populated.
211 """
212 try:
213 loaded = yaml.safe_load(path.read_text(encoding="utf-8"))
214 except (OSError, UnicodeError, yaml.YAMLError) as exc:
215 return {}, str(exc)
216 if not isinstance(loaded, dict) or not isinstance(loaded.get("jobs"), dict):
217 return {}, "workflow has no jobs mapping"
218 return loaded["jobs"], None
219
220
221def _runs_on_labels(job: object) -> list[str] | None:
222 """Return literal labels from one job's ``runs-on`` field.
223
224 Args:
225 job: Parsed job mapping.
226
227 Returns:
228 Literal label list, or None for a missing/dynamic/non-list field.
229 """
230 if not isinstance(job, dict):
231 return None
232 value = job.get("runs-on")
233 if isinstance(value, str):
234 return [value]
235 if isinstance(value, list) and all(isinstance(label, str) for label in value):
236 return value
237 return None
238
239
240def _check_claimed_hil_workflow(
241 host_name: str, workflow: str, expected: set[str], repo_root: Path
242) -> list[str]:
243 """Check every job in one fleet-owned HIL workflow.
244
245 Args:
246 host_name: Fleet host owning the listener.
247 workflow: Repository-relative workflow path.
248 expected: Exact literal label set every job must request.
249 repo_root: Repository root or selftest fixture.
250
251 Returns:
252 One message per missing, unreadable, dynamic or drifted workflow job.
253 """
254 path = repo_root / workflow
255 if not path.is_file():
256 return [f"{host_name}: declared HIL workflow '{workflow}' does not exist"]
257 jobs, error = _workflow_jobs(path)
258 if error is not None:
259 return [f"{workflow}: {error}"]
260 problems = []
261 for job_name, job in jobs.items():
262 actual = _runs_on_labels(job)
263 if actual is None:
264 problems.append(
265 f"{workflow}:{job_name}: runs-on is not a literal string/list, so its "
266 "HIL labels cannot be checked against infra/fleet.yml"
267 )
268 elif len(actual) != len(set(actual)) or set(actual) != expected:
269 problems.append(
270 f"{workflow}:{job_name}: runs-on {actual!r} does not exactly match "
271 f"declared labels {sorted(expected)!r}"
272 )
273 return problems
274
275
276def _check_unclaimed_hil_workflows(
277 repo_root: Path, claims: set[str], custom_labels: set[str]
278) -> list[str]:
279 """Reject a workflow using native HIL labels without fleet ownership.
280
281 Args:
282 repo_root: Repository root or selftest fixture.
283 claims: Workflow paths owned by native listener declarations.
284 custom_labels: Every custom native HIL label in the fleet.
285
286 Returns:
287 One message per unclaimed job using a native HIL label.
288 """
289 problems = []
290 workflows_dir = repo_root / ".github" / "workflows"
291 paths = sorted([*workflows_dir.glob("*.yml"), *workflows_dir.glob("*.yaml")])
292 for path in paths:
293 rel = path.relative_to(repo_root).as_posix()
294 if rel in claims:
295 continue
296 jobs, error = _workflow_jobs(path)
297 if error is not None:
298 continue
299 for job_name, job in jobs.items():
300 actual = _runs_on_labels(job)
301 overlap = set(actual or []) & custom_labels
302 if overlap:
303 problems.append(
304 f"{rel}:{job_name}: uses native HIL label(s) {sorted(overlap)!r} "
305 "but no hil_runner declaration owns this workflow"
306 )
307 return problems
308
309
310def _check_hil_workflows(data: dict[str, Any], repo_root: Path = REPO_ROOT) -> list[str]:
311 """Cross-check native HIL declarations against literal workflow labels.
312
313 Args:
314 data: Parsed fleet declaration.
315 repo_root: Repository root, overridden by the selftest fixture.
316
317 Returns:
318 One message per missing workflow, dynamic label set, label mismatch,
319 or undeclared workflow using a native HIL label.
320 """
321 problems = []
322 claims: set[str] = set()
323 all_custom_labels: set[str] = set()
324 for host_name, host in data["hosts"].items():
325 declared = host.get("hil_runner")
326 if not isinstance(declared, dict):
327 continue
328 workflow = declared.get("workflow")
329 labels = declared.get("labels")
330 if (
331 not isinstance(workflow, str)
332 or not isinstance(labels, list)
333 or any(not isinstance(label, str) for label in labels)
334 ):
335 continue
336 expected = {"self-hosted", *labels}
337 claims.add(workflow)
338 all_custom_labels.update(labels)
339 problems += _check_claimed_hil_workflow(host_name, workflow, expected, repo_root)
340 problems += _check_unclaimed_hil_workflows(repo_root, claims, all_custom_labels)
341 return problems
342
343
344def _is_literal(destination: str) -> bool:
345 """Whether an ssh destination is an address rather than a config alias.
346
347 Args:
348 destination: ``[user@]address`` as it would appear on an ssh command
349 line.
350
351 Returns:
352 True when it carries a dot or a colon, i.e. an IPv4/IPv6 literal or a
353 qualified name; False for a bare label, which only resolves through
354 somebody's ``~/.ssh/config``.
355 """
356 address = destination.rpartition("@")[2]
357 return "." in address or ":" in address
358
359
360def _inventory_destinations(entry: str) -> list[str]:
361 """Every host address one generated inventory line hands to Ansible.
362
363 Args:
364 entry: One line of the generated inventory.
365
366 Returns:
367 The ``ansible_host`` value plus every ``ProxyJump`` hop, empty for a
368 ``connection=local`` host, which Ansible never dials.
369 """
370 out = []
371 for field in ("ansible_host=", "-o ProxyJump="):
372 _, found, tail = entry.partition(field)
373 if not found:
374 continue
375 value = tail.split("'")[0].split()[0]
376 out += value.split(",")
377 return out
378
379
380def _check_derived_reach(data: dict[str, Any]) -> list[str]:
381 """No ssh command or inventory line the model builds names an alias.
382
383 Args:
384 data: The parsed declaration. The selftest hands it one whose
385 derivation is broken, so a detector that stopped matching cannot
386 report the real fleet clean forever.
387
388 Returns:
389 One message per derived destination that is a bare label.
390 """
391 problems = []
392 for name in data["hosts"]:
393 argv = fr.ssh_target(data, name)
394 derived = {
395 "the ssh command this tooling builds": [
396 argv[-1],
397 *fr.jump_chain(data, name),
398 ],
399 "the generated Ansible inventory": _inventory_destinations(
400 fm.inventory_entry(data, name)
401 ),
402 }
403 problems += [
404 f"{name}: {what} dials '{token}', a bare label rather than an address. It "
405 "would resolve only on a machine whose ~/.ssh/config happened to define it, "
406 "which is exactly the fault #526 removed -- one layer further down."
407 for what, tokens in derived.items()
408 for token in tokens
409 if not _is_literal(token)
410 ]
411 return problems
412
413
414def _good_hosts() -> dict[str, Any]:
415 """Return the minimal legal host mapping used by the selftest."""
416 return {
417 "builder": {
418 "class": "arc_k8s",
419 "connect": {"address": "10.0.0.3", "user": "builder"},
420 "provisions": ["ci-runner"],
421 "runners": {
422 "instances": 1,
423 "cpus": 4,
424 "memory_gb": 8,
425 "cpu_request": 1,
426 "memory_request_gb": 2,
427 "labels": ["ra8-ci"],
428 },
429 "budget": {"mode": "burst", "threads": 4, "memory_gb": 8},
430 },
431 "nas": {
432 "class": "docker_linux",
433 "connect": {"address": "10.0.0.2", "user": "deploy"},
434 "provisions": ["ci-runner-docker"],
435 "runners": {
436 "instances": 2,
437 "cpus": 4,
438 "memory_gb": 8,
439 "labels": ["ra8-ci"],
440 },
441 "budget": {"mode": "reserved", "threads": 8, "memory_gb": 16},
442 },
443 "dev": {
444 "class": "dev_box",
445 "connect": {"address": "10.0.0.4", "user": "developer"},
446 "provisions": ["dev-box"],
447 "hil_runner": {
448 "name": "dev-hil",
449 "repository": "https://github.com/example/firmware",
450 "labels": ["hil", "ra8d2"],
451 "workflow": ".github/workflows/hil.yml",
452 "bench": {"host": "bench", "aliases": ["bench.local"]},
453 },
454 },
455 "bench": {
456 "class": "hil_bench",
457 "connect": {"address": "10.0.0.9", "user": "pi"},
458 "provisions": ["hil-bench"],
459 "board_interface": {
460 "name": "eth0",
461 "mac": "02:00:00:00:00:09",
462 "sysfs_device": "/sys/devices/platform/bench-ethernet",
463 "phc_index": 0,
464 },
465 },
466 }
467
468
469def _good_declaration() -> dict[str, Any]:
470 """A minimal legal declaration for the selftest to mutate.
471
472 Returns:
473 A one-host fleet that satisfies every rule.
474 """
475 return {
476 "sizing": {"build_parallelism": 4, "memory_per_instance_gb": 8},
477 "runner_image": {
478 "source_host": "builder",
479 "image": "localhost/ra8-ci-runner:v2",
480 "archive": "/var/lib/runner/ra8-ci-runner.tar",
481 },
482 "hosts": _good_hosts(),
483 }
484
485
486# name -> a mutation that must produce at least one problem. Each is a rule
487# this gate claims to enforce; a rule with no row here is a rule nothing proves
488# still fires. Grouped into the three families the validator has, so a family
489# that stopped firing is attributable at a glance.
490def _mutations() -> dict[str, Any]:
491 """The broken declarations the selftest asserts are rejected.
492
493 Returns:
494 Rule name to a function that damages a good declaration.
495 """
496 return {
497 **_reach_mutations(),
498 **_capacity_mutations(),
499 **_runner_image_mutations(),
500 **_dev_slice_mutations(),
501 **_hil_listener_mutations(),
502 **_hil_interface_mutations(),
503 }
504
505
506def _runner_image_mutations() -> dict[str, Any]:
507 """Breakages in the canonical image producer declaration.
508
509 Returns:
510 Rule name to a function that damages a good declaration.
511 """
512 return {
513 "runner image source is not declared": lambda d: d["runner_image"].update(
514 source_host="missing"
515 ),
516 "runner image source does not build it": lambda d: d["runner_image"].update(
517 source_host="nas"
518 ),
519 "runner image ref is empty": lambda d: d["runner_image"].update(image=""),
520 "runner image archive is empty": lambda d: d["runner_image"].update(archive=""),
521 }
522
523
524def _hil_listener_mutations() -> dict[str, Any]:
525 """Return mutations of the listener-to-bench relationship."""
526 return {
527 "HIL listener on wrong class": lambda d: d["hosts"]["dev"].update(**{"class": "hil_bench"}),
528 "HIL listener with no name": lambda d: d["hosts"]["dev"]["hil_runner"].update(name=""),
529 "HIL listener with no labels": lambda d: d["hosts"]["dev"]["hil_runner"].update(labels=[]),
530 "HIL listener with a non-string label": lambda d: d["hosts"]["dev"]["hil_runner"].update(
531 labels=["hil", 8]
532 ),
533 "HIL listener with implicit label repeated": lambda d: d["hosts"]["dev"][
534 "hil_runner"
535 ].update(labels=["self-hosted", "hil"]),
536 "HIL listener with no workflow": lambda d: d["hosts"]["dev"]["hil_runner"].update(
537 workflow=""
538 ),
539 "HIL listener with unsafe workflow path": lambda d: d["hosts"]["dev"]["hil_runner"].update(
540 workflow="../hil.yml"
541 ),
542 "HIL listener with malformed repository": lambda d: d["hosts"]["dev"]["hil_runner"].update(
543 repository="owner/repo"
544 ),
545 "HIL listener with unknown bench": lambda d: d["hosts"]["dev"]["hil_runner"][
546 "bench"
547 ].update(host="missing"),
548 "HIL listener targeting non-bench host": lambda d: d["hosts"]["dev"]["hil_runner"][
549 "bench"
550 ].update(host="nas"),
551 "HIL listener with malformed bench aliases": lambda d: d["hosts"]["dev"]["hil_runner"][
552 "bench"
553 ].update(aliases="bench.local"),
554 "duplicate HIL registration name": lambda d: _duplicate_hil(
555 d, workflow=".github/workflows/hil-second.yml"
556 ),
557 "duplicate HIL workflow owner": lambda d: _duplicate_hil(d, name="dev-hil-second"),
558 }
559
560
561def _hil_interface_mutations() -> dict[str, Any]:
562 """Return mutations of the permanent board-interface identity."""
563 return {
564 "HIL bench with missing board interface": lambda d: d["hosts"]["bench"].pop(
565 "board_interface"
566 ),
567 "HIL bench with virtual board interface": lambda d: d["hosts"]["bench"].update(
568 board_interface={
569 "name": "eth0.42",
570 "mac": "02:00:00:00:00:09",
571 "sysfs_device": "/sys/devices/platform/bench-ethernet",
572 "phc_index": 0,
573 }
574 ),
575 "HIL bench with malformed permanent MAC": lambda d: d["hosts"]["bench"][
576 "board_interface"
577 ].update(mac="not-a-mac"),
578 "HIL bench with unsafe sysfs identity": lambda d: d["hosts"]["bench"][
579 "board_interface"
580 ].update(sysfs_device="/sys/devices/../escape"),
581 "HIL bench with invalid PHC identity": lambda d: d["hosts"]["bench"][
582 "board_interface"
583 ].update(phc_index=-1),
584 }
585
586
587def _duplicate_hil(data: dict[str, Any], **override: str) -> None:
588 """Add a second legal dev-box shape sharing one listener identity field.
589
590 Args:
591 data: Declaration being damaged.
592 override: Unique field used to leave exactly one duplicate behind.
593 """
594 duplicate = deepcopy(data["hosts"]["dev"])
595 duplicate["connect"]["address"] = "10.0.0.5"
596 duplicate["hil_runner"].update(override)
597 data["hosts"]["dev-second"] = duplicate
598
599
600def _reach_mutations() -> dict[str, Any]:
601 """Breakages in how a machine is declared and reached (#526).
602
603 Returns:
604 Rule name to a function that damages a good declaration.
605 """
606 return {
607 "unknown class": lambda d: d["hosts"]["nas"].update(class_="x") or _set(d, "class", "nope"),
608 "unknown play": lambda d: d["hosts"]["nas"].update(provisions=["not-a-play"]),
609 "no connect.address": lambda d: d["hosts"]["nas"]["connect"].clear(),
610 # THE regression guard. A bare label is an ~/.ssh/config alias, and a
611 # fleet addressed by aliases is drivable only from whichever machine
612 # defines them -- which is how truenas sat at half capacity with nothing
613 # able to converge it back.
614 "address is an ssh alias": lambda d: d["hosts"]["nas"]["connect"].update(address="nas"),
615 "address carries the login user": lambda d: d["hosts"]["nas"]["connect"].update(
616 address="deploy@10.0.0.2"
617 ),
618 "address with whitespace in it": lambda d: d["hosts"]["nas"]["connect"].update(
619 address="10.0.0.2 "
620 ),
621 "jump is not a declared host": lambda d: d["hosts"]["nas"]["connect"].update(
622 jump="bastion"
623 ),
624 "jump chain revisits a host": lambda d: d["hosts"]["nas"]["connect"].update(jump="nas"),
625 }
626
627
628def _capacity_mutations() -> dict[str, Any]:
629 """Breakages in what a host promises its runners, and when.
630
631 Returns:
632 Rule name to a function that damages a good declaration.
633 """
634 return {
635 "wrong budget mode": lambda d: d["hosts"]["nas"]["budget"].update(mode="burst"),
636 "capacity over budget": lambda d: d["hosts"]["nas"]["runners"].update(instances=4),
637 "instance under the CPU floor": lambda d: d["hosts"]["nas"]["runners"].update(cpus=2),
638 "instance under the memory floor": lambda d: d["hosts"]["nas"]["runners"].update(
639 memory_gb=4
640 ),
641 "unexplained instance count": lambda d: d["hosts"]["nas"]["runners"].update(instances=1),
642 "no labels": lambda d: d["hosts"]["nas"]["runners"].update(labels=[]),
643 "bad quiet window": lambda d: d["hosts"]["nas"].update(
644 quiet_hours={"window": "evening", "days": "Fri", "instances": 0}
645 ),
646 "bad quiet day": lambda d: d["hosts"]["nas"].update(
647 quiet_hours={"window": "18:00-23:00", "days": "Funday", "instances": 0}
648 ),
649 "quiet target is not a reduction": lambda d: d["hosts"]["nas"].update(
650 quiet_hours={"window": "18:00-23:00", "days": "Fri", "instances": 2}
651 ),
652 "capacity on a non-runner class": lambda d: d["hosts"].update(
653 {
654 "box": {
655 "class": "dev_box",
656 "connect": {"address": "10.0.0.3"},
657 "provisions": ["dev-box"],
658 "runners": {"instances": 1},
659 }
660 }
661 ),
662 "bad sizing constant": lambda d: d["sizing"].update(build_parallelism=0),
663 }
664
665
666def _dev_slice_mutations() -> dict[str, Any]:
667 """Breakages in the slice a runner host lends back to agents.
668
669 Returns:
670 Rule name to a function that damages a good declaration.
671 """
672 return {
673 "dev slice not weighted below CI": lambda d: _lend(d, cpu_weight=100),
674 "dev slice weight out of range": lambda d: _lend(d, cpu_weight=0),
675 "dev slice memory over what the runners leave": lambda d: _lend(d, memory_gb=8),
676 "dev slice swap the host does not have": lambda d: _lend(d, swap_gb=4),
677 "dev slice with no parallel bound": lambda d: _lend(d, max_jobs=0),
678 "dev slice missing a required key": lambda d: d["hosts"]["nas"].update(
679 dev_slice={"cpu_weight": 10, "memory_gb": 4}
680 ),
681 "dev slice on a class that runs none": lambda d: d["hosts"].update(
682 {
683 "box": {
684 "class": "dev_box",
685 "connect": {"address": "10.0.0.3"},
686 "provisions": ["dev-box"],
687 "dev_slice": {"cpu_weight": 10, "memory_gb": 4, "max_jobs": 4},
688 }
689 }
690 ),
691 }
692
693
694def _lend(data: dict[str, Any], **override: int) -> None:
695 """Give the selftest's host a dev slice, with one field made wrong.
696
697 The base block is legal on the fixture host -- 2 runners x 8 GB of a 16 GB
698 budget leaves nothing, so the memory field is what has to give: the fixture
699 lends 0 GB is not legal either, hence the budget bump. Each caller then
700 breaks exactly one field, so a rule that stopped firing is attributable.
701
702 Args:
703 data: The declaration being damaged.
704 override: The one field to set to an illegal value.
705 """
706 data["hosts"]["nas"]["budget"]["memory_gb"] = 20
707 data["hosts"]["nas"]["budget"]["swap_gb"] = 2
708 data["hosts"]["nas"]["sizing_note"] = "fixture: budget raised to leave room to lend"
709 slice_: dict[str, int] = {
710 "cpu_weight": 10,
711 "memory_gb": 4,
712 "swap_gb": 2,
713 "max_jobs": 4,
714 }
715 slice_.update(override)
716 data["hosts"]["nas"]["dev_slice"] = slice_
717
718
719def _set(data: dict[str, Any], key: str, value: object) -> None:
720 """Set a key on the selftest's single host.
721
722 Args:
723 data: The declaration being damaged.
724 key: Key to set.
725 value: Value to set it to. Deliberately ``object``: the point of a
726 mutation is to write something the schema does not expect.
727 """
728 data["hosts"]["nas"][key] = value
729
730
731def _jumped_declaration() -> dict[str, Any]:
732 """A legal two-host fleet where one machine is reached through the other.
733
734 Returns:
735 The good declaration plus a bench the NAS is reached through.
736 """
737 data = _good_declaration()
738 data["hosts"]["nas"]["connect"]["jump"] = "bench"
739 return data
740
741
742def _check_jump_resolves(host_vars_dir: Path) -> list[str]:
743 """A declared hop must reach the ssh command line as an ADDRESS.
744
745 The mutation table proves a bad hop is rejected; this proves a good one is
746 honoured, and honoured as a literal. ``-J bench`` would satisfy every input
747 rule and still only work on a machine that defined that alias -- the same
748 defect the addresses themselves had.
749
750 Args:
751 host_vars_dir: Empty fixture directory for the validator.
752
753 Returns:
754 One message per way the hop failed to reach the command line.
755 """
756 data = _jumped_declaration()
757 problems = [f" a legal ProxyJump was rejected: {p}" for p in fm.validate(data, host_vars_dir)]
758 argv = fr.ssh_target(data, "nas")
759 if "-J" not in argv:
760 problems.append(" a declared connect.jump produced no -J on the ssh command line")
761 elif argv[argv.index("-J") + 1] != "pi@10.0.0.9":
762 hop = argv[argv.index("-J") + 1]
763 problems.append(f" the ProxyJump hop is '{hop}', not the hop host's address")
764 if "ProxyJump=pi@10.0.0.9" not in fm.inventory_entry(data, "nas"):
765 problems.append(" the generated inventory does not hand Ansible the ProxyJump hop")
766 if "ProxyJump bench" not in fr.render_ssh_config(data):
767 problems.append(" the generated ssh config does not carry the hop")
768 return problems
769
770
771def _check_hil_selftest(root: Path) -> list[str]:
772 """Prove HIL workflow labels and declarations reject drift both ways."""
773 failures: list[str] = []
774 good_declaration = _good_declaration()
775 workflow = root / ".github" / "workflows" / "hil.yml"
776 workflow.parent.mkdir(parents=True)
777 workflow.write_text(
778 "---\nname: hil\non: workflow_dispatch\njobs:\n"
779 " hil-all:\n runs-on: [self-hosted, hil, ra8d2]\n steps: []\n",
780 encoding="utf-8",
781 )
782 if _check_hil_workflows(good_declaration, root):
783 failures.append(" a workflow matching its declared HIL labels was rejected")
784 workflow.write_text(
785 "---\nname: hil\non: workflow_dispatch\njobs:\n"
786 " hil-all:\n runs-on: [self-hosted, hil, wrong-board]\n steps: []\n",
787 encoding="utf-8",
788 )
789 if not _check_hil_workflows(good_declaration, root):
790 failures.append(" drift in a HIL workflow label was not reported")
791 workflow.write_text(
792 "---\nname: hil\non: workflow_dispatch\njobs:\n"
793 " hil-all:\n runs-on: [self-hosted, hil, ra8d2]\n steps: []\n",
794 encoding="utf-8",
795 )
796 declaration_drift = deepcopy(good_declaration)
797 declaration_drift["hosts"]["dev"]["hil_runner"]["labels"] = ["hil", "ra8p1"]
798 if not _check_hil_workflows(declaration_drift, root):
799 failures.append(" drift in a declared HIL label was not reported")
800 undeclared = workflow.with_name("undeclared.yml")
801 undeclared.write_text(workflow.read_text(encoding="utf-8"), encoding="utf-8")
802 if not _check_hil_workflows(good_declaration, root):
803 failures.append(" an undeclared workflow using HIL labels was not reported")
804 undeclared.unlink()
805 return failures
806
807
808def _check_hil_service_selftest(root: Path) -> list[str]:
809 """Prove the managed systemd template contract accepts and rejects."""
810 failures: list[str] = []
811 template = root / HIL_SERVICE_TEMPLATE
812 template.parent.mkdir(parents=True, exist_ok=True)
813 good = "\n".join(sorted(HIL_SERVICE_REQUIRED)) + "\n"
814 template.write_text(good, encoding="utf-8")
815 declared = set(JINJA_VARIABLE.findall(good))
816 if _check_hil_service_template(root, declared):
817 failures.append(" the hardened HIL systemd template was rejected")
818 template.write_text(good.replace("NoNewPrivileges=true\n", ""), encoding="utf-8")
819 if not _check_hil_service_template(root, declared):
820 failures.append(" a HIL systemd template missing its sandbox was accepted")
821 template.write_text(good + "Environment={{ undeclared_secret }}\n", encoding="utf-8")
822 if not _check_hil_service_template(root, declared):
823 failures.append(" an undeclared HIL service variable was accepted")
824 return failures
825
826
827def _selftest() -> int:
828 """Assert every rule fires on a broken fleet and none fires on a legal one.
829
830 Returns:
831 0 when the checker demonstrably still has teeth, 1 otherwise.
832 """
833 failures = []
834 with tempfile.TemporaryDirectory() as tmp:
835 empty = Path(tmp)
836 good_declaration = _good_declaration()
837 if fm.validate(good_declaration, host_vars_dir=empty):
838 failures.append(" a legal declaration was rejected")
839 failures += _check_hil_selftest(empty)
840 failures += _check_hil_service_selftest(empty)
841 failures += hctr.selftest(REPO_ROOT)
842 # A validator that rejected EVERY dev slice would pass every mutation
843 # below while making the feature unusable, so the legal shape is
844 # asserted too -- the same reason the legal declaration above is.
845 legal_lend = _good_declaration()
846 _lend(legal_lend)
847 if fm.validate(legal_lend, host_vars_dir=empty):
848 failures.append(" a legal dev_slice was rejected")
849 failures += _check_jump_resolves(empty)
850 if _check_derived_reach(_jumped_declaration()):
851 failures.append(" a fleet reachable only by address was reported unreachable")
852 # Both directions for the derivation check itself: an alias in the
853 # declaration must come out the far end as an alias on a command line,
854 # or the check is decoration.
855 aliased = _jumped_declaration()
856 aliased["hosts"]["bench"]["connect"]["address"] = "bench"
857 if not _check_derived_reach(aliased):
858 failures.append(" an ssh alias survived into a derived ssh command unreported")
859 for rule, damage in _mutations().items():
860 broken = deepcopy(_good_declaration())
861 damage(broken)
862 if not fm.validate(broken, host_vars_dir=empty):
863 failures.append(f" rule not enforced: {rule}")
864 good = _good_declaration()
865 (empty / "nas.yml").write_text("ci_runner_docker_cpus: '9'\n", encoding="utf-8")
866 if not fm.validate(good, host_vars_dir=empty):
867 failures.append(" a host_vars file re-declaring a fleet-owned knob was accepted")
868 failures.extend(_selftest_authority_errors())
869 if failures:
870 print("check_fleet_declaration selftest FAILED:", file=sys.stderr)
871 print("\n".join(failures), file=sys.stderr)
872 return 1
873 print(
874 f"selftest OK: {len(_mutations())} rules fire, a legal declaration and "
875 "the standalone cache-only HIL execution contract passes"
876 )
877 return 0
878
879
880def _selftest_authority_errors() -> list[str]:
881 """Return failures in lint-provider versus policy-ownership boundaries."""
882 failures = []
883 if LINT_PROVIDER_INPUTS != (HIL_SERVICE_TEMPLATE,):
884 failures.append(" --list-files no longer reports only its semantic template input")
885 if len(LINT_PROVIDER_INPUTS) != len(set(LINT_PROVIDER_INPUTS)):
886 failures.append(" --list-files reports duplicate semantic template inputs")
887 if len(hctr.policy_input_files(REPO_ROOT)) <= len(LINT_PROVIDER_INPUTS):
888 failures.append(" authored-file ownership census collapsed into lint provider inputs")
889 return failures
890
891
892def main(argv: list[str] | None = None) -> int:
893 """Entry point.
894
895 Args:
896 argv: Command line, defaulting to ``sys.argv[1:]``.
897
898 Returns:
899 0 when the declaration is sound, 1 otherwise.
900 """
901 parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
902 parser.add_argument("--selftest", action="store_true", help="prove the rules still fire")
903 parser.add_argument("--list-files", action="store_true", help="list exact template inputs")
904 args = parser.parse_args(argv)
905 if args.list_files:
906 print(*LINT_PROVIDER_INPUTS, sep="\n")
907 return 0
908 if args.selftest:
909 return _selftest()
910 try:
911 data = fm.load()
912 except fm.FleetError as exc:
913 print(f"check_fleet_declaration: {exc}", file=sys.stderr)
914 return 1
915 problems = (
916 fm.validate(data)
917 + _check_derived_vars()
918 + _check_shared_constants()
919 + _check_hil_service_template()
920 + hctr.check(REPO_ROOT, data)
921 + _check_derived_reach(data)
922 + _check_hil_workflows(data)
923 )
924 if problems:
925 print(f"infra/fleet.yml: {len(problems)} problem(s):", file=sys.stderr)
926 for problem in problems:
927 print(f" {problem}", file=sys.stderr)
928 return 1
929 runners = sum(int((h.get("runners") or {}).get("instances", 0)) for h in data["hosts"].values())
930 native_hil = sum(1 for host in data["hosts"].values() if host.get("hil_runner"))
931 print(
932 f"infra/fleet.yml OK: {len(data['hosts'])} host(s), {runners} capacity-managed "
933 f"runner instance(s), {native_hil} native HIL listener(s)"
934 )
935 return 0
936
937
938if __name__ == "__main__":
939 sys.exit(main())
void main(void)
The application entry point Reset_Handler hands control to.
Definition main.c:298