IRIX 7 — Testing Strategy

Status: living document Principle: tests are the specification’s executable half. Benchmarks inform; they do not gate. Related: milestones.md, ../spec/system.md, hardware-lab.md, release.md.

0. Mesh gates (current)

The architecture adds a second test axis: the same logical computer at 1, 2, 8 and 128 nodes. These are hard gates, not benchmark notes.

The policy-level gates already run in crates/mesh-sim against the real mesh-service membership table and mesh-core rendezvous. The kernel now prints its single-node identity and initialises the same membership policy at boot; this is M1 boot evidence, not a mesh. A host integration test, crates/mesh-sim/tests/single_node.rs, joins the real membership table, namespace/CAS, actor directory/run queue and capability engine on one node and constructs no fabric frame. The loadable logical module runs the same policy on iris outside the compressed PROM; it prints its local root, actor and verified 1-of-1 capability, then ticks. The kernel now starts a local namespace/CAS, actor and capability policy at boot as well; the module remains the signed 1-of-1 probe. The slot-B Wasm module now runs the local policy from a Rust guest through the staged i7host surface and prints evidence matching the native m1: line; the fixed-region module multiplexer is implemented and its arena tick is boot-tested. Mesh has no transport; the kernel counters are self-tested and the M1 lines assert zero. This is a construction invariant, not a packet-level measurement of a future transport. The fabric, capability, CAS and Wasm host crates have host tests; the 128-node execution gate requires the native shim and transport work.

Architecture CI additions:

The first mesh-real rung is the M2 two-node rig in tools/m2-rig. It starts two independent iris processes, asserts BOOT-READY, distinct MACs, NodeIds and guest addresses, runs a two-way HTTP probe through the host relay, connects the two guest mesh listeners so both kernels exchange framed MeshEnvelope records, and drives the M2-2 signed join. Both nodes must converge on the same epoch and full member snapshot; the rig recomputes the snapshot digest independently, requires the member’s acknowledgement, restarts the member and requires it to rejoin at the same epoch. The no-peer and wrong-authority fault modes are negative evidence: the first preserves the M1 path with no listener, the second proves the member rejects an unauthenticated epoch. The kernel still prints mesh=single; that is the M1 bootstrap view, and the confirmed membership is the module’s mesh-join: evidence. M2-1’s framing, short read/write, deadline, heartbeat, backoff and tag-release tests live in crates/mesh-transport; the join codec, Ed25519 confirmation, replay/equivocation and restart rules live in crates/mesh-join; the home-lease, capability, MOVED and replay tests live in crates/mesh-route. The M2-4 rig drives a remote stat, a wire MOVED with a local retry and a local MOVED with a wire retry on both kernels, and no-route preserves the M2-1/M2-2/M2-3 markers. The M2-6 rig adds kill-peer and partition: the bounded suspect/failed/removed transition, the recomputed one-member homes, a fence row, a stale-epoch refusal of a replayed lease frame, a local commit without a quorum, and the isolated member’s no-authority commit with a read from its unexpired lease, followed by a rejoin at a new epoch. L0 migration remains M2-5; failure/partition fencing is M2-6. The rig is a manual command until the M2-8 CI job wraps it; the evidence bundle under target/m2-rig/evidence/ is the review artefact.

The fabric conformance tests are part of the canonical cargo test -p fabric -p mesh-route path. crates/fabric/tests/ exercises cancellation-versus-completion interleaving, a 10,000-tag in-flight table, and send-queue/stream-window exhaustion against a slow in-process consumer. crates/mesh-route/tests/trace_propagation.rs sets a W3C trace id on a request, carries the MeshEnvelope through mesh-transport one byte at a time, and asserts the same trace id on the home’s response and the sender’s completion, with a negative case for a mismatched response. The live MCP module keeps an 8-entry pending table (PENDING_CAPACITY in modules/mcp/src/mesh.rs); the 10,000-tag result is a fabric-model result and is not a claim that the live module holds 10,000 outstanding requests.

The independent endian and migration tests are listed in milestones.md and must include a big-endian MIPS → little-endian x86_64/AArch64 state hash.

1. Test pyramid

2. Test categories

Kernel

• Trap/exception vectors, TLB refill, cache maintenance, timer calibration.
• Frame/slab allocator invariants; huge-page promotion; OOM killer.
• Scheduler: EDF deadlines, fair share, priority inheritance, is_running.
• Capability rights: derivation, delegation, revocation, stale handles.
• IPC: synchronous channels, shared memory, cancellation, timeouts.
• Panic path: registers, stack, serial output.

Drivers

• ctl parser fuzzing, data boundaries, status stability.
• Lifecycle: start/stop/reset/remove ×100.
• IRQ event delivery and coalescing.
• DMA cache sync on iris and real hardware.
• Mock device for SCSI, Seeq, HAL2, Newport, Vino.
• Error injection: timeout, parity, disconnect, FIFO overrun.

Compatibility

• o32/n32 ELF loading, syscall decode, errno, signals, fork/exec/sproc.
• libc subset: write, read, open, malloc, printf, pthread (later).
• IRIX binary corpus: shell utilities, X11 clients, GL/dmedia samples.
• ABI conformance: register conventions, stack alignment, big-endian.
• Xlib/Xt/Motif subset; X Test Suite.
• XFS read-only: clean images, large files, sparse files, symlinks, corrupted images.

Native

• Cell loader: relocation, symbols, ABI version.
• Capability ABI: rights, revocation, delegation, fuzz.
• Native libc: open/read/write/mmap/fork/exec.
• Display protocol: golden frames, damage, cursor.
• 9P: interoperability with a Linux/Plan 9 client.

Network and MCP

• smoltcp: DHCP, DNS, TCP, IPv6, HTTP Range, Git smart HTTP.
• /net tree semantics.
• MCP: initialize, tools/list, tools/call, token rejection, rate limit, cancellation.
• Prompt-injection resilience is a client concern; test that tool results are data, not instructions.

Security

• Capability fuzzing; revocation races; stale handles.
• Fuzz: ELF, EROFS, XFS, 9P, HTTP, Git, MCP JSON-RPC.
• Secret scanning and licence checks in CI.
• No unsafe outside the framework/drivers without a written invariant.

3. Test corpus

Large binaries and images live in R2 or Git LFS, with a manifest and SHA-256 in the repo.

The kernel sash is exercised manually on both transports: telnet help, pwd, ls and cat /hello.txt, and the local keyboard/display after the target-2 image returns. The local path uses the same parser but writes through the console byte stream, so it works with net=off.

The current bring-up gates are the static std image under user/std-hello (args, time, sleep, std::fs on a tiny ramfs, chdir) and the external irix7/relibc n32 gate (TLS, setjmp, pthreads). Neither is a complete POSIX system; threads, spawn, signals, termios and sockets remain open.

The CI boot job builds the release kernel and ROM and installs the published IRIS release through irix7/iris/.github/actions/setup-iris. Its first boot attaches no SCSI disk at all: DHCP and the GitHub autoload path bring up the network, the target-2 loader fails cleanly, local and telnet sash come up, and the test runs packages against the GitHub index and checks the v0.1.3 MCP record. This is the primary diskless gate; a SCSI disk is not a boot, discovery or module prerequisite. Package installation from GitHub is exercised separately because its release/TLS path is not part of the M1 boot gate.

A slot-B boot serves a 1,347,640-byte wasm module over local HTTP, sets wasm.url and wasm.sha256 in NVRAM, and asserts wasm: host=1, wasm: run=42, m1-wasm: root=... actor=... cap=ok rights=35 home=..., m1: ... mesh=0/0/0, the post-smoke m1-post: mesh=0/0/0 line and BOOT-READY with MCP loaded. The checked-in v0.1.3 MCP release artifact has a known HTTP bug; a current-tree build answers when probed from a peer. A host-to-guest HTTP probe is not yet a CI gate. A second local boot uses wasm.local on SCSI target 2 with net=off, so the same boot asserts BOOT-READY network=off mcp=disabled and the Wasm policy evidence. The CI compares the m1-wasm rights and home against the native m1: line; the guest root and actor must differ and the guest actor is adopted for userland. The boot checks objects=9 actors=1 queued=1 caps=1. The guest drives the namespace, actor and capability choice; the host supplies only staged CAS/table primitives. A failed or trapped guest aborts the stage, so no partial policy is published. The image must stay within the 1,572,864-byte slot-B limit.

The no-network and slot-B policy boots are both part of the same boot CI job. The M1 software path is reproducible from a clean checkout on iris. No physical Indy was available in this workspace: the real-Indy execution gate is not claimed, and is recorded as the hardware limitation in §5. The policy guest is a boot-time proof, not yet a resident or multiplexed policy service.

Readiness and optional Wasm are measured separately. With the local MCP fixture, MCP-only boots reached BOOT-READY in 7.3–7.7 s. The optional slot-B Wasm probe starts after BOOT-READY; the instrumented run reports autoload: wasm took 2140 ms over HTTP and 1600 ms from the local SCSI cache, with wasm: runtime=0 ms, so fetch/copy (not the guest policy) dominate. The full-path load time is recorded as an observation under #214; it is not a gate, does not delay BOOT-READY, and the local cache is the preferred optional source. The M1 readiness budget does not include that optional probe, so readiness is not regressed by Wasm. The observation is not a claim that the Wasm policy is cheap to load on the R4600.

A second boot uses net=off, attaches no SCSI disk and has no local module. It asserts the kernel’s m1: policy line including mesh=0/0/0, the post-smoke m1-post: mesh=0/0/0 line, BOOT-READY with network=off and mcp=disabled, the expected target-2 load failure and the sash prompt. This is the M1 no-network, no-module boot.

The same job then runs the offline recovery image. It wraps user/hello as a 256 KiB SCSI target-2 image and appends the logical module to the same disk, writes an mcp.local NVRAM record, sets net=off, and waits for the module’s cap=ok line, BOOT-READY with network=off and mcp=local, hello from user, pipe=ok, winsize=100x40, the exit marker and logical: tick 3, then uploads the serial transcript. The action’s release build needs IRIS_NO_IDLE=1; docs/ci.md in the IRIS repository documents that limitation. The serial waits follow log order because the default iris-ci consumer cursor is forward-only (irix7/iris#110).

The boot job also builds the patched Rust std rlib and the static user/std-hello image, strips and pads it to target 2, and boots it with net=off. It asserts hello from std, file=, write=ok, chdir=ok, cwd=/etc, relative=Welcome to IRIX 7., stat_size=17, entry=hello.txt, dev_null=ok, exit 0 and the post-smoke m1-post: mesh=0/0/0 line. The same image is also booted without a Wasm module to exercise the native policy fallback. This is now a CI artifact rather than a local-only run.

The CI boot gate also builds user/o32-hello, rewrites its e_flags to EF_MIPS_ABI_O32, and boots it with the same local logical module and net=off. It checks hello from o32, a read of /hello.txt through o32 open/read/write/close, and exit 42. This is a synthetic loader and syscall-personality probe, not an IRIX 6.5 binary; the IRIX libc, rld, sproc, signals and full decoder remain future work.

4. CI

The CI must be reproducible from a clean checkout; no manual steps.

5. Emulator and hardware

• iris is the first test target; every demo must run there.
• Real Indy is the second target; the milestone is not done until it runs there (or the limitation is recorded). The M1 gate currently has no physical Indy IP24 or lab runner, so real SCC, Seeq DMA, SCSI, REX3 and DS1386 timing/cache behaviour is unvalidated; see #16.
• Differential testing: run the same test on iris and the real Indy; compare serial, screen and block hashes.
• Snapshots: iris snapshots make long tests resumable; record the snapshot hash.
• Hardware-in-the-loop: the indy self-hosted runner is described in hardware-lab.md.

6. Tools

7. Benchmarks

Tracked, never gated:

• cold and warm boot time on iris and Indy;
• block throughput (SCSI, R2 Range);
• X11 xterm start and keystroke latency;
• audio buffer underruns per minute;
• display damage repair time;
• MCP round-trip latency;
• build time for the kernel and the image.

Store results in docs/bench/YYYY-MM-DD.md or a Project field; compare over time.

8. Release gates

A release requires:

• all required CI jobs green;
• the milestone demo on iris;
• the milestone demo on a real Indy (or a recorded limitation);
• the compatibility corpus passes;
• the security and licence checks pass;
• the docs build and the install/upgrade path is tested;
• the SBOM and signatures are attached;
• the release notes are complete.

9. Test data management

• Small fixtures in the repo; large images in R2/LFS with a manifest.
• Every image has a SHA-256 and a provenance note.
• Do not commit IRIX binaries that cannot be redistributed; keep them in the private corpus and record hashes.
• Fuzz corpora are minimised before commit.

10. Ownership

An area without a test owner is a risk (see risks.md).