Test Pyramid

This page is the explicit test pyramid (issue #429, TEST-01): the answer to “where does this test go?” that a reviewer can get without asking. Every 0.6.x milestone assumes a layer exists to hold its tests; this page names the layers, the failure class each one owns, and how each maps onto CI path gating (.github/filters.yaml, issue #264).

The rule of the page is one sentence:

A test goes in the lowest layer that can catch its failure class — and a failure class with no owning layer is a gap to file, not a test to absorb into the nearest layer.

Coverage percentages are deliberately not the acceptance structure — see the anti-goal at the bottom. The diff-based coverage gate (issue #267) is a separate, orthogonal mechanism; this page is about which layer a test belongs to, not how much of a change is covered.

The layers at a glance

Layer	Responsible for	Must not be used for	Lives in	Runs via (local)	CI job / filter
Backend unit	Pure logic, no DB: parsing, temporal math, relationship-type inversion, cadence, validation	Persistence, GORM hooks, route wiring	*_test.go co-located per package	go test ./...	backend-tests legs + backend-checks / backend
DB/integration	Everything that touches the real migrated schema: hooks, flat-field derivation, delete semantics, ownership scoping, transactions, sync/backup/import services	Pure logic; JSON contract; format bytes	any *_test.go using dbtest.New or database.InitDB(t.TempDir())	go test ./...	backend-tests legs / backend
API contract	Route↔openapi.yaml drift, request-body binding, response DTO shapes, captured-response fixtures	Business logic, persistence, UI	backend/openapi*_test.go, testdata/contract-fixtures/	go test ./..., npx vitest run, ./gradlew testDebugUnitTest :app:testObtainiumDebugUnitTest	openapi (+ schemathesis.yml)
Import/export interop	vCard 3/4, JSContact, CSV, iCal/CalDAV against the RFC golden fixtures; round-trip semantics	Neutral-model logic, persistence	backend/{vcard3,vcard4,jscontact,correspondence}/, internal/rfctest/, docs/golden-fixtures/	go test ./...	backend-tests/backend-checks (fuzz) / backend
Migration	Schema evolution: up/down pairs, version/dirty tracking, data preservation, interrupted-migration recovery	Current-schema app behavior; deploy sequencing	backend/database/migrations/, backend/database/migrate_*_test.go	go test ./...	backend-tests (rest leg) / backend
Frontend unit	Component state/rendering, hooks, i18n key parity, contract-fixture parsing (vitest)	Real network, end-to-end flows	frontend/src/**/*.test.ts(x), frontend/viteConfig.test.ts	npx vitest run + npx tsc --noEmit	frontend job / frontend
Android unit/Robolectric	View models, editors, screens, network parsing, offline/local-DB logic	Emulator/device flows, real backend	android/**/src/test/	./gradlew testDebugUnitTest :app:testObtainiumDebugUnitTest :app:testFossDebugUnitTest	test job / android
E2E web	Complete user flows through the shipped artifact (image + compose + nginx + backend)	Anything reachable in a lower layer	frontend/e2e/	npx playwright test	e2e job / frontend+openapi+infra
E2E Android	Real app on emulator against the real backend; the Playwright analog	JVM-testable logic	android/app/src/androidTest/	:app:connectedObtainiumDebugAndroidTest (see README-developer.md)	android-e2e job / android+openapi+infra
Release/install smoke	Clean install from nothing; misconfiguration diagnostics; startup ordering	Anything presupposing a working install	backend/cmd/deploysmoke/, backend/config/startup_smoke_test.go, .github/workflows/deploy-smoke.yml	go test ./cmd/deploysmoke/... ./config/...; go run ./cmd/deploysmoke against a fresh docker compose up	deploy-smoke job / infra; also a release-tier gate run against the build-once candidate image (#1484, release-gates.md)
Post-publish smoke	The published GHCR image (by digest) boots and serves the same real workflow	A from-source build (that’s the row above); pre-publish gating (nothing here can block a tag already pushed; the pre-publish gates are the release battery, which tests the same digest that publishes – #1484)	docker-compose.published-smoke.yml, .github/workflows/docker-publish.yml (post-publish-smoke job)	MYCORRHIZAL_IMAGE=<ref>@<digest> docker compose -f docker-compose.published-smoke.yml up -d --wait; go run ./cmd/deploysmoke	post-publish-smoke job, every tag push (amd64 and arm64 matrix legs, issue #1485)
ARM64 Go suite	vet + go test (no -race, no coverage) of the rest and controllers legs natively on ubuntu-24.04-arm: architecture-sensitive code (build tags, atomics alignment, unsafe, seccomp) and main_test.go’s host-arch embedded handshake	Race detection and coverage (amd64 owns both); Android’s GOOS=android cross-compile (android-tests.yml’s embedded-server step)	.github/workflows/arm64-tests.yml	cd backend && go test $(go list ./... \| grep -vE 'mycorrhizal/(controllers\|services\|database\|internal/(schemafixture\|propertytest\|largedata\|contactgen))$') on an arm64 host	arm64-tests.yml (release-tier; rest leg per backend PR, controllers on main/nightly)
Performance/load	N+1/query-count regressions, benchmark bodies, concurrent-write smoke vs the deployed artifact, scale (planned)	Correctness (that’s the pyramid)	backend/**/benchmark tests, backend/cmd/loadsmoke	go test -bench . -benchtime=1x, go run ./cmd/loadsmoke	backend-checks + e2e loadsmoke step
Security/adversarial	BOLA/IDOR, spec fuzzing, DAST, static analysis — the vulnerability classes no pyramid layer is shaped to catch	—	their own workflows	per-workflow	schemathesis.yml, zap-dast.yml, codeql.yml, sast.yml, …
Context / agent tooling	Validity of the agent-context tree: AGENTS.md file set and aliases, budgets, skill contract, reference integrity, Suites, static path/command resolution, fact parity	Agent-behaviour scoring (separate row)	backend/internal/agentcontext/, backend/cmd/contextcheck/	cd backend && go run ./cmd/contextcheck	docs-citations (also the pre-commit hook)
Agent behaviour eval	One representative task per Tier 1 suite: the agent’s diff stays in scope, generated artifacts are regenerated, and the suite’s DoD commands pass; a stripped-context control records the delta	Merge/release gating (advisory; dispatch-only or the weekly off-peak collector)	testdata/agent-eval/, backend/internal/agenteval/, backend/cmd/agenteval/	cd backend && go run ./cmd/agenteval -task database-migration	agent-eval.yml (workflow_dispatch only) and agent-eval-collect.yml (weekly, Sunday 04:00 UTC off-peak); both upload the JSON/Markdown artifact

Detail and the hard-won traps for each layer follow.

Backend layers

Backend unit

• Responsible for pure computations with no database and no HTTP: vCard/CSV/iCal line encode and decode, date and timezone math, RelationshipEdge.Type inversion (relationship_type_registry.go), cadence and Activity.Qualifying() logic, name and sort-key computation, validation rules, i18n tokens.
• Must not be used for anything that touches the schema, GORM hooks / BeforeSave derivation, controller routing, or ownership scoping — those are DB/integration or contract concerns, and a unit test would silently miss them.
• Lives in *_test.go files co-located with each package (models/, vcard3/, vcard4/, jscontact/, correspondence/, contactmodel/).
• Runs via go test ./.... The same go test process also runs the DB/integration tests — the layer is a property of what the test touches, not which command runs it.

Time-dependent logic: the injected clock (issue #1494)

Expiry, lockout windows, reminder scheduling, retention cutoffs and day-boundary decisions are pinned with a fake clock, never a sleep. backend/internal/clock provides Clock (Now()), clock.System, and clock.Fake (NewFake, Set, Advance; concurrency-safe).

Layer	Production source	Test seam
Controllers, middleware handlers	clock.FromContext(c).Now() (routes.go installs clock.System{} via clockMiddleware)	router.Use(func(c){ clock.Install(c, fake); c.Next() }) before registering routes; clock.InstallIfAbsent keeps it
Services (free functions)	services.Now()	defer services.SetClock(fake)()
AccountRateLimiter / IPRateLimiter	their own clk field	limiter.SetClock(fake)
JWT exp/iat/nbf	middleware.ValidateTimeClaims(claims, now) (parsers use jwt.WithoutClaimsValidation())	the same fakes above

Write the boundary table for the area you touch: token valid at exp-1ns and invalid at exp; session expires_at and idle timeout at == and +1ns; a purge row at exactly the cutoff; midnight in the reminder zone, DST, leap day. Pick instants far from the real clock (the suites use 2030) so a call that bypasses the seam fails rather than passing by coincidence. Examples: middleware/clock_boundary_test.go, services/clock_test.go, controllers/clock_boundary_test.go, controllers/date_02_reminder_zone_test.go.

Enforcement. go run ./cmd/rawtimecheck ./... (analyzer internal/lint/rawtime; a step in the required Backend (Go) job and the pre-commit hook) fails on any time.Now/time.Since/time.Until in controllers, services or middleware. Three ways out, in order of preference: migrate the site to the clock; for an elapsed-duration measurement, a net.Conn deadline or a PRNG seed, put // rawtime:allow <reason> on the line (a marker with no reason is itself a finding, like # pragma: no cover); for a bulk that cannot move in the same change, add a rawtime.Allowlist entry with a Reason and an exact Count — TestAllowlistIsExact fails when the count is stale in either direction, so the list can only shrink. Hand-verify the gate by adding a raw time.Now() to a controller and watching it fail. Real sleeps that remain in tests are for waiting on a different goroutine/process (e.g. asserting a background webhook did not fire), not for waiting out a time-based rule.

DB/integration

• Responsible for everything that persists to or reads from the real migrated schema:
  • GORM hooks and BeforeSave/BeforeCreate derivation, and the RecordForContact-vs-RecordFromContact / contact_card_merge.go data-loss bug classes (backend/AGENTS.md traps #2 and #3);
  • soft- vs hard-delete semantics and the DeleteContact/DeleteUser cascade lists (trap #6/#7), including unique-index interactions with soft-deleted rows;
  • ownership enforced at the data-access boundary — a missing or inconsistent check is a security defect (trap #5);
  • transactions, SQLITE_BUSY / _txlock=immediate behavior (database/concurrent_write_test.go, trap #9);
  • services that orchestrate multi-row operations: CardDAV/CalDAV sync, backup/restore (database/backup_test.go), import.
  • mid-request database failure (issue #1476): the DB-fault sweep, below.
• Must not be used for pure logic (unit), the JSON wire contract (API contract), or format bytes (interop).
• The DB-fault sweep (routes/db_fault_sweep_test.go, issue #1476) reaches the if err := tx.X().Error; err != nil { return ErrDatabase(...) } arms no HTTP test can trigger. dbtest.New(t, dbtest.WithFaults()) registers the internal/dbfault GORM callback plugin (idle until armed, test-only); for each route in faultScenarios() the sweep records the K statements the request issues on its own goroutine (fire-and-forget audit/webhook goroutines are deliberately invisible to the injector), then re-runs on a fresh identical database K times failing statement i. Every run must produce a well-formed 4xx/5xx envelope that leaks no SQL (marker, sqlite, SQL, table names), and leave every table’s row count + checksum byte-identical to before the request — a mid-request failure that leaves a partial write fails the sweep. An outcome that is legitimately best-effort (a post-commit response read, the audit snapshot, per-row import errors) must be declared in faultAllowlist with a written reason; an entry that no longer matches fails as stale, and an allowlisted 2xx body is still leak-checked. Statements the auth middleware issues first are covered by TestDBFaultSweep_AuthMiddlewareFailure. Failing COMMIT/BEGIN is out of scope (they are not GORM statements). Adding a mutating route that writes more than one row means adding a scenario; a # pragma: no cover — DB failure only on a branch the sweep reaches should be removed (the sweep already removed those in user_delete_cascade.go). Hand-verify: removing the db.Transaction wrapper from DeleteContact, CompleteReminder or ChangePassword fails the sweep. The sweep’s first run found and fixed two partial-write bugs (reminder completion, password-change revocation) and a SQL-text leak in import row errors.
  • Read routes (TestDBFaultReadSweep, routes/db_fault_read_sweep_test.go, issue #1497): the same injection for every GET route in faultReadRoutes. Reads never mutate, so one seeded database serves every injected statement. For each statement after the auth middleware’s own prefix the response must be the route’s normal 2xx (a declared best-effort enrichment read) or a well-formed 5xx envelope with a documented server-fault code, must leak no SQL or table name, and across the whole route at least one failure must surface as a 5xx (a handler that swallowed every database error would answer an outage with fabricated empty data). Integration routes that need an external system are not listed; their own controller tests cover them.
  • CRUD write scenarios (TestDBFaultSweepCRUD, routes/db_fault_crud_scenarios_test.go, issue #1497): 49 further faultScenario entries (notes, activities, circles, households, tags, field definitions, life events, gifts, obligations, events, cadence, data-decay, webhooks, external identities, edges) run through the same write sweep, with reasoned faultAllowlist entries for best-effort audit reads. Post-commit reads never turn a committed write into a 5xx (#1663): cadence create/update and activity update fall back to the committed entity when the follow-up read fails, so the allowlist’s old “recorded post-commit 5xx” class is gone and those routes assert a 2xx and that the write persisted. The sweep also found a real bug: CreateActivity appended its contact links in a separate statement from the insert, so a failed association left an orphan activity a client retry would duplicate; both writes now share one transaction.
• The handler guard matrix (TestHandlerGuardMatrix, controllers/handler_guard_matrix_test.go, issue #1497) invokes every route-bound handler directly on a bare gin context in two states — no authenticated user (must answer 401 UNAUTHORIZED and never touch the database) and an authenticated user with no validated body, path id or query (must answer its own typed 4xx envelope, or a well-formed success for a handler that takes no input; never a panic or a bare error) — and pins "<status> <code>" per handler in guardExpect, so a handler that drops its auth, validation or path-id guard fails by name. The table is hand-listed (guardHandlers and guardExpect must have the same length), not derived from the router: a new handler is covered only once someone adds it, which is why routes/authorization_matrix_test.go (derived from the live router) stays the route-completeness gate and this matrix is the per-handler guard pin.
• The non-negotiable fixture rule (backend/AGENTS.md trap #1): test against the real migrated schema, never AutoMigrate. Prefer internal/dbtest.New(t), which builds the migrated template once per test binary and hands each test an isolated copy (issue #632); the database package’s own tests call database.InitDB(filepath.Join(t.TempDir(), "x.db")) directly because dbtest cannot import it (deliberate import cycle).
• Lives in any *_test.go that takes a *gorm.DB — controllers/, services/, models/, database/.
• Runs via go test ./...; CI backend-tests matrix (controllers / services / rest legs, gotestsum retry-on-failure + a separate no-rerun coverage pass).

API contract

• Responsible for the boundary between the OpenAPI spec and the implementation:
  • route-vs-spec drift and schema presence (backend/openapi_test.go);
  • request-body binding shapes and whether validation tags actually run (openapi_request_test.go, issue #256);
  • response DTOs not silently dropping required keys — the omitempty + required TS field crash (frontend/AGENTS.md trap #8);
  • captured real responses, one canonical copy consumed by both web and Android parse tests: testdata/contract-fixtures/ → frontend/src/api/contractFixtures.test.ts and android/core/network/.../ContractFixtureTest.kt (issue #257; the spec-derived shared fixture is issue #266);
  • property-based spec fuzzing of the running app (Schemathesis + cmd/schemagate, issue #369) and the deterministic cross-account BOLA sweep (cmd/bolacheck);
  • design consistency of the spec itself — operation tags, unique operationIds, the error envelope, unused components — via backend/vacuum.yaml in the Backend checks job (OpenAPI linting, issue #1909).
• Must not be used for business logic, persistence, or UI behavior.
• Runs via go test ./... (drift tests), npx vitest run + ./gradlew testDebugUnitTest plus :app’s flavor-qualified tests (fixture consumers), and the schemathesis.yml workflow (fuzz). The live counterpart is frontend/e2e/apiContract.spec.ts.

Import/export interop

• Responsible for the formats crossing the boundary: vCard 3.0/4.0, JSContact, CSV, iCal/CalDAV.
  • Directional correctness against the RFC-verbatim golden fixtures (ADR-0003): docs/golden-fixtures/ is the external test oracle; adapters copy fixtures into backend/internal/rfctest/fixtures/ unchanged and tests assert against the RFC bytes. A green test that shares a misconception with its code proves nothing — the fixtures are what anchor the bytes.
  • Round-trip semantics, not byte identity. A round trip canonical → format → canonical must preserve meaning; repeated conversions must not progressively corrupt data. This is the milestone’s “distinguish semantic equivalence from byte-for-byte representation.”
  • Differential testing against independent reference implementations (TEST-08, issue #680): every corpus contact is also pushed through an implementation that shares no code with ours — Python vobject (vCard 3.0/4.0), Rust calcard (JSContact), Go golang-ical (iCalendar) — and the round trip must stay semantically equivalent per TEST-03’s comparator. This is the countermeasure to the round-trip suite’s own blind spot (code and tests sharing one wrong reading of the spec). Known reference-side divergences are pinned per corpus entry with drift detection; a confirmed our-side bug is fixed, never pinned (it caught the CATEGORIES comma-list escaping bug and the RFC 9553 created/updated wire-shape bug, both fixed). Details: the “Differential suite” section below.
  • Lossy/unknown-field behavior, sensitivity filtering in the query, malformed input reject/ignore/preserve rules.
  • Hostile-input fuzzing seeded from the fixtures (issues #265/#376; extended to the importer paths and client-controlled request parsers by #1625/#1626).
• Must not be used for the neutral model’s own logic (unit) or persistence (DB/integration).
• Runs via go test ./...; the 21 native fuzz targets (.github/scripts/run-fuzz-targets.sh) run as a short 15s-per-target smoke in backend-checks on PRs/pushes, and on the nightly schedule sharded by package across the fuzz-nightly matrix job at 2m per target (one go test process per target is the constraint, so the shard keeps them within the job budget). backend/fuzz_registration_test.go pins every func Fuzz* to a runner entry and the runner’s packages to the matrix.

Migration

• Responsible for schema evolution:
  • hand-written up/down pairs, append-only numbering, version tracking and the dirty flag (schema_migrations) — see migrate_version_test.go;
  • data preservation across migrations: real production data exists since v0.2.0-alpha-candidate (deployed 2026-08-04), so a rename/drop/retype needs a backfill, not a silent clean removal (migrate_datapreservation_test.go);
  • interrupted-migration behavior and failure diagnostics (v0.6.4, #436/#437/#438).
  • Historical migration paths (MIG-01 #436 + MIG-02 #437, v0.6.4): one schema-only dump per supported release (floor v1.0.0, issue #529; raised from v0.6.0 at the 1.0 major, issue #1170 — pre-v1.0.0 dumps remain as frozen historical artifacts), populated at test time from the TEST-02 manifest; the CI migration-tests workflow then matrixes one job per release through database.InitDB (v1.0.0 → current as the longest skip), round-trips every migration up → down → up against a populated fixture, and gates on every migration shipping its .down.sql. A release without a dump fails CI.
  • Real-release data (issue #1489): internal/realrelease + cmd/realrelease + scripts/realrelease-leg.sh boot the published image of each supported release, seed it through its public API, upgrade the data it wrote with the current boot path, compare the API read-back, run doctor/audit-chain, then roll back to the pre-migration backup with the old image. The real-release-upgrade job in migration-tests.yml runs it per SupportedReleases entry (main/nightly/dispatch; needs the network). The release-registration commit’s push skips the just-cut tag (its image isn’t published yet, .github/scripts/realrelease-image-expected.sh); that release’s leg runs from docker-publish.yml’s real-release-leg job after publication (issue #1713). See docs/upgrade-compatibility.md#how-upgrades-are-tested.
• Must not be used for current-schema application behavior (DB/integration) or deploy sequencing (release/install smoke).
• Runs via go test ./... (the database package lands in the rest leg) plus the migration-tests.yml matrix job (gated on the backend filter).

Frontend layer

Frontend unit (vitest)

• Responsible for component state and rendering, hooks/data-fetching logic, i18n key parity across all five locales (src/i18n/locales.test.ts), date-format providers, and contract-fixture parsing. Network is mocked; no browser. The src/api/* modules are tested at the request level with msw (src/test/mswServer.ts, issue #1482): a call runs through the real apiFetch and fetch, and the test asserts the request that went on the wire (method, URL including query string, headers, JSON/multipart body) and that a 4xx backend envelope maps to the module’s typed error. Use setupMswServer() / mockApi() / errorEnvelope() there; multipart tests also call setupNativeMultipart() (jsdom’s FormData cannot be serialized by Node’s fetch). src/api/requestContracts.test.ts is the table-driven home for thin modules (add a row per new endpoint); modules with richer logic get their own file. Component editors assert the save payload passed to onChange, not just rendering.
• Must not be used for end-to-end flows (that’s Playwright).
• Gotchas that live here (frontend/AGENTS.md traps): no auto-cleanup and no globals: true — add afterEach(cleanup) explicitly; MUI appends " *" to a required field’s accessible label; don’t nest <Chip> in <Typography>.
• Runs via npx vitest run + npx tsc --noEmit; CI unit-tests.yml frontend job (yarn test:coverage, type check, lint).

Android layers

Android unit / Robolectric

• Responsible for view models, editors (MultiValueEditor, AddressEditor), screens, network parsing (ContractFixtureTest.kt), and offline/local-DB logic, all on the JVM via Robolectric.
• Must not be used for real device/emulator flows (instrumented) or a real backend.
• Runs via ./gradlew testDebugUnitTest :app:testObtainiumDebugUnitTest; CI android-tests.yml test job (also runs detekt — including the ktlint formatting wrapper — flavor-qualified lintDebug with warnings as errors, debug assembles, and the aggregated JaCoCo report). Run the analyzers locally with cd android && ./gradlew --warning-mode=fail detekt lintDebug :app:lintObtainiumDebug :app:lintFossDebug; --auto-correct fixes formatting. detekt 2.x supports the JDK 25 build JVM, so no downgrade is needed; do it for any Kotlin or resource change before pushing. android/lint.xml holds the reasoned lint exemptions (no lint-baseline.xml).
• Known gap, filed: SmsReceiver is untestable under the current Hilt whole-app-graph test component — issue #327.

E2E Android (instrumented)

• Responsible for the real app on an emulator/device against the real backend (docker-compose.test.yml): login → list → detail → edit → favorites → archive/delete + audit undo (issues #212, #238). This replaced the old manual Pixel 8a gate; runbook in README-developer.md.
• Must not be used for JVM-testable logic.
• Runs via ./gradlew :app:connectedObtainiumDebugAndroidTest (emulator), or adb reverse tcp:7300 tcp:7300 + the same with -Pandroid.testInstrumentationRunnerArguments.serverUrl=http://127.0.0.1:7300 on a physical device. CI android-tests.yml android-e2e job is deliberately off the PR path (issue #578, B5): push:main + nightly + manual, so an Android PR is gated by Robolectric only and the emulator signal lands post-merge and nightly.
• Skips are gated, not trusted (issue #1483). A JUnit Assume failure is a skip and a skip reports green, so on this required job it would otherwise be a silent loss of coverage. Two mechanisms close that:
  • Tests that need a CI-provided dependency (the docker-compose.compat-test.yml backends: ForceUpdateGateE2ETest, OldServerCompatibilityE2ETest) use com.mycorrhizal.crm.testing.assumeOrFailInCi (android/core/testing), the Android analogue of Go’s citest.SkipOrRequire: it skips locally but fails when the workflow passes -Pandroid.testInstrumentationRunnerArguments.requireReferences=true (every android-tests.yml E2E run does). Use it for any condition CI always satisfies.
  • Tests whose condition is legitimately false on the x86_64 CI emulator (the arm64-only embedded server, ADR 0028: LocalOnlyModeE2eTest, LocalBundleRoundTripE2eTest) keep a plain assumeTrue and are listed, with a reason, in android/e2e-expected-skips.txt (fully.qualified.Class#method | reason). After the run, cd backend && go run ./cmd/androidskipcheck -results ../android/app/build/outputs/androidTest-results/connected parses AGP’s JUnit XML and fails unless the skipped set equals the allowlist: a new skip, a listed test that stopped skipping (delete its entry), a listed test that did not run, or a run with zero test cases all fail. The skip count and per-test reasons land in the job summary. androidskipcheck -static runs on every PR (docs-citations job, pre-commit hook) and fails if an entry no longer names a real test method. Do not allowlist a test because its dependency is flaky – use assumeOrFailInCi.
• Known gap, accepted: this suite (and every other instrumented job) runs the debug build only. Neither a release/R8-minified APK nor an install-release-N-then-install-N+1 upgrade is ever driven by an instrumented test — RoomMigrationEncryptedTest (issue #480, PR #804) proves the Room migration chain in isolation on the debug variant, not through a real two-APK install of the shipped artifact. Accepted, not built: see release-gates.md — issue #994.

E2E web (Playwright)

• Responsible for complete user workflows through the shipped artifact — the all-in-one image (nginx + backend under supervisord) via docker-compose.test.yml on localhost:7300: register → login → create/edit contact → search → import/export → merge → favorites → archive → audit undo → backup/restore → admin/system-status → service worker → security headers → timeline endpoints. The only layer that exercises the whole deployed web stack, including the real first-boot migration path against a fresh DB.
• Must not be used for anything reachable cheaper in a lower layer.
• Lives in frontend/e2e/ (specs, fixtures.ts, global-setup.ts).
• Runs via npx playwright test; CI e2e-tests.yml e2e job (builds the all-in-one image, starts compose, waits on /health/live, runs Playwright, then the cmd/loadsmoke concurrent-write pass against the running instance). The workflow also runs on the nightly schedule (un-path-gated, 03:25 UTC) so a backend-only change that breaks a shipped user flow is caught overnight — see Layers vs CI path gating.
• Production-default pass (issue #274): the main suite runs against docker-compose.test.yml, which raises the API rate limit and enables CalDAV so a full parallel run can finish — so the shipped defaults (real rate limit, CardDAV off) are never exercised unless a job asks for them. A second CI job (e2e-prod-defaults) boots the same image with docker-compose.prod-defaults.yml layered on top and runs only the @prod-defaults-tagged subset (auth, contacts, dashboard) single-worker, which keeps it comfortably under the production limit (burst 1000, 600ms refill). The override nulls the rate-limit vars to empty strings rather than pinning their values, so the run exercises the genuine getIntEnv default path and would catch a future change to the shipped default.

Per-worker test users (issue #1480)

Specs do not share one account. e2e/fixtures.ts overrides Playwright’s storageState fixture so each worker authenticates as its own account (e2e_w<worker>_<run-id>, e2e/workerUser.ts): a worker-scoped workerUser fixture registers it through the API on first use, seeds the same five sample contacts as global-setup.ts, logs in through the UI once (so the cached user info is in the saved state), and hard-deletes it through the admin API on teardown. Account-level settings (/users/enabled-contact-fields, /users/date-format, /notifications/config, /immich/config, …) are singleton rows per user, so a test that mutates one only ever affects its own worker — there is no settings lock, no describe.serial for settings, and no PLAYWRIGHT_WORKERS=1 pin on the release run (CI runs 4 workers).

• Import test/expect from ./fixtures, never from @playwright/test (e2e/workerUser.vitest.ts fails otherwise): the override lives only there, and the project config sets no storageState.
• test.use({ storageState: LOGGED_OUT }) still opts out (auth/2FA/login specs).
• test.use({ sharedUser: true }) opts a file/describe into the shared testuser — the auto-admin (first registered account) — for specs that need admin rights (userManagement, the User Management mobile cases) or assert on the seeded dataset’s exact contents (visual, accessibility). That user’s settings are immutable by contract: while it is active, a browser request that would write an account-level setting (isSettingsMutation in workerUser.ts) is aborted and fails the test. (API-only request traffic is not intercepted.)
• Admin-only cleanup (deleteThrowawayUser(username)) always goes through the shared admin’s saved state, never the calling test’s own (non-admin) user.
• isolation.spec.ts asserts the harness itself: two worker-style users never see each other’s contacts or settings.
• A spec that must read the worker’s credentials (e.g. sessionExpiry’s in-place re-auth) takes the workerUser fixture.

Feature-seam specs and the route-coverage guard (issue #1481)

Four features had backend deep tests and mocked component tests but no spec driving the real bundle against the real server. Each now has one thin happy-path-plus-one-failure spec (the lower layers own the matrix):

Spec	Drives
occasions.spec.ts	gift obligation created on a contact page → “Needed” on /occasions → a purchased gift flips it → deactivating it (the “handled” switch; there is no separate done button) removes it → bad anchor rejected → contact delete takes the obligation
passkeys.spec.ts	Chromium CDP virtual authenticator (WebAuthn.enable + addVirtualAuthenticator): register in Settings → log out → passkey is the second factor → wrong-code removal refused → recovery-code removal. Skipped on non-Chromium (no CDP WebAuthn domain; #1479)
systemEvents.spec.ts	a webhook receiver answering 404 makes the delivery path record an integration_failed event; asserts severity/component/result on /system-events, detail, a server-side filter, and “Load more” growing past the first 100 rows; also /system-status and the non-admin 403
apiTokens.spec.ts	create → one-time secret + clipboard → bearer-only client gets 200 (cookie-less 401) → secret absent after reload → revoke → same bearer 401
legacyRedirects.spec.ts	/api-tokens → /settings, /tags → /circles?tab=tags

Intentionally E2E-exempt: GeoPulse (#160). It is an outbound integration to a user-hosted GeoPulse server; the default test stack has none and the feature has no UI beyond the connection form, so a spec would only test a stub. The client and service are owned by services/geopulse_client_test.go and geopulse_service_test.go, the form/suggestions by useGeoPulse.test.ts/ActivitiesPage.test.tsx.

Route-coverage guard. frontend/scripts/check-e2e-routes.mjs (yarn e2e:routes) parses the <Route path> table in src/App.tsx and fails if a path is not mentioned as a whole path literal in some e2e/*.spec.ts (/contacts is not satisfied by /contacts/${id}/prep), unless it is in e2e/route-exemptions.json ({ "/path": "reason" }, reason ≥ 15 chars). A stale exemption (route gone, or a spec now covers it) also fails, so the list only shrinks. It is enforced in the required Vitest job by scripts/check-e2e-routes.test.mjs, whose last case runs it against the real repo. The first run left the exemption file empty: every route was or became covered. Adding a page therefore means adding a spec (or a reasoned exemption).

These specs run the automatic per-test axe scan (fixtures.ts), which found three real light-mode contrast/naming defects on the admin pages — outlined success/error chip text and error text on a hovered row, and an unnamed storage progressbar — fixed via src/utils/statusText.ts and an aria-label.

Failure-mode specs (issue #1478)

The error UX used to be verified only in vitest with a mocked fetch. frontend/e2e/failureModes.spec.ts (tagged @failure-modes, runs on every PR with the rest of the suite) drives the real bundle in a real browser while the backend misbehaves, one test per failure class:

Failure	Mechanism	Asserts
503 on dashboard / contacts list / contact detail	page.route fulfilling the backend’s real error envelope (backend/errors/errors.go)	visible, retryable error (not a blank list, not “Contact not found”); request_id shown; the details sentinel never reaches the DOM; Try again recovers
Genuine 404 on contact detail	real backend	still “Contact not found”, no retry
Network down mid-session	context.setOffline	error alert, then recovery after going back online
429 from the rate limiter	stubbed RATE_LIMIT_EXCEEDED + Retry-After	the limiter’s message shown; retry recovers
401 mid-save of the add-contact form	stubbed 401 on POST /contacts	in-place re-auth (no redirect to /login), draft preserved, retry saves
Field-level validation error	stubbed 400 VALIDATION_ERROR with details: {"card.name": ...}	message on the first-name field (aria-invalid), draft preserved, edit clears it
Slow response (>5s)	route delaying 6s	skeleton (no error) while waiting, content after

Every test also fails on an uncaught pageerror or an unexpected console.error (only the browser’s own “Failed to load resource” line and the app’s structured [operation] Error: log are allowed), and ends on a state the automatic axe scan from fixtures.ts then checks, so the error state is scanned too. Two traps when writing more of these: page.unroute matches by function reference (a predicate rebuilt on each call silently removes nothing – the spec memoises its matchers), and the re-auth dialog and the dialog behind it can both contain the text “Session expired” – locate the re-auth prompt by role + accessible name. The real backend reports validation failures as 400 VALIDATION_ERROR, not 422. 412 Precondition Failed is covered by the two-tab specs ifMatchConflict.spec.ts (contact edit) and ifMatchEntityConflict.spec.ts (notes, reminders, and favorite-then-edit): the web client sends If-Match (ADR 0008) and shows the conflict dialog.

Visual regression (issue #258)

frontend/e2e/visual.spec.ts snapshots a small, curated set of stable views — the dashboard, the contacts list, a contact detail page and the “Add reminder” dialog, at desktop (1280×720) and phone (390×844) widths. An unintended layout or theme change to a pinned view fails the e2e job as a pixel diff. Baselines are committed under frontend/e2e/visual.spec.ts-snapshots/ and compared as part of the normal Playwright run; the automatic per-test a11y scan (fixtures.ts) also runs after each shot, so the views are pixel-identical and axe-clean.

Regenerate baselines after an intentional visual change:

cd frontend
npx playwright test visual.spec.ts --update-snapshots

then review the regenerated images (the HTML report / --ui shows before/after and the diff) before committing them. Snapshot files are suffixed -chromium-linux — CI’s platform, and the only suffix that should be committed.

The shots are made deterministic on purpose — see the spec header comment. Dates are pinned with a frozen page clock, the app’s primary font is injected as a committed webfont (e2e/fixtures/fonts/, route-intercepted) so rendering never depends on host fonts, and the dashboard/list responses are intercepted with fixed payloads (the dashboard’s “Stay in Touch” column is server-random). Only add a view here that you are willing to regenerate when the design changes.

Service-worker upgrade suite (issue #476)

playwright.sw.config.ts runs a separate Playwright config (specs under e2e/sw-upgrade/) that stages service-worker upgrades between two real production builds. The main e2e stack serves a single fixed build on :7300 and cannot express an upgrade; this suite replaces that server with its own harness (e2e/sw-upgrade/sw-server.mjs) on the same port, which serves build “a”, then build “b”, and lets each test flip the active build mid-test. The two builds are generated by e2e/sw-upgrade/fixtures.mjs from the current source (vite build, differing in the entry-chunk label injected via MYCORRHIZAL_SW_FIXTURE in vite.config.ts and, since issue #475, in the release version stamped into each fixture — enough to make service-worker.js byte-different and the update real). No backend is needed: these tests assert on the app shell, the worker lifecycle and CacheStorage.

It pins the lifecycle the milestone names “service-worker upgrades are tested” and “old cached assets cannot permanently strand users”:

• a deploy installs and waits behind an open tab (old build keeps serving, no mixed-version assets) until the update prompt’s reload is clicked;
• activation cleans the previous build’s precache entries (bounded, repeated rapid updates do not grow the cache);
• a user offline through a deploy converges on the new build on reconnect;
• the escape hatch (/_recovery.html, see service-worker-updates.md) recovers a user stuck on a deliberately broken worker;
• a client that converged on a rolled-forward deploy converges back when the operator rolls back (issue #477);
• the stale-client backstop (issue #475, staleClient.spec.ts): a mid-session deploy that raises min_client_version above the open tab’s build — or announces a different api_contract_version — forces the tab onto the new build through a real service-worker swap, shows a manual dialog instead of looping when the reload cannot converge, and fails open (never forces anything) while /health is unreachable. The harness’s /health advertises the active build’s own identity by default and lets each test override min_client_version / api_contract_version or fail /health outright.

Interrupted-deployment specs (issue #477, WEB-03) share the harness and add two scenarios a single-build stack cannot express:

• asset skew (assetSkew.spec.ts, serviceWorkers: 'block' — a network load, so no worker/precache can hide the 404): the harness withholds a chunk only build “a” has, a client loads build “a”’s index.html and hits the removed chunk, and the asset-skew bootstrap (public/asset-skew.js, the one piece of the app that runs when the bundle itself 404s) reloads the client onto the completed deploy — no opaque error, and a pre-existing sessionStorage draft survives the recovery. The bounded half stages a deploy that never completes: two automatic retries, then a clear “could not be loaded” message with a retry button, never a reload loop.
• backend up-but-not-ready (readiness.spec.ts): the harness flips /health/ready to 503 not_ready (the mid-migration state), and ServerStartingGate shows the starting-up screen instead of letting the app mount and fire a wall of failed requests, then mounts the app once the readiness poll clears. The fail-open half (ambiguous/unreachable readiness answer mounts the app as before) is unit-tested in src/readiness/readiness.test.ts, since the shared harness origin cannot produce a connection-refused response.

The harness’s control surface is shared server state; every spec file’s beforeEach calls resetHarness() (active build, /health, /health/ready and asset-404 blocks) so a failed test cannot leak state into the next file.

It runs on Chromium and Firefox. Run locally with npx playwright test -c playwright.sw.config.ts (the webServer builds the fixtures on first run); the e2e-sw-upgrade job in e2e-tests.yml runs it in CI. frontend-dev is useless here (see frontend/AGENTS.md): every build under test is a real vite build.

WebKit smoke (issue #992)

frontend/e2e/webkitSmoke.spec.ts is the only WebKit coverage in this repo — every other browser job above drives Chromium or Firefox, leaving the engine the documented Safari/iOS ≥16.4 floor actually names (see the supported runtime matrix’s “Browsers” row) completely untested until this issue. It runs under the main playwright.config.ts’s own webkit project (testMatch-scoped to just this one file, so it doesn’t drag the rest of the chromium-only-verified suite onto an untested engine) against the same docker-compose.test.yml stack as the main e2e job: login → dashboard render → service-worker registration → the Push API surface (window.PushManager) is present.

That last assertion was flipped once already: an earlier draft asserted PushManager was absent, based on a standalone WebKit2GTK 4.1 GObject-introspection check against the distro package rather than Playwright’s own bundled webkit build — a reasonable first check, but not the exact binary CI actually runs. The real CI run on Playwright’s WebKit showed PushManager present, so the spec (and this note) were corrected. See the WebKit engine caveat and the spec’s own file header for the full story and the current, precise scope of what is and isn’t proven — a real Push subscribe() round trip against a live push service still isn’t exercised here, on any engine. Run locally with npx playwright test --project=webkit (needs npx playwright install --with-deps webkit first); the e2e-webkit-smoke job in e2e-tests.yml runs it in CI on the same PR/nightly cadence as e2e.

Firefox smoke + nightly full suite (issue #1479)

Firefox is the engine most self-hosters on Linux run, and it already bit this project once (registering a service worker against the SPA-fallback index.html fails with “The operation is insecure”; see frontend/AGENTS.md’s dev-server note). playwright.config.ts therefore has a firefox project with two scopes:

• Every PR — smoke. frontend/e2e/firefoxSmoke.spec.ts, run by the e2e-firefox-smoke job (a parallel job on the same docker-compose.test.yml stack as e2e, so it adds nothing to the critical path). Six flows where engines disagree: login sets a HttpOnly + SameSite=Strict cookie that script cannot read (and logout ends the session); contact create + edit in the UI; a birthday rendered in the date format chosen in Settings; service-worker registration on the production build (the script is served as JavaScript and the registration survives a reload); the CSV export download; logout. It authenticates as the per-worker user from #1480.
• Nightly — full suite. With E2E_ALL_BROWSERS=1 the firefox project’s testMatch widens to every spec the chromium project runs (still excluding sw-upgrade/, which has its own config that already covers Firefox, and webkitSmoke). Only the scheduled e2e-firefox-full job sets it, so the per-PR path never pays for it; the scheduled run has zero retries, and nightly-failure-alert.yml already watches e2e-tests.yml. A spec that only makes sense on Chromium (screenshot baselines are Chromium-on-Linux only, CDP-only APIs) opts out with test.skip(({ browserName }) => browserName !== 'chromium', '<reason>'); a genuine Firefox-only defect is test.fixme with a reason and a tracking issue, never a silent skip.

Run locally against the stack with npx playwright install --with-deps firefox once, then npx playwright test --project=firefox (smoke) or E2E_ALL_BROWSERS=1 npx playwright test --project=firefox (everything). The smoke leg’s service-worker test is verified to fail when registration breaks (route /service-worker.js to HTML and the registration poll goes red).

Release/install smoke (DEPLOY-01, issue #450)

• Responsible for a clean install that proves the workflow, not the boot. .github/workflows/deploy-smoke.yml starts from nothing (fresh checkout, no volumes, no DB, only the config docs/getting-started.md tells a new operator to set), builds and boots the checked-out tree through docker-compose.yml + docker-compose.build.yml, then runs backend/cmd/deploysmoke against it: register the first user, log in, create a contact with several field types, relate it to a second contact, attach a file, upload a profile photo, search, export, and read every field back — each step touching a different subsystem a fresh install gets wrong (empty-DB migration, FTS index + triggers, attachment/photo directory permissions, JWT signing).
• Also owns startup ordering — the job asserts from the boot log that config validation and the empty-DB migration run both precede the HTTP listener, and that the first /health/ready already reports migrations: ok (issue #421) — and the misconfiguration cases real operators hit: missing JWT_SECRET_KEY, relative ATTACHMENTS_DIR/PROFILE_PHOTO_DIR, empty/* FRONTEND_URL. Those are unit-tested per field in backend/config; asserted at the real-binary boundary in backend/config/startup_smoke_test.go (builds mycorrhizal, runs it with a broken env, expects a non-zero exit whose output names the variable); and asserted again through the shipped container in the workflow for the vars the container entrypoint does not pre-empt (JWT_SECRET_KEY, FRONTEND_URL — a relative photo/attachment dir fails in entrypoint.sh’s chown first, so its validator message is a binary-boundary assertion only). A non-matching CORS Origin is also checked against the configured FRONTEND_URL.
• Gates under infra (Dockerfile, docker-compose*, docker/, .env.example, backend/cmd/deploysmoke/**) and runs nightly.
• backend/config/startup_smoke_test.go builds and runs the server binary, so it is skipped under go test -short.
• This is the layer that v0.6.6 (install/upgrade/backup/recovery) and its sibling DEPLOY-02 (#451, upgrade) / DEPLOY-03 (#452, interrupted startup) build on.

Post-publish functional smoke (DEPLOY-04, issue #996)

• Responsible for proving the published artifact boots, not a from-source build of the release commit. Everything above builds the checked-out tree through the docker-compose.build.yml override; docker-publish.yml’s own verify-release-assets job only proves the pushed image resolves in the registry (docker buildx imagetools inspect — presence, not behavior). A release-path-only defect (a broken entrypoint, a bad baked-in env default, a migration that only fails inside the real Dockerfile’s build context) could reach operators completely undetected between those two checks — split from #923 into #996.
• docker-publish.yml’s post-publish-smoke job (needs: verify-release-assets, if: github.event_name == 'push') cosign verifys the just-published all-in-one image’s identity, resolves its digest, and boots it with docker-compose.published-smoke.yml — a standalone compose file (no build: anywhere in it, since Compose merges a build: present in the file set with an image: override by building anyway rather than pulling, which would silently defeat the point) that references ${MYCORRHIZAL_IMAGE} by digest, never the mutable tag. It then runs the same backend/cmd/deploysmoke workflow the from-source job runs, against the real artifact.
• Runs once per release, inside the tag-triggered docker-publish.yml — not a separate schedule, since there is nothing to smoke-test between releases.

Cross-cutting: unexpected server logs (issue #1474)

A passing test can hide a server that is quietly misbehaving: issue #1471’s FK-failing audit hooks emitted 698 WARN lines inside a green E2E run and were found only by a person reading the log. The convention is therefore any warn-or-above server log line in a happy-path run is a failure unless it is explicitly allowed with a reason. A 4xx is the server working as designed, so errors.LogError and the validation middleware log client errors at info; only a 5xx (or a genuinely swallowed internal failure) is error/warn.

• Go tests: internal/logtest. logtest.Capture(t) swaps the global logger for an in-memory JSON sink (Records(), Warnings(), AssertNoWarnings(t, allowedMessages...)). logtest.Guard(t) is Capture plus an automatic end-of-test AssertNoWarnings; it is wired into the controllers’ setupRouter(t) and every services setup*TestDB(t) helper, so a new happy-path test inherits it. A test that deliberately drives a logging path (an injected DB failure, a rejected upload) opts out with logtest.AllowWarnings(t, "<reason>") – the reason is mandatory and covers the test’s subtests. The swap is process-global, so it is not safe with t.Parallel(); a test that installs its own logger capture must do so after calling the helper (the helper’s Guard would otherwise replace it).
• E2E / deploy smoke: cmd/logguard. cd backend && go run ./cmd/logguard reads the server’s JSON log on stdin and fails on any warn-or-above line not matched by an entry in backend/cmd/logguard/allowlist.json ({message, fields?, max_count, reason}; all but fields required), or on an entry matching more lines than its max_count. An entry may carry fields: a map of log-field name to an RE2 regexp that must match the whole stringified value (e.g. "path": "/api/v1/immich/people", "status": "503"). A generic message such as HTTP request / Request error (the 5xx access and error logs) is therefore only excused on the exact routes the E2E specs drive, not on any route; the same message on another path/status, or with the field missing, is reported as unlisted with its method/path/status. The first matching entry in file order takes a line, so a scoped entry never spills into a later catch-all’s budget. Bounds are the exact count the spec produces times 2 for Playwright’s single CI retry, with any further headroom stated in the entry’s reason; issue #1664 re-derived them from observed E2E counts so a bound is not loose enough to hide a regression. Non-JSON lines (nginx, gin text) are ignored. Every run prints an entry | observed | max_count table to stdout on success and failure (zero-observed entries included), and appends a markdown copy to GITHUB_STEP_SUMMARY titled with the job, so a stale entry or a bound with a wide margin is visible without re-running (observedTable / appendSummary, cmd/logguard/main.go). It runs after the Playwright suite in e2e-tests.yml (e2e and e2e-prod-defaults), after the workflow in deploy-smoke.yml, and over the server log in both reference-clients-e2e.yml legs (which set GIN_MODE=release, since a non-release mode forces console-format logs). Adding an allowlist entry is a reviewed, reasoned weakening of the gate, like any other (.github/AGENTS.md “Gates are not silently weakened”).

Cross-cutting: performance/load and security

Two sets are not pyramid layers in the “write a test here” sense, but they own failure classes the 0.6.x milestones name, so they get explicit homes.

Performance/load

• Owns N+1 / query-count regressions (the *_QueryCountIsBounded tests, issue #261), benchmark bodies (-bench . -benchtime=1x in backend-checks), and concurrent-write stability against the deployed artifact (cmd/loadsmoke in the e2e job — the deployed counterpart to database/concurrent_write_test.go).
• Owns scale characterization and the large-dataset migration test — the PERF-01 basis (issue #468) and issue #495: internal/largedata scales the TEST-02 manifest (same shapes, more rows, pathological records at scale), cmd/migratebench measures migration duration / peak memory / peak disk per supported path, and the recorded numbers live in docs/development/scale-testing.md. The resource-exhaustion half is the chaos job’s large-migration-disk-full (ENOSPC during a large migration fails closed) — see docs/development/fault-injection.md.

Soak / long-run (issue #1496)

Everything above is short-lived; the soak is the one test that ages a real server. backend/internal/soak + cmd/soak run a sustained mixed workload against the real server (in-process via embedded.Start) and fail on growth of RSS / heap / goroutines / fds / WAL / rate-limiter entries, latency degradation, any 5xx, a failed integrity_check, FTS drift, a doctor violation or a wedged scheduler. Per-PR it is a ~15 s smoke in the Backend (Go) job (healthy must pass, injected leaks must fail); soak.yml is the weekly 45-minute form (advisory tier). See soak-testing.md and the committed thresholds in soak-baseline.md.

Security/adversarial tooling

• Owns the vulnerability classes no layer above is shaped to catch: BOLA/IDOR (cmd/bolacheck, the deterministic complement to Schemathesis), spec-derived fuzzing for 5xx/auth (Schemathesis + cmd/schemagate + ignore list), DAST (ZAP, weekly), static analysis (CodeQL, golangci-lint’s backend/.golangci.yml set — gosec, errcheck/staticcheck and the #1906 additions — cmd/gormerrcheck, detekt, mobsfscan), dependency and workflow audits. These are their own workflows with their own triggers (PR/push/nightly per workflow); a feature PR writes tests in the pyramid, not in these. The security checklist (docs/security/asvs-l2.md) is the tracking surface for which control each one evidences.
• Owns the failure-injection / chaos scenarios — TEST-06, issue #434 (v0.6.3): the in-process faults harness, the external-fault CI job (chaos-tests.yml), and the fault catalog (docs/development/fault-injection.md). It is the mechanism behind v0.8.0’s adversarial audit; the domain milestones (MIG-04/05 #439/#440, DEPLOY-03 #452, CON-04 #459, #526) consume it.

Agent behaviour eval (issue #1959)

This is deliberately not a layer that gates anything. The harness (backend/cmd/agenteval, backend/internal/agenteval) runs one repository-shaped task per Tier 1 suite in a disposable git worktree from a pinned commit, executes the same Suites commands the skills and CI use, checks the changed paths against the task’s expect_paths/forbid_paths, and flags a changed generated artifact whose editable source did not change. A second worktree with the skills tree, nested AGENTS.md files, and root routing table stripped is the control; the artifact records what was stripped and the delta.

A missing requires tool skips the task with a recorded reason. A seed patch that does not apply fails before the agent runs. The workflow is workflow_dispatch only, has no write token, exposes only the model credential, and is not in .github/release-gates.json or any release workflow.

Failure classes by owner

The 0.6.x milestones name failure classes, not layers. Each one below is owned by exactly one layer (or a filed gap issue); a class with no row here is a gap to file, never something to silently absorb.

Failure class	Owning layer	Milestone / issue
Pure-computation bugs: parsing, temporal/DST math, relationship-type inversion, cadence, validation	Backend unit	v0.6.11
ORM-vs-migration schema drift, GORM column derivation	DB/integration	everywhere (trap #1)
Persistence bugs: silently swallowed .Error, flat-field derivation, RecordForContact vs RecordFromContact, delete/cascade semantics, soft-deleted unique-index collisions	DB/integration	v0.6.4, v0.6.8 (traps #2/#3/#6/#7)
IDOR / missing ownership scoping	DB/integration (+ cmd/bolacheck evidence)	v0.6.12 (trap #5)
Concurrent writes, SQLITE_BUSY, lost updates, stale writes, idempotency	DB/integration (concurrent_write_test.go) + E2E web (loadsmoke)	v0.6.7
Route↔spec drift, binding-shape drift, dropped response keys	API contract	v0.6.3 (#266)
Client/server contract mismatch	API contract + E2E web/Android	v0.6.10
vCard/JSContact/CSV/iCal format correctness, lossy conversions, unknown fields, malformed input, sensitivity filtering	Import/export interop	v0.6.5
Round-trip fidelity, repeated-conversion stability	Import/export interop	v0.6.3 (TEST-03 #431, pulled forward from v0.6.5); v0.6.5 (DATA-03 #443 — idempotence-after-first-conversion)
Migration data loss, semantic drift across schema versions, dirty/version mishandling	Migration	v0.6.4
Interrupted migration, migration rollback/recovery, first-boot migration on empty DB	Migration + release/install smoke	v0.6.4, v0.6.6, #438/#452
Monica/Meerkat import mapping errors	Import/export interop (fixtures per DATA-*)	v0.6.4 (#351/#353)
Clean-install failure (env vars, CORS, permissions, empty-DB migration)	Release/install smoke (deploy-smoke.yml + cmd/deploysmoke + config/startup_smoke_test.go)	v0.6.6 (#450)
Release-path-only defect (broken entrypoint, bad baked-in env default, a migration that only fails inside the real Dockerfile’s build context) reaching the published image undetected	Post-publish smoke (docker-publish.yml’s post-publish-smoke job + docker-compose.published-smoke.yml + cmd/deploysmoke)	#996
Backup/restore/cross-version restore failure	DB/integration (backup_test.go, restore tests) + release/install smoke	v0.6.6
Component/hook state bugs, i18n key/placeholder drift, format-provider bugs	Frontend unit	v0.6.11
Android view-model/editor/offline/local-migration bugs	Android unit/Robolectric	v0.6.7, v0.6.10
Whole-user-flow breakage (register→create→search→export), shipped-artifact boot	E2E web	v0.6.3, v0.6.6
Layout/theme regressions (a CSS refactor shifting a card or breaking a dialog)	E2E web (visual.spec.ts)	v0.6.3 (#258)
Service-worker lifecycle, stale-cache stranding	E2E web (serviceWorker.spec.ts)	v0.6.10
Interrupted deploy — asset skew (stale index.html referencing chunks the new build deleted → blank boot)	E2E web (e2e/sw-upgrade/assetSkew.spec.ts, client half in public/asset-skew.js)	v0.6.10 (#477)
Interrupted deploy — backend up but not ready (wall of failed requests while the server migrates)	E2E web (e2e/sw-upgrade/readiness.spec.ts, client half in ServerStartingGate)	v0.6.10 (#477)
Android real-app flows (favorites, archive/delete + undo)	E2E Android	#212/#238
Referential integrity, orphan detection, reciprocal-relationship consistency, derived-data (FTS/flat-columns/cadence) drift	DB/integration (DB-01 checker, SEARCH-*)	v0.6.8 (#460/#461–463/#497)
External-integration failure behavior, retries	DB/integration (services-level) + failure injection (docs/development/fault-injection.md)	v0.6.9, #434
Query/scale performance, resource exhaustion, N+1	Performance/load	v0.6.9 (#468, #261)
Security vulnerabilities (BOLA, auth bypass, injection, SSRF)	Security/adversarial tooling	v0.6.12, v0.8.0
Adversarial/chaos scenarios	Failure injection (TEST-06, #434 — in-process seams + external chaos job)	v0.8.0

Differential suite (TEST-08, issue #680)

The round-trip suite (TEST-03) and the golden fixtures (ADR-0003) both compare our output against oracles we wrote. The failure mode those cannot see is code and tests consistently, confidently wrong — a parser and an exporter sharing one (wrong) reading of the spec pass each other’s round trips. The differential suite runs the same corpus through an independent, pinned reference implementation and requires semantic equivalence (TEST-03’s comparator), so a disagreement is evidence one of us misreads the RFC, and the RFC is the tiebreaker.

The reference must share no code with our adapters. That rules out emersion/go-vcard and emersion/go-ical, which our vcard3/vcard4 and caldav packages are built on. The pinned references are:

Format	Reference	Runs	Harness
vCard 3.0 / 4.0	Python vobject 0.9.9	per-PR (unit-tests.yml, pip-installed) + nightly	backend/differential/reference/vobject/vcard_ref.py
JSContact	Rust calcard 0.3.13	scheduled + path-gated (differential-e2e.yml, built in a digest-pinned rust image)	backend/differential/reference/calcard/
iCalendar (CalDAV)	Go github.com/arran4/golang-ical (go.mod-pinned)	per-PR, no runtime deps	in-package tests in caldav/ + services/

Each vCard/JSContact leg runs both directions over the shared corpus (TEST-02 fixture + golden fixtures + pinned generated seeds): ours → reference (our export, reference parses) and reference → ours (reference emits, our import), asserting semanticequal equivalence per TEST-03. The iCalendar leg compares VEVENT object fields (uid/summary/description/location/dtstart/rrule) by name.

Divergence policy (the #496 pattern): known reference-side divergences are pinned per corpus entry + direction + concept with a written reason, and the drift check fails when a pin stops reproducing (the reference got fixed) or an unpinned disagreement appears. A confirmed our-side bug is fixed, never pinned — the suite has already caught and fixed two:

• the vCard CATEGORIES comma-list escaping bug (CATEGORIES:a\,b meant “one category with a comma” to any spec-honoring reader, not two categories; our importer happened to split the unescaped value back), fixed in vcard3/ vcard4; and
• the JSContact created/updated wire shape (RFC 9553 defines them as UTCDateTime strings; our adapter emitted a @type-discriminated object), fixed in jscontact.

Supply-chain pins: vobject is installed in CI from a hash-pinned requirements file (backend/differential/reference/vobject/requirements.txt, --require-hashes — zizmor audits pip installs for unpinned supply-chain refs); calcard is version-pinned in reference/calcard/Cargo.toml (=0.3.13) with a committed Cargo.lock for the transitive graph, and the build toolchain is a digest-pinned rust image; golang-ical is pinned in backend/go.mod.

Running locally: the vCard leg needs python3 with vobject importable (it skips with a message otherwise). The JSContact leg resolves $MYCORRHIZAL_CALCARD_CMD (CI sets it to the built binary), then a prebuilt reference/calcard/target/{release,debug}/calcard-ref (cargo build --release --locked), then skips. The iCalendar leg is pure Go.

Where a row names a milestone but the concrete ticket lives elsewhere (e.g. the “test infrastructure supports historical fixtures” acceptance criterion → MIG-01 #436, and “large/representative datasets” → TEST-02 #430), that ticket is the implementer of the layer’s capability, not a different layer.

Real-server + real-client interoperability (TEST-09, issue #681)

TEST-08 (above) proves our serialization against reference parsers. TEST-09 proves our servers against real third-party implementations — the missing half of the “code and tests consistently, confidently wrong” shape that #496 warned about. Two directions:

1. Reference servers → our client (the #496 breadth). The full sync lifecycle + TEST-02 fixture round trip runs against real CardDAV servers in Docker, selected by MYCORRHIZAL_CARDDAV_SERVER_ID:

   Server	Container	Layout	Divergence register
   Radicale	tomsquest/docker-radicale (digest-pinned)	/<user>/contacts/	vobject: celine rejected (multi-N ALTID); vobject truncates N/ADR to their RFC 6350 component counts, so bob photo/ADR is mangled and the I18N personas lose the RFC 9554 additions (carmen/joao secondary surname, somchai subdistrict/district) — per-contact pinned
   Baikal	ckulka/baikal	/dav.php/addressbooks/<user>/contacts/	Sabre VObject serves vCard 3.0: the 4.0-only concepts land in passthrough (per-contact pinned)
   Nextcloud	nextcloud:stable	/remote.php/dav/addressbooks/users/<user>/contacts/	same as Baikal, plus eve rejected (BDAY re-validated as iCalendar datetime)
   DAViCal	—	—	not run: the only image (janlo/davical) is a legacy Debian-stretch build whose provisioning (SCRAM-incompatible libpq, no headless user/collection creation) is unpinnable. Covered structurally by the Sabre-family analysis via Baikal/Nextcloud.

   The suite is TestCardDAVReferenceServer_RoundTrip in backend/services/carddav_radicale_integration_test.go, gated by MYCORRHIZAL_CARDDAV_SERVER_URL (skips otherwise). The workflow carddav-e2e.yml is a three-way matrix; each server is provisioned headlessly by .github/scripts/carddav-reference/provision-{baikal,nextcloud}.sh (Baikal has no headless setup story — the script drives its two-step web wizard over HTTP, normalizes the case-sensitive dav_auth_type YAML value, and seeds the DAV user/principal/address book directly in the SQLite DB). The divergence registers pin exactly what each server does to the fixture; anything else is a failure that names the server + concept.

2. Reference client → our server (the core deliverable). A real vdirsyncer (a mature CardDAV/CalDAV client with its own protocol implementation and its own vCard parser) consumes OUR server end-to-end: provision/discover, pull (asserting the client’s stored bytes are semantically equal to the staged fixture — the pathological fixture round-trips through it), ETag-driven quiescence, and PUT-create/update/ delete pushed by the client. TestCardDAVVdirsyncer_ClientRoundTrip in backend/services/carddav_vdirsyncer_integration_test.go, gated by MYCORRHIZAL_OUR_CARDDAV_URL (skips otherwise). The workflow reference-clients-e2e.yml builds our backend, starts it with CardDAV enabled, pip-installs vdirsyncer from a hash-pinned requirements file (.github/scripts/carddav-reference/vdirsyncer-requirements.txt, cp312 wheels — regenerate per its header if setup-python moves), and runs the test, which registers its own throwaway user.

3. Reference GUI client → our server (issue #917, the reference-client matrix’s automated leg — see docs/development/reference-client-matrix.md for the full matrix and what’s manual vs. automated). A real, pinned DAVx5-OSE APK — sideloaded, not our own instrumented code — is driven through account setup on an Android emulator by .github/scripts/reference-client-davx5/run.sh via adb/uiautomator, locating every UI element by exact text/content-desc (DAVx5 is a Jetpack Compose app with no resource-ids in its accessibility tree) rather than hardcoded coordinates. Unlike vdirsyncer above, the assertion isn’t a Go test reading the client’s in-memory state — it’s adb shell content query against Android’s real ContactsContract, the same surface a human tester’s eyeball would check, seeded with the canonical pathological dataset (issue #430) so non-ASCII names are part of the assertion. This also stands in for the matrix’s “Android native contacts (via the Android provider)” row: DAVx5 syncs through Android’s real sync adapter into the real provider, so one pass covers both. The workflow job (davx5 in reference-clients-e2e.yml) seeds a server with pentestseed, boots the emulator via reactivecircus/android-emulator-runner (the same action android-tests.yml uses), and runs the script — same nightly + path-gated schedule as the vdirsyncer job.

Running locally: start our server (see the launch.json note in AGENTS.md), then

cd backend
MYCORRHIZAL_OUR_CARDDAV_URL=http://127.0.0.1:8090 \
MYCORRHIZAL_VDIRSYNCER_CMD=/path/to/vdirsyncer \
  go test ./services/ -run '^TestCardDAVVdirsyncer_ClientRoundTrip$' -count=1 -v

For the server matrix, start one of the containers (the workflow scripts are reusable: provision-baikal.sh / provision-nextcloud.sh), then

MYCORRHIZAL_CARDDAV_SERVER_ID=baikal \
MYCORRHIZAL_CARDDAV_SERVER_URL=http://127.0.0.1:5233 \
MYCORRHIZAL_CARDDAV_USER=syncuser MYCORRHIZAL_CARDDAV_PASSWORD=syncsecret \
  go test ./services/ -run '^TestCardDAVReferenceServer_RoundTrip$' -count=1 -v

What the matrix has already caught and fixed (all three were invisible to the fake suite):

• Filter-less addressbook-query returned an empty collection. A query with no <card:filter> (RFC 6352 §8.6 — how Thunderbird, Apple Contacts, DAVx5 and vdirsyncer enumerate) was routed through go-webdav’s Filter, whose empty FilterAnyOf matches nothing. Our own client’s full-refetch fallback is a filter-less query, so our server used to answer our client with “no contacts”. Fixed in backend/carddav/backend.go with a red handler-level test.
• MKCOL for an unsupported second address book returned 500, which clients read as “server broken” instead of the interop-correct “single address book by design” (403). Fixed with a red test.
• GET/DELETE of a missing card returned 500 instead of 404 — vdirsyncer’s delete round-trip surfaced it. Fixed with a red test.
• .well-known/{carddav,caldav} 404’d on PROPFIND, only accepting GET. RFC 6764 only requires GET, but a real DAVx5 client issues PROPFIND directly at the well-known URI during autodiscovery and got a 404 instead of the redirect — account setup failed outright with “Couldn’t find CalDAV or CardDAV service.” Found by the DAVx5 leg (#917) on its first-ever run; a Go-only test of the handler couldn’t have caught this since the handler itself is method-agnostic — the bug was the router registration. Fixed in backend/routes/routes.go, pinned by TestWellKnownDAVDiscovery_AcceptsPROPFIND.

Known limitation (documented, not yet fixed): our go-webdav client cannot negotiate an address-data version (go-webdav v0.7.0 AddressDataRequest has no version field), so against SabreDAV-based servers (Baikal/Nextcloud) the full-refetch fallback would receive vCard 3.0 re-serializations; the pinned divergence registers encode exactly that. Requesting vCard 4.0 from those servers would collapse most of the register. Tracked as a follow-up.

ETag hand-verification (per AGENTS.md): the matrix’s quiescence and update assertions ARE the automated ETag-correctness proof — vdirsyncer detects an update only if our server’s ETag changes on PUT, and the “unchanged server makes no second pass” assertion fails if our server returns unstable ETags. To hand-verify the reverse (reintroduce the historical etag-column mapping bug — remove the gorm:"column:etag" tag on ContactSyncLink.ETag so GORM writes e_tag): run TestCardDAVReferenceServer_RoundTrip against a real server — the real migrated schema has no e_tag column, so the sync fails loudly (“table contact_sync_links has no column named e_tag”) instead of silently carrying an empty etag column that would break incremental sync. Restore the tag after confirming.

The manual client matrix (Apple Contacts macOS/iOS, Thunderbird, Android native, DAVx5) — the clients that cannot be scripted — is a documented checklist with last-run dates in docs/development/reference-client-matrix.md.

Real-server integration contract tests (issue #1490)

Every external integration other than CardDAV used to be tested only against a hand-written httptest fake, which encodes our reading of the other side’s API — the “shared misconception” failure ADR-0003 names for formats. INT-02’s failure-behavior suite proves what we do when an integration fails; backend/integrations/realserver proves we speak its protocol when it succeeds, against the actual server:

Server (pinned in backend/integrations/realserver/stack/)	Test file	What it proves
Keycloak 26 (OIDC)	oidc_test.go	full authorization-code + PKCE (S256) login through our real /auth/oidc/* handlers, PKCE verifier enforced, RP-initiated logout (end_session_endpoint, id_token_hint, post_logout_redirect_uri) ends the IdP session, signing-key rotation after the JWKS was cached
ntfy, Gotify	notifications_test.go	a notification sent through our delivery code is read back from the server’s own API with the title/message we meant; a rejected request / bad token surfaces as an error
Mailpit (SMTP)	smtp_test.go	SendEmail’s SMTP transport delivers; envelope, Q-encoded UTF-8 subject and HTML body read back from the server
Nextcloud (WebDAV)	webdav_test.go	Ping/ListDir against a real PROPFIND: dir vs file, decoded non-ASCII names, oc:fileid, 401/404 sentinels
Paperless-ngx	paperless_test.go	Test Connection (Ping/GetMe), ListDocuments (listing + full-text query) and GetDocument over a real consumed document
Immich	immich_test.go	Ping/GetMyUser/ListPeople/GetStatistics/RecentAssets (POST /api/search/metadata) and both thumbnail fetches over a real uploaded asset and person
Seafile 11	seafile_test.go	Ping/PingAuth/ListLibraries/ListDir over a real library, folder and file

It earned its keep on the first run: it found that ntfy was being sent a JSON body at /<topic> (a real ntfy displays that raw JSON as the message, with no title — ntfy’s JSON API posts to the server root), that Paperless Test Connection could never succeed (GET /api/ is a 302 and /api/auth/me/ is not an endpoint) and that Paperless’s file name was read from a field the API does not serve. The fakes had all agreed with the code.

Running it. .github/workflows/integration-real-servers.yml runs nightly, on dispatch, and on a PR only when the realservers filter in .github/filters.yaml fires; it is a release-tier gate composed by release-validate.yml. It sets MYCORRHIZAL_REQUIRE_REFERENCES=1, and every test goes through internal/citest.SkipOrRequire, so a server that fails to provision fails the job rather than skipping green. Locally, bring up one stack at a time and export the MYCORRHIZAL_RS_* variables the workflow lists:

cd backend/integrations/realserver/stack
docker compose -f docker-compose.light.yml up -d        # ntfy, Gotify, Mailpit, Keycloak
cd ../../.. && MYCORRHIZAL_RS_OIDC_ISSUER=http://127.0.0.1:18082/realms/mycorrhizal \
  MYCORRHIZAL_RS_NTFY_URL=http://127.0.0.1:18080 \
  go test ./integrations/realserver/ -run 'Ntfy|OIDC' -v

The ports and credentials in the workflow’s env: block are the ones each docker-compose.*.yml publishes; the heavy stacks (paperless, immich, seafile) are separate files so only one runs at a time, and Nextcloud reuses .github/scripts/carddav-reference/provision-nextcloud.sh. Images are pinned by tag@digest; Dependabot’s docker entry for the stack/ directory proposes bumps, and a bump that breaks a client is the signal.

Coverage map. backend/integrations/realserver_coverage_test.go has a row for every Registry() integration: real-server tests that exist, or a reasoned Exclusion (hosted vendor APIs — geocoder, Resend, HIBP, the update check, browser push services — webhooks whose “server” is the user’s own receiver, and the not-yet-wired CalDAV and GeoPulse legs). A new integration cannot land without one or the other, and the same file asserts every MYCORRHIZAL_RS_* variable the tests read is set by the workflow.

Hand-verifying a leg: break the client and watch it fail — e.g. rename the username JSON tag on PaperlessUser, or change sendNtfyMessage to POST to /<topic> again — the unit tests stay green (the fakes follow the code) and the real-server leg goes red.

CI plumbing around the suites

Three workflows exist to make the suites above trustworthy rather than to test anything themselves; none is a merge gate.

• Test Report (test-report.yml, issue #268): a workflow_run job that turns each suite’s JUnit artifact into a check run. Since issue #1687 its job-level if skips a run whose tested commit is no longer the PR head, and skips cancelled runs, because dorny/test-reporter resolves the current PR head rather than the tested commit and would otherwise publish a superseded run’s (possibly failing) report onto a newer commit. Fork PRs carry an empty pull_requests array, where it falls back to head_sha and is already correct. The head’s own run still reports its own results.
• Runner-image preflight (runner-image-preflight.yml, issue #1659): dispatch-only, never scheduled (so nightlyalertcheck does not apply). GitHub moves ubuntu-latest to Ubuntu 26.04 on 2026-10-19; the workflow runs the image-sensitive suites (e2e-tests, unit-tests, android-tests, min-version-tests, reference-clients-e2e, deploy-smoke) on an explicit image (default ubuntu-26.04) through a new runner input on each of them (empty keeps ubuntu-latest, so their normal triggers are unchanged), so breakage arrives as a triaged list before the label moves instead of a wave of red nightlies. The Playwright apt and browser caches key on the detected OS image, so a stale cache from the old image cannot be restored onto the new one.
• Bypassed-check alert (bypassed-check-alert.yml, issue #1661): on every push to main it checks whether each context the main-protection ruleset requires was green on the merged PR’s head, and opens a ci+p1 issue if not. Advisory by design (it cannot block a merge that already happened), so it is absent from the ruleset and release-gates.json; semantics, the 40-minute settle window for checks still running at merge, and the issues: write scope are in repo-governance.md.

Layers vs CI path gating

Each layer maps onto an existing area of .github/filters.yaml (issue #264) — the single source of truth for path gating. No new filter is required — every layer has a home, and a layer’s tests are gated exactly when its filter fires — with one deliberate exception: the carddav filter (issue #496), the real-server (Radicale) complement to the fake-based CardDAV suite, which is too slow and too flake-prone to run on every backend diff (same rationale as the chaos filter, #434). The changes job’s named outputs gate the suite jobs; skipped suites report success, so all checks can be required in branch protection.

Layer	Filter area(s) in .github/filters.yaml
Backend unit	backend
DB/integration	backend
API contract	openapi (drift tests ride backend; fixture consumers ride frontend/android; spec fuzz + cmd/schemagate ride openapi)
Import/export interop	backend
CardDAV real-server interop	carddav (the fake-based CardDAV suite rides backend on every PR; the real-server Radicale job runs on the schedule + manual dispatch, and on a PR only when the carddav filter fires — issue #496)
Integration real-server contract	realservers (nightly + dispatch always; on a PR only when the clients, the suite or its stacks change — issue #1490)
Migration	backend
Frontend unit	frontend
Android unit/Robolectric	android
E2E web	frontend + openapi + infra (builds/runs the deployed artifact) + nightly full-suite run (schedule, 2026-08-28)
E2E Android	android + openapi + infra (push:main + nightly + manual only, #578)
Release/install smoke	infra (deploy-smoke.yml — clean-install workflow + startup ordering + misconfig cases; also nightly, #450)
Performance/load	backend (benchmarks) + infra (loadsmoke in the e2e job)
Security/adversarial	not a filter area — each workflow has its own triggers

The workflows filter (.github/**) re-runs everything, because a workflow change can affect any check. docs/** intentionally maps to nothing: a change to this page triggers only the docs job — correct, because a documentation edit cannot break a test layer.

Two nuances worth writing down, because the mapping is almost clean:

• backend/openapi.yaml matches both backend and openapi, so a contract change re-runs the Go suite and the spec-fuzz/contract suites, and the frontend/android jobs re-run their fixture-consuming tests. That is intended: the contract’s home is openapi.
• The web E2E suite gates on frontend/openapi/infra/workflows, not backend: a pure backend/** change is covered by the Go checks in unit-tests.yml and does not re-run the full e2e stack on the PR. A backend-only change that breaks a shipped user flow is therefore caught by the nightly full web suite — e2e-tests.yml also triggers on the schedule event (03:25 UTC, un-path-gated since a scheduled run has no diff to gate on) — not by the PR. That is a deliberate cost/speed trade: the PR path skips the deployed-artifact stack for backend-only diffs, and the overnight run is where that signal lands.

Nightly failure alerting

Nightly is the only zero-retry tier: a flake on the PR/push path gets an automatic retry (gradle test-retry, gotestsum reruns, Playwright’s retry config); a scheduled run does not. It is also where the slowest and most failure-prone suites live — mutation testing (go-mutation.yml, stryker.yml), the large-dataset performance/capacity jobs (migration-tests.yml’s nightly leg), chaos injection (chaos-tests.yml), and the real-server CardDAV/DAVx5/vdirsyncer interop suites — so a red nightly run is disproportionately likely to be a real regression, not noise. Previously nothing alerted on one: a scheduled workflow going red just sat on the Actions tab.

nightly-failure-alert.yml closes that gap. It is a workflow_run listener that reacts to the completion of every workflow with a schedule: trigger (29 as of this writing), gated to the run that actually fired on its schedule event rather than a push/PR/dispatch run of the same workflow:

• On failure, timeout, startup failure, or a cancellation it opens a nightly-failure-labelled issue titled Nightly failure: <workflow name> (or comments on the one already open for that workflow), linking the run and naming the non-success job(s).
• On success it comments on that workflow’s open nightly-failure issue, if any, and closes it — the alert clears itself once the schedule is green again, rather than needing someone to notice and close it by hand.

The workflows: list in that file’s on.workflow_run.workflows must equal exactly the set of workflow name:s that declare a schedule: trigger. cd backend && go run ./cmd/nightlyalertcheck (a Docs & security-doc citations step, every PR — unconditional, like citecheck/docscheck above, since .github/filters.yaml maps a workflow-only change to workflows, not backend) fails the build if the two sets ever diverge: a newly-scheduled workflow that nobody registered would otherwise alert on nothing, and a renamed or de-scheduled workflow would leave a dead entry.

Flake ledger (issue #1488)

Retry-and-forget hid flakes: gotestsum --rerun-fails (PR/push), Playwright retries: 1 and the Android second emulator boot each turn a failed first attempt into a green check, and the only trace used to be a log line. A test failing 1-in-5 was invisible until it failed twice in a row on someone’s PR. The flake ledger is the longitudinal record.

Inputs. Each suite uploads a small flake-* artifact (14-day retention, separate from the bulky 7-day report artifacts):

Artifact	Producer	Contents
flake-go-<leg>	unit-tests.yml backend legs	junit-<leg>.xml (the -race pass) + gotestsum’s rerun-fails-<leg>.txt
flake-playwright-<job>	e2e-tests.yml (e2e, prod-defaults, webkit)	playwright-results/results.json (Playwright’s JSON reporter, now always on; status: flaky = failed then passed)
flake-android-<leg>	android-tests.yml (API 35, API 26)	final-attempt JUnit + the preserved flaky-attempt-* XML
flake-signals-<name>	anything else that retries	a flake-signals-*.json in the neutral format below

The neutral format lets a new retry mechanism feed the ledger without a new parser: {"suite":"go/floor","tests":[{"test":"pkg.TestX","outcome":"passed_on_retry"}]} with outcome one of passed, passed_on_retry, failed. (The Go failed-package re-run script and the ZAP re-scan proposed in #1487 are the intended users: emit that file and upload it as flake-signals-<name>.)

The ledger. flake-ledger.yml runs daily (06:15 UTC; also workflow_dispatch), lists the last 14 days of runs of unit-tests.yml, e2e-tests.yml and android-tests.yml, downloads their flake-* artifacts and runs backend/cmd/flakeledger (collect -> report -> issues). Per (suite, test) it records {runs, failed-first-attempt, passed-on-retry, failed} and ranks by flake rate (failed-first-attempt / runs). The rendered markdown is the job summary and the ledger-report artifact; it is not committed (a daily commit to protected main would only add noise). The renderer’s output format is pinned by a golden test (backend/internal/flakeledger/testdata/ledger.golden.md; go test ./internal/flakeledger -update regenerates it).

Policy. A test with 3 or more passed-on-retry events in 14 days gets an auto-opened issue titled Flaky test: <suite> <test>, labelled flaky-test and p1, in the in-flight milestone (the open milestone with the earliest due date), naming the test and linking the runs. At most 5 are opened per day; an open issue with the same title suppresses a duplicate. Closing requires a root-cause note (what was racing or leaking, and the fix) — raising a retry count or timeout is not a resolution. Events at or before an issue’s close time do not count towards re-opening it, so a closed issue is not undone by the same events still inside the window; a recurrence opens a fresh one.

Advisory. The ledger never fails a PR and is not a required check. Its own failures (API outage, missing artifacts) are warnings in the log; a red flake-ledger.yml run is reported by nightly-failure-alert.yml like any scheduled workflow.

Verify it. Run it with workflow_dispatch against the repository’s recent history and confirm known flaky suites (the credentialStorage IndexedDB spec, the Android emulator leg) rank near the top; a deliberately time.Sleep-flaky test on a branch appears within a day of its third passed-on-retry run.

Anti-goal: coverage percentage is not the acceptance criterion

The milestone states it verbatim, and this project means it: coverage is measured where useful, but coverage percentage is not the primary acceptance criterion.

• Acceptance is layer ownership. A failure class is “covered” when its owning layer has a test that would catch it — not when a number crosses a threshold. “Where does this test go?” is answered by the layer map above; “is this failure class covered anywhere?” is answered by the ownership table.
• The coverage gate is a different mechanism. Issue #267’s codecov/patch gate (target 100% on new uncovered lines in changed files, per area: Go / vitest / JaCoCo) is a gate on changes, not an acceptance criterion for milestones. The project-wide codecov/project number is deliberately informational. Full mechanics: docs/development/coverage.md.
• Coverage ≠ correctness. A line can be 100% covered while every assertion misses what a planted bug changes — that is exactly what mutation testing measures, and why it complements rather than replaces the layers. Issue #915 found the original version of this (frontend-only, advisory, never reaching the Go safety-critical paths) insufficient to back that claim, so it is now two nightly, threshold-gated workflows rather than one advisory one:
  • stryker.yml — frontend, every src/api/*.ts module, every src/hooks/use*.ts hook and every src/utils/*.ts pure helper (issues #1482, #1624 §3; the scope was six files — the contacts / relationship-edge / life-event modules — which were already the best-tested, so the gate said little). frontend/stryker.conf.json’s thresholds.break fails the run itself on a score drop below the committed baseline. Baseline measured 2026-10-06: 79.62% overall (7,660 mutants; src/api 89.41%, src/hooks 66.35%); src/utils measured 2026-10-12: 85.07% (701 mutants). Static mutants are ignored (ignoreStatic: 6% of mutants but 58% of the run time). The ~1h45m figure was a 16-core dev box; on the 4-vCPU CI runner the api+hooks scope projects to ~5h (issue #1571: the first nightly at this scope was cancelled by the old 180-minute timeout at 53%), so the workflow runs three parallel legs, api, hooks and utils, each with a 240-minute timeout (api ~4.4k mutants, hooks ~3.3k, utils 701). Each leg derives its config from stryker.conf.json with jq and carries its own break in the workflow matrix (api 83, hooks 93, utils 79 — ~6 points under each half’s measured score); stryker.conf.json itself stays the combined local-run config. Issue #1624 §3 added the first tests for the largest untested hooks (useContactMerge, useSystemEvents, useAudit, useSeafile, useImmich), killing 121 survivors and lifting the hooks half from 66.35% to 70.12% (2026-10-13, 2277 killed / 785 survived / 186 no-coverage; 70.18% by Stryker’s formula). Issue #1662 measured the residual instead of assuming it: only 156 of 969 survivors were dependency-array candidates and 13 were logging literals; 800 were real gaps (stale-request guards, loading/error flags, || fallbacks, payload shape, unexercised branches). Hook tests now pin those; the proven-equivalent mutants carry per-site // Stryker disable comments with reasons, so ignored mutants leave the score denominator. A follow-up audit of all 113 disables found 28 that were too broad — Stryker’s disable suppresses every replacement of the named mutator, so ConditionalExpression hid the false (or true) replacement even where the reason only justified the other: the useState(true) initial values (the first render is observable — NetworkPage tells “empty” from “not loaded yet” by useCircles’ initial loading), the never-clear pollTimer guard, the never-truncate useRowKeys guard, the created?.id empty-id guards, the birthday blank guard. Those guards were rewritten without the redundant branch (unconditional clearInterval(timer ?? undefined) / keys.length = length) or killed by tests (useFirstRenderState.test.tsx records every render’s value), and 88 disables remain, each equivalent for all of its replacements. A per-file re-measure (never the full leg locally) of the 22 touched hooks, substituted into the CI hooks report, estimates the hooks half at 99.18% (3134 killed / 3160 valid; 26 survivors). Rule for new disables: only when every replacement of the mutator is equivalent; otherwise restructure so the equivalent branch disappears. Directive trap: Stryker only honors disable next-line as a leading comment of a node that starts on the mutant’s line — a comment above a closing }, []); attaches to nothing and is silently ignored, so for deps arrays pass the array as its own argument line with the comment above it. Single-site request counters (requestRef.current + 1) are equivalent, but counters shared by several call sites (loadMore + refetch + reset) are not: pin them with an interleaving test. Locally run a slice with npx stryker run --mutate src/api/notes.ts (the dry run still executes the whole suite, ~5 min).
  • go-mutation.yml — backend, gremlins against the paths where silent data loss lives: migration/upgrade and backup/restore (database), data-integrity invariants (atrest), delete cascade (the two files contact_controller.go/admin_user_controller.go name in backend trap 6), import ingestion (services’ import-source files), and the three exporters (vcard3, vcard4, jscontact). Issue #1491 added the decision-heavy code whose silent breakage is worst: auth middleware (middleware-auth: auth.go, admin.go, idempotency.go, login_lockout.go), the SSRF guard (ssrf: all of httputil), merge (services-merge, controllers-merge), date/threshold arithmetic (schedule-math: cadence, reminder, data-decay, occasion), and sync reconcile (sync-reconcile: contact_sync_service.go; carddav-backend, caldav-backend: the DAV backends). gremlins v0.6.0 mutates comparisons, arithmetic and increments only — it does not delete an if !ok arm or an AND user_id = ? SQL fragment, so ownership scoping stays covered by routes/ownership_matrix_test.go, not by a mutation leg. The complete scope and each leg’s threshold (with the baseline run it ratchets from) live in backend/internal/mutationscope.Scopes, generated into backend/.gremlins/*.yaml by cmd/genmutationscope — regenerate after adding or removing a file in a scoped package (controllers/services are far larger than the scope, so the exclude list is mechanical, not hand-maintained). Issue #1624 added services-auth (TOTP/session/reset/ API-token/WebAuthn/OIDC/feed), sensitivity-filter (the sensitivity = normal query filters on the services query sites) and sensitivity-filter-records (the share/sync read projections in models/contact_record.go/field_selection.go — the filters the controller share/read entry points thread an include_sensitive flag into), nineteen legs total. Issue #1946 added services-backfill (the one-shot v1.5.0 startup rewrites, services/gift_date_backfill.go and services/hobby_preference_backfill.go, where a flipped condition rewrites data and the ledger then makes it permanent) and folded models/contact_personal_info.go’s per-entry personalInfo filter and inbound remote merge into sensitivity-filter-records — twenty legs total.

  Both stay nightly + manual-dispatch rather than per-PR: mutation testing is O(test suite × mutant count), too slow to gate every diff. Neither is a release gate (docs/development/release-gates.md) for the same reason — the ratchet catches a regression by morning, not by blocking the PR that introduced it.

Mutation tools evaluated and not adopted (issue #1624). Two cross-cutting gaps are recorded rather than fixed:

• Statement/branch-deletion mutators. gremlins v0.6.0 only mutates comparisons, arithmetic and increments; it never deletes an if !ok arm or a WHERE user_id = ? filter. The one Go tool with statement/remove, branch/if and branch/else (go-mutesting) is unmaintained since 2021 and panics at startup on this repo’s Go toolchain — a nil *go/types.StdSizes (its vendored golang.org/x/tools never populates types.Config.Sizes), on the middleware-auth and controllers-delete-cascade targets and on any file — so it is not viable without a maintained fork. ooze is maintained but is a library, not a CLI, and its virus set (comparisons/arithmetic/increments plus loop break/condition/range) does not include statement/branch deletion, so it does not close this gap. Declined; ownership scoping stays guarded by routes/ownership_matrix_test.go.
• Android (pitest). There is no mutation leg for the Android core modules. The vanilla gradle-pitest-plugin targets JVM (sourceSets.main) projects and does not support AGP’s com.android.library modules, so it cannot target the pure-logic :core:model. The Android-capable free plugin (pl.droidsonroids.pitest 0.2.18) fails to apply on this repo’s Gradle 9.5 — “Cannot mutate configuration container for buildscript of project ‘:core:model’ using create(String). Configurations cannot be added or removed from the buildscript configuration container” — and Arcmutate’s Kotlin/Android support is commercial. Declined; Android logic keeps its JUnit/Robolectric suites and the coverage ratchet.

Determinism and isolation (the “test infrastructure” acceptance criteria)

The layers above presuppose tests that are deterministic and isolated from developer-specific state:

• Real schema, per test. internal/dbtest.New(t) hands each test an isolated byte copy of the per-binary migrated template (issue #632); the database package uses database.InitDB(filepath.Join(t.TempDir(), "x.db")). No test ever depends on a developer’s working DB.
• Fresh artifact per e2e run. The Playwright, instrumented-Android, and Schemathesis suites boot docker-compose.test.yml (down -v teardown) — a throwaway DB, never production data.
• Deterministic identities. global-setup.ts seeds the TEST_USER; tests do not depend on local state or ordering (Playwright reuses a storageState captured in the setup project).

Order-dependence pass (issue #1492)

CI otherwise runs tests in declaration order with -count=1, which hides a test that only passes because an earlier one left state behind (a package-level global, the process-wide metrics registry, a rate limiter) or that cannot run twice in one process. The nightly schedule run of unit-tests.yml’s backend-tests job therefore adds a second step, Order-dependence pass (shuffle + count=2, nightly only), per leg: go test <leg pkgs> -run <leg shard> -race -shuffle=on -count=2.

• Zero retries. A red pass is the signal; nightly-failure-alert.yml opens the issue (unit-tests.yml is already a registered scheduled workflow, so nightlyalertcheck needs no change).
• Reproducing. go test prints -test.shuffle <seed>; the step copies the seed, the leg’s package list and its -run shard regex into the job summary. The sharded legs (controllers-1/-2, services-1/-2) shuffle within their shard only, so the seed alone does not reproduce — use all three: cd backend && go test <pkgs> -run '<shard>' -race -shuffle=<seed> -count=2.
• Skipped leg. property (rapid/largedata/contactgen) owns no cross-test state and doubling its 8000-iteration nightly depth would blow the job timeout. RAPID_CHECKS/ADDRESS_PROPERTY_ITERATIONS are pinned to 200 for this pass — it probes ordering, not search depth.
• Fix at the source. A new failure is fixed with a reset helper, a t.Cleanup that restores the global, or injection (#1493) — never by pinning order or -p 1. A test asserting on a process-global counter asserts the delta across its own actions (see sample in middleware/metrics_test.go).
• Locally: cd backend && go test ./<pkg> -shuffle=on -count=2 (one package at a time; the full tree is heavy).
• Follow-up (not done here): after 30 clean nights, add -shuffle=on (single count) to the per-PR run so the class stays closed.

Bug disposition: a test that catches a bug owns it

The layers this page describes are the gate for every milestone from v0.6.3 forward, so a bug they catch must not wait for its “natural” domain milestone to be fixed. Standing policy (2026-08-28, recorded in the v0.6.3 milestone description and gate #533):

• File or fix it in the milestone where it was found. Any bug a test layer reveals is fixed as part of that milestone, or filed as an issue in that milestone with an explicit disposition — never silently deferred to an arbitrary later milestone. The milestone that builds the tests is the milestone that owns the bugs they catch; that is the whole reason the layers land before the domains they validate.
• A failing test in these layers blocks a milestone. The gate #533 standing criterion is that these layers’ tests are green on the merge commit of every milestone from here forward. A bug the tests catch keeps a milestone blocked until it is fixed — it does not get “exempted because it was found late.”

References

• Issue #429 (TEST-01) — this pyramid; #533 — the v0.6.3 milestone gate.
• Issue #264 — path gating; .github/filters.yaml is the source of truth.
• Issue #267 — the diff-based coverage gate; docs/development/coverage.md.
• ADR-0003 — golden fixtures as the external test oracle; docs/golden-fixtures/.
• TEST-02 (#430) — the canonical pathological dataset that the DB, migration, and interop layers will consume; MIG-01 (#436) — versioned migration fixtures.
• TEST-03 (#431) — the semantic round-trip comparison (pulled forward into v0.6.3 so TEST-07 #435 can consume it).
• TEST-08 (#680) — differential testing against independent, pinned reference implementations (vobject / calcard / golang-ical); the format-layer counterpart of #496’s reference server. Divergence pins live in backend/differential/pins_*.go.
• DATA-03 (#443) — repeated-conversion / idempotence-after-first-conversion testing: each format’s conversion run N times over the TEST-02 fixture (serialized form byte-identical from pass 2 onward, successive-pass diffs empty, passthrough byte-stable and never growing, diagnostics not accumulating), cross-format chains (vCard 4 → vCard 3 → vCard 4) converging rather than degrading, and the same property wired into the TEST-07 generative suite. The single round trip proves one conversion is faithful; this proves conversions compose (the property CardDAV sync runs on).
• DEPLOY-01 (#450) — the release/install smoke layer; TEST-06 (#434) — failure injection; PERF-01 (#468) — performance at scale.
• Issue #257 — contract fixtures; #266 — the spec-derived contract fixture.