# Verifying Release Artifacts

Every tagged release (`docker-publish.yml`) publishes three Docker images and a signed Android
APK. This page is the operator-facing "how do I check this is genuine" guide: exact commands to
verify a downloaded binary/container actually came from this repository's CI, unmodified. No
supply-chain background is assumed.

It documents what's already in place (issues #272, #328) — it introduces no new signing. See
`docs/security/asvs-l2.md` (rows 1.14.2, 10.3.1, 14.2.5) for how this fits the ASVS checklist.
OpenSSF Scorecard's `Signed-Releases` check (issue #355) is satisfied because the release carries
the APK's cosign bundle as an asset (`mycorrhizal-apk.sigstore.json`). That check is purely
filename-based: it scans release assets for signature extensions (`.asc`, `.sig`, `.minisig`,
`.sigstore`, `.sigstore.json`, ...) and ignores everything else — the attestations API, the OCI
registry, and workflow artifacts.

## How a release is cut

`release.yml` (REL-06, `workflow_dispatch`) is the whole release process. Inputs: the version
(e.g. `v0.6.6`, or `v1.0.0-rc.1` for a release candidate); `ref` (`main` for a final release,
`release/vX.Y.0` for an RC); `dry_run` (run every gate + regenerate the fixture, make no
commit/push/tag — this is how the workflow is exercised without cutting a release, including
against the last shipped version); `ack_asvs_current` (a reason to proceed when the ASVS/MASVS
re-verification row is absent — recorded, not silent). `release-dry-run.yml` dispatches it with
`dry_run: true` against the last shipped version weekly and on demand, so this rehearsal is
actually run by CI rather than only documented (issue #929).

It, in three jobs (ADR 0021, [composition](../adrs/0021-release-validation-composition.md)):

1. **`preflight`** — validates the version string, that no such tag exists (a registered-but-untagged
   version is *resumed*, issue #1142), that the checkout is the true tip of `ref`, and runs
   `go run ./cmd/citecheck` and `go run ./cmd/releasegatecheck`.
2. **`validate`** — calls the reusable
   [`release-validate.yml`](https://github.com/DrewBrunning/mycorrhizal-crm/blob/main/.github/workflows/release-validate.yml),
   which **composes every gate** — the `release_gate: true` per-PR checks and the release-tier suites —
   as the same workflow files the per-PR/`push: main`/schedule paths run, with real `needs:`
   dependencies. There is no polling. It also enforces the ASVS/MASVS re-verification row (issue #608)
   and the per-release adversarial delta (issue #953) for a final release, each with a recorded
   `ack_*` dispatch escape.
3. **`release`** — regenerates and commits the schema fixture (final only; an RC skips it, see below),
   runs the schemafixture/genschema/releaselist test gates, and pushes the tag with a GitHub App
   token. `dry_run: true` runs all of the above except the commit/push/tag.

`create-release` in `docker-publish.yml` then owns the GitHub Release and **every asset on it** —
including `release-metadata.json` (ADR 0021, issue #1163), which is regenerated from the tagged tree
there. No workflow waits for a Release another workflow creates.

The tag push triggers `docker-publish.yml`, which builds and signs everything listed below and
creates the GitHub Release. That hand-off works only because the push uses a **GitHub App token**
(repo variable `RELEASE_APP_ID` + secret `RELEASE_APP_PRIVATE_KEY`, App on `main`'s
branch-protection bypass list) — a tag pushed with the default `GITHUB_TOKEN` cannot trigger
another workflow. The App token is scoped to `contents: write` and is used by this one
`workflow_dispatch`-only workflow; there is no PR-triggered path to it.

Because the tag points at a real commit on `main` (the one carrying the dump), the
`schema-fixture-gate` in `docker-publish.yml` passes and source↔release correspondence (below)
is exact — there is no post-review "move the tag" step.

**The workflow is re-entrant before the tag exists (issue #1142).** If a run fails after the
schema fixture is committed but before the tag is pushed — the common case being a composed gate that
fails — fix the cause on `main` and re-dispatch `release.yml` with the same version. `preflight`
consults the durable **release-readiness artifact** the previous attempt uploaded (issue #1164) and,
when one exists with no tag, *resumes* instead of refusing; the release commit becomes the checked-out
tip of `main` (so the fix is in the release) and the full composed battery re-runs before the tag. If
the artifact has aged out (90-day retention), the permanent git registration is the backstop, and the
two are cross-checked fail-closed: an artifact that records the fixture as registered while
`SupportedReleases` disagrees refuses the cut. Once a tag exists, `release.yml` still refuses it (a
released tag is never moved): if `docker-publish.yml` fails after the tag is pushed, re-run it from
its own **Run workflow** button with the `tag` input.

### Release candidates and promotion (RC-02)

An `-rc.N` version is a **release candidate** (full policy:
[`docs/release-candidate-process.md`](../release-candidate-process.md)). `release.yml` composes the
same gate battery for it, but skips the schema-fixture registration (registered at promotion, not
per-RC) and the ASVS §10-row gate (a final-release obligation). `docker-publish.yml` marks the RC's
GitHub Release a **pre-release** and never `make_latest`, keeping it off the in-app update check
(`/releases/latest` excludes pre-releases) and Obtainium.

**Promotion** (`promote-rc.yml`, `workflow_dispatch`, input `rc_tag`) ships the *tested* artifact,
not a rebuild: it re-tags the RC's container images **by digest**, copies every Release asset
byte-for-byte (verified against the RC's `SHA256SUMS`), registers the final schema fixture against
the RC's tree, merges `release/vX.Y.0` back into `main`, and pushes the final `vX.Y.Z` tag with
`GITHUB_TOKEN` so `docker-publish.yml` does **not** run. `promotion-metadata.json` on the final
Release records `digest_rc == digest_final` for every image — the check that promotion copied
rather than rebuilt (`promote-rc.yml` fails if any digest differs). A promoted image therefore
carries the RC pipeline's original `cosign` signature and GH build-provenance attestation (both
digest-scoped, still valid) **plus** an additional `cosign` signature with the `promote-rc.yml`
identity.

**One pinning exception.** Every other Action in `.github/workflows/` is pinned to a commit SHA.
The `apk-provenance` job's `slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0`
is referenced by **semver tag** because the trusted-builder model requires the reusable workflow
resolve its own ref to establish the builder identity — a commit pin is unsupported and breaks
the provenance this job produces. This is a permanent, accepted exception:
`zizmor`'s `unpinned-uses` finding is suppressed inline with `# zizmor: ignore[unpinned-uses]`,
and OSSF Scorecard's Pinned-Dependencies check (which has no suppression mechanism) flags it at
sub-score 9 by design — the reasoning is recorded in
[`docs/dependency-upgrade-policy.md`](../dependency-upgrade-policy.md). Generator version bumps
are a deliberate, reviewed tag change.

## What's attached to a release, and what it proves

| Artifact | Signal | Proves | Expires? |
|---|---|---|---|
| Docker images (all-in-one, `-backend`, `-frontend`) | cosign keyless signature (pushed to the registry alongside the image) | The image was signed by this repo's `docker-publish.yml` via GitHub's OIDC identity | No — lives in the registry as long as the image does |
| Docker images | SLSA build provenance, two forms: GitHub-native attestation + buildkit in-toto attestation | Which commit/workflow run produced this exact digest | No — GitHub attestation is permanent; buildkit provenance lives in the registry |
| Docker images | SBOM (SPDX, buildkit-embedded referrer) | Full dependency list for the exact image you pulled | No — lives in the registry |
| Docker images | Standalone signed SBOM (SPDX + CycloneDX, cosign-signed) | Same dependency list, as a portable file + signature | **Yes — 30-day GitHub Actions artifact retention** |
| Android release APK | Keystore signature (`SIGNING_*` secrets) | The APK is installable and matches every other release signed with the same key (Android's own trust mechanism) | No |
| Android release APK | GitHub-native SLSA build provenance | Which commit/workflow run built this exact APK; shows a "Verified" badge on the Release page | No — permanent |
| Android release APK | cosign keyless co-signature (additive, does not replace keystore signing) | Independent Sigstore-backed verifier on top of the GitHub attestation; what Scorecard's `Signed-Releases` check counts for the 8/10 tier | No — attached to the Release as `mycorrhizal-apk.sigstore.json` (a copy is also kept as a 30-day workflow artifact) |
| Android release APK | SLSA build provenance from the `slsa-github-generator` reusable workflow (`apk-provenance` job) | A verifiable in-toto SLSA statement over the APK's sha256, signed keyless; what Scorecard's `Signed-Releases` check counts for the **10/10** tier | No — attached to the Release as `mycorrhizal-apk.intoto.jsonl` |
| All release assets | `SHA256SUMS` — a plain `sha256sum` manifest over every asset on the Release, generated last by `verify-release-assets` | One file to check the integrity of everything you downloaded from the Release | No — attached to the Release as `SHA256SUMS` |
| The release run itself | `release-metadata.json` — version, migration version, source revision, dry-run/resumed flags, gate results, and the residual-risk statement (open accept items, dependency-exception expiry, ASVS/MASVS exception counts) | Which commit `release.yml` cut the release from, which gates it verified, and what was accepted on the way (issue #953) | No — attached to the Release (also a 90-day workflow artifact) |
| The candidate decision | `release-readiness.json` — version, source commit/ref, migration version, per-gate results, `fixture_registered`, the two ASVS/adversarial acknowledgement reasons, and the residual-risk statement (issue #1164) | What `release.yml`'s composed `validate` battery recorded before it tagged; the durable state a re-dispatch resumes from | No — attached to the Release by `docker-publish.yml` (also a 90-day workflow artifact) |

The one "expires" row is a workflow *run* artifact (`actions/upload-artifact`), not a GitHub
Release asset — it is only downloadable from the specific `docker-publish.yml` run's Actions
page, and only for 30 days after that run. For a release older than that, skip that check and
rely on the permanent ones (cosign image signature, both provenance forms, the APK's cosign
bundle on the Release, source↔release correspondence below) — they cover the same ground.

## Prerequisites

Install two CLIs:

```sh
# cosign — signature and SBOM verification. **v3 or newer**: the release
# pipeline signs images with cosign v3 (sigstore-go / OCI 1.1 referrers), and
# cosign v2 does not discover those signatures — `cosign verify` reports
# "no signatures found" against a correctly-signed release (issue #1160).
brew install cosign            # macOS
# or: go install github.com/sigstore/cosign/v3/cmd/cosign@latest
# or download a release binary: https://github.com/sigstore/cosign/releases

# GitHub CLI — attestation verification
brew install gh                # macOS
# or: https://github.com/cli/cli#installation
```

`gh attestation verify` works unauthenticated against this public repo, but `gh auth login` first
avoids GitHub's unauthenticated API rate limit if you're verifying more than a couple of times in
a row.

## Verifying a Docker image

Replace `<TAG>` with the release you're deploying, e.g. `0.6.1`. The published
image tag drops the leading `v` from the git tag (`v0.6.1` → `0.6.1`), so
`docker pull …:v0.6.1` would 404.

**1. Pull the image and pin the digest** (a tag is mutable; the digest is what's actually signed):

```sh
docker pull ghcr.io/drewbrunning/mycorrhizal-crm:<TAG>
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ghcr.io/drewbrunning/mycorrhizal-crm:<TAG>)
echo "$DIGEST"   # ghcr.io/drewbrunning/mycorrhizal-crm@sha256:...
```

(For the split images, use `ghcr.io/drewbrunning/mycorrhizal-crm-backend:<TAG>` or
`...-frontend:<TAG>` instead.)

**2. Verify the cosign signature:**

```sh
cosign verify "$DIGEST" \
  --certificate-identity-regexp '^https://github\.com/DrewBrunning/mycorrhizal-crm/\.github/workflows/(docker-publish|promote-rc)\.yml@refs/tags/v' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
```

A successful verification prints the signed payload and exits `0`. A tampered or unsigned image
fails with `Error: no matching signatures`.

The `--certificate-identity-regexp` pins the signature to **a release workflow on a tag ref**
(#513): only `docker-publish.yml` (a normal release) or `promote-rc.yml` (a promoted RC), running
on a `refs/tags/v*` ref, can produce a Sigstore certificate whose identity matches. An identity
mismatch means the signature was produced by some other workflow — treat it as a red flag, not a
version-skew nuisance. (Releases signed
before this pin landed carry the older repo-wide identity; for those, loosen the regexp to
`https://github\.com/DrewBrunning/mycorrhizal-crm/` and check the run manually.)

**3. Verify SLSA build provenance** (GitHub-native attestation — shows which commit and workflow
run produced this digest):

```sh
gh attestation verify "oci://$DIGEST" -R DrewBrunning/mycorrhizal-crm
```

**4. Inspect the buildkit-embedded provenance and SBOM directly** (a second, independent source
for the same facts, stored as OCI referrers in the registry rather than GitHub's attestation
store):

```sh
docker buildx imagetools inspect ghcr.io/drewbrunning/mycorrhizal-crm:<TAG> --format '{{ json .Provenance }}'
docker buildx imagetools inspect ghcr.io/drewbrunning/mycorrhizal-crm:<TAG> --format '{{ json .SBOM }}'
```

The provenance JSON's `predicate.invocation` / `predicate.materials` fields name the exact source
commit — cross-check it against the tag as described below.

## Verifying the Android release APK

Download `app-release.apk` from the release's GitHub Release page
(`https://github.com/DrewBrunning/mycorrhizal-crm/releases/tag/<TAG>`).

**1. SLSA build provenance** (works for every release, indefinitely):

```sh
gh attestation verify app-release.apk -R DrewBrunning/mycorrhizal-crm
```

The Release page itself also shows a "Verified" badge next to the asset when this attestation is
present.

**2. cosign co-signature** (additional Sigstore-backed signal): download
`mycorrhizal-apk.sigstore.json` from the release's GitHub Release page (attached as a release
asset; a copy is also kept as a 30-day `apk-cosign-signature` workflow artifact on the
`docker-publish.yml` run). It is cosign v3's standardized bundle — cert, signature, and
transparency log entry in one file — then:

```sh
cosign verify-blob \
  --bundle mycorrhizal-apk.sigstore.json \
  --certificate-identity-regexp '^https://github\.com/DrewBrunning/mycorrhizal-crm/\.github/workflows/(docker-publish|promote-rc)\.yml@refs/tags/v' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  app-release.apk
```

**3. SLSA in-toto provenance** (`mycorrhizal-apk.intoto.jsonl`, attached to the Release): a real
SLSA statement produced by the `slsa-framework/slsa-github-generator` reusable workflow, not a
renamed attestation. Verify the APK against it with [`slsa-verifier`](https://github.com/slsa-framework/slsa-verifier):

```sh
slsa-verifier verify-artifact app-release.apk \
  --provenance-path mycorrhizal-apk.intoto.jsonl \
  --source-uri github.com/DrewBrunning/mycorrhizal-crm
```

**4. Installability** — the keystore signature that actually lets Android install/upgrade the
APK is separate from all of the above and is checked automatically by the OS (or by `apksigner
verify app-release.apk` if you want to confirm it yourself); it's what proves this release was
built with the same signing key as every prior release, so an update can't be substituted by
someone without that key.

## Verifying the SHA256SUMS manifest

Every Release carries a `SHA256SUMS` file listing the sha256 of every other asset. After
downloading the assets you want plus `SHA256SUMS` into one directory:

```sh
sha256sum --check --ignore-missing SHA256SUMS
```

`SHA256SUMS` is not itself signed; it is a convenience over the per-artifact signatures above,
which are the real integrity roots. Cross-check at least one asset's line against its cosign /
SLSA verification, then trust the manifest for the rest.

## Verifying source↔release correspondence

The three checks above establish "this artifact was produced by this repo's CI." A separate,
useful question is "which commit, exactly" — and whether the thing you're *running* matches the
tag you think you deployed.

Every image is stamped at build time with the commit it was built from
(`backend/buildinfo/buildinfo.go`), exposed on the running instance:

```sh
curl -s https://your-instance.example.com/health | jq '{version, commit, build_date}'
```

Compare the `commit` field (a 12-character prefix) against the tag's actual commit on GitHub:

```sh
git ls-remote https://github.com/DrewBrunning/mycorrhizal-crm.git refs/tags/<TAG>
```

(Release tags are lightweight and point directly at a commit on `main` — the one `release.yml`
pushed, carrying that release's schema dump. `refs/tags/<TAG>^{}` still works and resolves to the
same commit.) The two should share the same prefix. This same commit also appears in the
provenance JSON pulled in step 4 above, so all three — the running binary, the git tag, and the
signed provenance — should agree.

## Verifying a standalone SBOM

Both the per-release SBOMs (`docker-publish.yml`, one per image) and the continuous main-branch
SBOM (`syft-sbom.yml`, one per merge to `main`) are uploaded as cosign-signed workflow artifacts —
**30-day retention**, unlike the APK cosign bundle above, which is attached to the Release
permanently. From the relevant workflow run's Actions page, download the SBOM artifact bundle
(`sbom-signatures[-backend|-frontend]` for a release, `sbom-signatures` for a main-branch SBOM)
and verify:

```sh
cosign verify-blob \
  --bundle sbom.spdx.json.sigstore.json \
  --certificate-identity-regexp '^https://github\.com/DrewBrunning/mycorrhizal-crm/\.github/workflows/(docker-publish|syft-sbom)\.yml@refs/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  sbom.spdx.json
```

Past the 30-day window, use the registry-embedded SBOM instead (step 4 under "Verifying a Docker
image" above) — it's the same document, just without a portable standalone signature.

## Reproducibility

The checks above prove an artifact came from this repo's CI. A related question — *can I rebuild
the source and get the same bytes?* — has its own page:
[Reproducible builds](reproducible-builds.md) (REL-04, issue #448). In short: the Go server
binary is byte-reproducible. `.github/workflows/reproducibility.yml`'s `go-binary` job proves
this is path-independent (double-built from two different paths on the same runner) — gated
on every PR touching the backend/build inputs (`backend/**`, the Dockerfiles, `docker/**`; it is
path-filtered, not every PR). A separate `go-binary-independent-rebuild` job goes further with a
true independent rebuild across two different runners, substantiating the end-to-end claim, but
only weekly (and on manual dispatch) — see issue #947. The `linux/amd64` image's config + layer digests are
double-build-compared every run (not yet a hard gate); the multi-arch manifest digest and the
signed Android APK are not bit-reproducible by design, and provenance is the mitigation. That
page carries the exact rebuild-and-compare commands and the full per-artifact table.