Learning from what CockroachDB's Pebble repo actually does in CI

2026-08-25 (4w ago)5 views

#ci#testing#go#github-actions

I went digging through cockroachdb/pebble's .github/workflows because I wanted to learn from a CI setup that's been battle-tested for years on a storage engine, where correctness bugs are the expensive kind. The YAML files mostly just trigger jobs and call scripts. The actual test logic lives in the Makefile and scripts/ directory. Here's what I found, roughly in order of how much I want to steal it.

Fast PR checks, slow nightly checks

There's a fast PR gate (ci.yaml) and a slow nightly sweep (nightlies.yaml + friends). The PR gate runs on every push/PR and finishes in 5-10 minutes: build+test on Linux, lint (go mod tidy + format check), a no-invariants build, a no-cgo build, the race detector, and a macOS build. Anything expensive (ASAN, MSAN, cross-arch builds, full stress runs, cross-version compatibility) only runs nightly.

The nightly workflows aren't copy-pasted per branch. Pebble maintains several release branches (crl-release-25.4, 26.1, 26.2, etc.) and each has its own nightlies-XX.Y.yaml, but all of them just call the same reusable workflow files (tests.yaml, stress.yaml, instrumented.yaml, crossversion.yaml, s390x.yaml, cockroach-go.yaml) via workflow_call, passing in sha, go_version, and file_issue_branch as inputs. One test definition, six release branches wrapping it with different parameters.

# nightlies.yaml (master)
tests:
  strategy:
    matrix:
      go: [ '1.26' ]
  uses: ./.github/workflows/tests.yaml
  with:
    sha: ${{ github.sha }}
    file_issue_branch: 'master'
    go_version: ${{ matrix.go }}

Every nightly job also ends with a step like this:

- name: Post issue on failure
  if: failure() && inputs.file_issue_branch != ''
  uses: $/.github/actions/post-issue
  with:
    title: "${{ inputs.file_issue_branch }}: nightly ${{ github.job }} failed"
    labels: "C-test-failure"

Nobody's watching the Actions tab for a 2am nightly run I guess, so instead of relying on that, a failure files (or updates) a GitHub issue labeled C-test-failure. If your CI has jobs that run unattended, this is the pattern: push failures into wherever your team actually looks, not into a dashboard nobody opens.

Metamorphic testing

Pebble's flagship testing technique lives in internal/metamorphic. Instead of writing TestFoo with a fixed expected output, it generates random sequences of database operations and checks invariants, like "two different code paths that should be equivalent actually agree," rather than "this call returns exactly X." For a storage engine, "correct" often isn't a fixed value, it's a consistency property, so this fits much better than table-driven tests would.

Two things build on top of it:

• make stressmeta runs TestMeta$ under the cockroachdb/stress tool with 3 parallel workers, for as long as you let it, hunting for rare interleavings.
• make crossversion-meta compiles the metamorphic test binary from both the latest release branch and HEAD, then runs them together:

crossversion-meta:
	$(eval LATEST_RELEASE := $(shell git branch -r --list '*/crl-release-*' | sort | tail -1))
	git checkout ${LATEST_RELEASE}; \
		go test -c ./internal/metamorphic -o './internal/metamorphic/crossversion/${LATEST_RELEASE}.test'; \
		git checkout -; \
		go test -c ./internal/metamorphic -o './internal/metamorphic/crossversion/head.test'; \
		go test -tags invariants -run 'TestMetaCrossVersion' ./internal/metamorphic/crossversion \
			--version '${LATEST_RELEASE},${LATEST_RELEASE},${LATEST_RELEASE}.test' \
			--version 'HEAD,HEAD,head.test'

This is how you fuzz for on-disk-format backwards-compatibility bugs without hand-writing "test that v1 can read v2's files": you generate random operations across two builds simultaneously and let the fuzzer find the incompatibility for you. This is my first time seeing something of the sort but then I guess it might be more common than I thought.

Testing the crossversion test itself

The part that impressed me most: scripts/crossversion_smoke_test.sh is a test of the crossversion test. It:

1. Runs the crossversion test with no bugs, which should pass.
2. Applies a patch (crossversion_smoke_test.patch) that deliberately breaks backwards compatibility (changes a metaindex encoding version constant from v6 to v7).
3. Confirms the single-version buggy build still passes on its own (the bug should only break cross-version compat, not the build itself).
4. Reverts the patch, then runs the cross-version test again with the bug re-applied, up to 10 times with different seeds, expecting at least one failure.
5. If the crossversion test never fails despite the injected bug, the smoke test itself fails, meaning the safety net has silently stopped working.

# from crossversion_smoke_test.sh
if [ "$PHASE2_FAILED" = false ]; then
    error "SMOKE TEST FAILED!"
    error "The crossversion test PASSED in all ${MAX_ATTEMPTS} attempts despite the intentional bug."
    exit 1
fi

If the crossversion test never fires under normal operation, you won't notice when it silently breaks.

Stress testing new tests before merging

On every PR (ci.yaml's stress-new-tests job), scripts/stress-new-tests.sh diffs against the base branch, pulls out only the newly added func Test... lines, builds a regex from them, and stress-runs just those new tests for 10 minutes:

added_tests=$(git diff --no-ext-diff "$BASE_BRANCH" --unified=0 -- "$pkg"/*.go \
  | grep '^+func Test' | awk '{print $2}' | cut -d'(' -f1 | sort -u)
 
go test --tags invariants \
  --exec 'stress -p 2 --maxruns 1000 --maxtime 10m --timeout 2m' \
  -v -run "^($regex)$" "$pkg"

Full-suite stress (scripts/stress.sh) only runs nightly, and it's tuned per package: state-heavy packages like metamorphic, sstable, and wal get 30 minutes at 75% parallelism, everything else gets 5 minutes at 100%. New code gets hammered for flakiness before it merges, the whole codebase gets hammered nightly, and no PR has to wait for a full-suite stress run to finish. If I only copy one thing from this whole post, it's this: a few lines of shell that catch racy or flaky new tests before they land, without slowing down every PR.

Running ASAN and MSAN

Beyond -race, the nightly instrumented.yaml workflow runs ASAN and MSAN, AddressSanitizer and MemorySanitizer, the C/C++ memory error detectors. Both matter here because Pebble uses cgo; -race only sees Go code:

testasan: testflags += -asan -timeout 20m
testasan: TAGS += slowbuild
testasan:
	ASAN_OPTIONS=detect_leaks=0 go test -tags '$(TAGS)' ...
 
testmsan: export CC=clang
testmsan: testflags += -msan -timeout 20m
testmsan: TAGS += slowbuild
testmsan: test

What I'm stealing

The stress-the-diff script and the workspace-dirty check are both a handful of shell lines, no new infrastructure, and they catch a real class of bug (flaky new tests, forgotten codegen) before merge. Those are going in my own projects first. The metamorphic and crossversion machinery is a bigger investment, but it's worth understanding if I ever build something where "correct" means "internally consistent" rather than "matches a fixture."