Build your own test runner, and what go test really does

go test feels like magic in a way the rest of Go does not. There is no registration, no decorator, no configuration file. You write a function whose name starts with Test, and it runs.

This chapter is how. By the end you will have a runner that discovers tests, builds a binary, runs them in parallel with timeouts and isolation, reports failures usefully, and emits machine-readable output. And you will know precisely which of Go's conveniences are compiler features and which are ordinary code you could have written.

The trick: it writes a program

go test does not interpret your test files. It *generates a main package* that imports your package, lists your test functions in a slice, and callstesting.Main with them. Then it compiles that program and runs it.

// Roughly what go test generates, simplified
func main() {
    tests := []testing.InternalTest{
        {Name: "TestAdd", F: pkg.TestAdd},
        {Name: "TestSub", F: pkg.TestSub},
    }
    testing.Main(matchString, tests, benchmarks, examples)
}

Step one: discovery

Parse the package with go/parser and find the functions. This is the samego/ast work grit sync does to read models, so the shape should be familiar:

func discover(dir string) ([]Test, error) {
    fset := token.NewFileSet()
    pkgs, err := parser.ParseDir(fset, dir, isTestFile,
        parser.ParseComments)
    if err != nil {
        return nil, err
    }

    var out []Test
    for _, pkg := range pkgs {
        for path, file := range pkg.Files {
            for _, decl := range file.Decls {
                fn, ok := decl.(*ast.FuncDecl)
                if !ok || !isTestFunc(fn) {
                    continue
                }
                out = append(out, Test{
                    Name: fn.Name.Name,
                    File: path,
                    Line: fset.Position(fn.Pos()).Line,
                })
            }
        }
    }
    sort.Slice(out, func(i, j int) bool {
        return out[i].Name < out[j].Name
    })
    return out, nil
}

isTestFunc is the rule, written out:

func isTestFunc(fn *ast.FuncDecl) bool {
    if fn.Recv != nil {                     // not a method
        return false
    }
    if !strings.HasPrefix(fn.Name.Name, "Test") {
        return false
    }
    // TestXxx: the character after "Test" must not be lowercase,
    // so Testing() is not a test and TestAdd is.
    rest := fn.Name.Name[len("Test"):]
    if rest != "" {
        r, _ := utf8.DecodeRuneInString(rest)
        if unicode.IsLower(r) {
            return false
        }
    }
    return hasSignature(fn, "*testing.T")
}

That lowercase rule is real and catches people: a function called Testing is not a test, and Testadd is not either.

Step two: generate and build

const mainTmpl = `package main

import (
    "os"
    "testing"

    target "{{.ImportPath}}"
)

func main() {
    tests := []testing.InternalTest{
{{range .Tests}}        {Name: "{{.Name}}", F: target.{{.Name}}},
{{end}}    }
    m := testing.MainStart(deps{}, tests, nil, nil, nil)
    os.Exit(m.Run())
}
`

Write it to a temporary directory, then:

go build -o runner.test ./tmpdir

Notice what this means: your test runner is a compiler front end. It produces Go source and hands it to the Go toolchain, which is how go test gets types, build tags and vet checks for free.

Step three: running, in parallel, in isolation

func (r *Runner) Run(ctx context.Context, tests []Test) Report {
    sem := make(chan struct{}, r.Parallel)
    results := make([]Result, len(tests))
    var wg sync.WaitGroup

    for i, t := range tests {
        wg.Add(1)
        go func(i int, t Test) {
            defer wg.Done()
            sem <- struct{}{}
            defer func() { <-sem }()

            results[i] = r.runOne(ctx, t)
        }(i, t)
    }
    wg.Wait()
    return report(results)
}

Each test runs as a subprocess invocation of the built binary with-test.run=^TestName$:

func (r *Runner) runOne(ctx context.Context, t Test) Result {
    ctx, cancel := context.WithTimeout(ctx, r.Timeout)
    defer cancel()

    dir, err := os.MkdirTemp("", "testrun-")
    if err != nil {
        return Result{Test: t, Err: err}
    }
    defer os.RemoveAll(dir)

    cmd := exec.CommandContext(ctx, r.Binary,
        "-test.run=^"+t.Name+"$", "-test.v")
    cmd.Dir = dir
    var out bytes.Buffer
    cmd.Stdout, cmd.Stderr = &out, &out

    start := time.Now()
    err = cmd.Run()

    return Result{
        Test:     t,
        Passed:   err == nil,
        Output:   out.String(),
        Duration: time.Since(start),
        TimedOut: ctx.Err() == context.DeadlineExceeded,
    }
}

A process per test is heavier than go test's goroutine per test, and it buys two things worth having: a panic kills one test rather than the run, and a test that leaks goroutines or global state cannot affect its neighbours.

Step four: output people can use

Two audiences, as in chapter forty nine.

For a human, failures only, with the file and line so an editor can jump there:

FAIL  TestInvoiceTotal  (0.03s)
  invoice_test.go:42: Total() = 1999, want 2000

ok    38 passed, 1 failed, 2 skipped  in 1.4s

For a machine, one JSON object per event, which is what go test -jsonproduces and what every CI system consumes:

{"Action":"run","Test":"TestInvoiceTotal"}
{"Action":"output","Test":"TestInvoiceTotal","Output":"..."}
{"Action":"fail","Test":"TestInvoiceTotal","Elapsed":0.03}

Streaming the events rather than printing a report at the end matters more than it sounds: a twenty minute suite that prints nothing until it finishes is indistinguishable from a hung one.

The features that make a runner good

Having built the core, these are each an afternoon and each one you will miss if you leave it out:

• Filtering
• -run with a regular expression. The most used flag there is
• Timeouts
• Per test and overall, with the stack dumped on timeout
• Retry flaky
• -retry 2, and report the retry rather than hiding it
• Shuffle
• Random order with a printed seed, so a failure is reproducible
• Sharding
• -shard 2/5, for splitting a suite across CI machines
• Slowest
• Print the ten slowest tests. The list is always surprising

What go test does that yours will not

Worth knowing, because the list is a map of the remaining depth:

• Caching
• Results keyed by the inputs: source, environment, files read. This is why an unchanged test prints (cached)
• Coverage
• Rewriting the source to add counters before compiling
• Benchmarks
• Adaptive iteration counts until the timing is stable
• Fuzzing
• Corpus management, mutation, minimisation of a failing input
• Subtests
• t.Run, with its own parallelism and naming rules
• Vet
• A subset of checks run automatically before the tests

The caching one is the most interesting piece of engineering in the list: it hashes everything that could affect the outcome, including the files the test read, and that is why it is safe to trust.

Assignment

Commit your work with the message Test Runner - Assignment One, exactly as written, so the leaderboard can score it.