notmyfault: is that failing test your fault?

You have re-run that build before.

Three-panel comic. A developer panics in front of a red build. The croissant mascot rushes in with the test's history in a notebook. The croissant points out "flaky" and the developer relaxes with a coffee.

Every failure gets a verdict

Each failed test is checked against its recent runs on main, retries within the run and re-runs of the same commit. The first rule that matches decides, and the comment says why in one sentence.

New failure

Nothing in the history explains it.

What to do: assume your change caused it and reproduce it locally.

Suspect

It failed once or twice on main, between two successful runs. Flaky, or a breakage fixed right after: too early to tell.

What to do: read the message. If it looks unrelated, re-run the job. A pass on the same commit proves it flaky.

Already failing

The latest runs on main failed too, so the problem predates your pull request.

What to do: nothing in your pull request. Update your branch once main is fixed.

Flaky

Known flaky when a retry or a re-run proved it, probably flaky when it keeps failing in isolation.

What to do: re-run if it blocks you, then fix the test, or let quarantine mode tolerate it.

Fixed

And when a test failing on main passes in your pull request, the comment says so, even if nothing else failed.

Read every rule and threshold

How it works

Runs on main build the history

After every run on your default branch, notmyfault records the outcome of each test on a notmyfault-history branch of your repository. A single commit, rewritten each time, so it never grows.
Pull requests are compared with it

When a test fails on a pull request, notmyfault looks at its last 50 runs on main, at retries within the run and at re-runs of the same commit.
The comment explains every failure

One comment per pull request, updated on every push. The job summary also ranks your least reliable tests.

A notmyfault comment on a pull request: one new failure related to the change, one test already failing on main and one known flaky test. Quarantine mode fails the check. — A real comment, on a pull request of the demo repository.

Add it after your test step

Your tests need to write JUnit XML, and almost every runner can: Vitest, Jest, pytest, Go, Maven, Gradle, cargo-nextest, Playwright, PHPUnit, RSpec, .NET.

if: ${{ !cancelled() }} runs notmyfault when the tests fail, which is exactly when you need it.
Runs on pushes to main build the history. Pull requests read it.
The permissions let notmyfault store its history and comment.

Follow the getting started guide

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: write        # store the history
  pull-requests: write   # comment on pull requests

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - run: npm test       # writes JUnit XML to reports/

      - uses: tashikomaaa/notmyfault@v1
        if: ${{ !cancelled() }}
        with:
          junit: reports/**/*.xml

The croissant mascot in sunglasses, above the words "Works on main (sometimes)".

Stop flaky tests from blocking merges

In quarantine mode, notmyfault decides whether the check fails. Known flaky tests stop blocking, failures that look real still do, and when in doubt, it blocks.

- run: npm test
  continue-on-error: true

- uses: tashikomaaa/notmyfault@v1
  with:
    junit: reports/**/*.xml
    mode: quarantine

Read the quarantine guide

See it on real pull requests

The demo repository is a small shop with a flaky payment test and a search test broken on main. Each of its open pull requests gets a different comment.

#1 Support fixed-amount discount codes
A new failure caused by the change, next to a test already failing on main and a known flaky test. Quarantine blocks the merge.
#2 Charge the reduced VAT rate on coffee beans, again
A suspect failure: the test failed once on main, when the same change landed and was reverted.
#3 Explain how to run the tests
Two failures, neither of them related to a README change. Quarantine lets the check pass.
#4 Find products regardless of accents again
A fix for the test broken on main, which the comment lists as fixed. The flaky test failed, passed on a re-run, and the comment turned green.

Nothing to host

notmyfault runs inside your workflow and keeps everything in your repository.

No server, no account: The history lives on a branch of your repository, as private as your code.
Zero runtime dependencies: One bundled file, rebuilt and verified by CI. Nothing to audit but the repository.
A couple of seconds: It parses the reports, pushes one small commit and calls the API once or twice.
Free: MIT licensed. It runs in your own workflows and calls no paid service.