It’s not your fault.

A test just failed on your pull request. notmyfault remembers how every test behaves on your main branch, and tells you which failures are new, which tests are known to be flaky and which were broken before you touched anything.

Add it to your workflow See a real pull request

Free, for GitHub, GitLab, Forgejo, Gitea and any CI with Node.js. MIT licensed, nothing to host.

You have re-run that build before.

Three-panel comic. A developer panics in front of a red build. The croissant mascot rushes in with the test's history in a notebook. The croissant points out "flaky" and the developer relaxes with a coffee.

Every failure gets a verdict

Each failed test is checked against its recent runs on main, retries within the run and re-runs of the same commit. The first rule that matches decides, and the comment says why in one sentence.

New failure

Nothing in the history explains it, or it failed with an error never seen on main, even if the test is known to be flaky.

What to do: assume your change caused it and reproduce it locally.

Suspect

It failed once or twice on main, between two successful runs. Flaky, or a breakage fixed right after: too early to tell.

What to do: read the message. If it looks unrelated, re-run the job. A pass on the same commit proves it flaky.

Already failing

The latest runs on main failed too, so the problem predates your pull request. For a flaky test, it takes more failures in a row than bad luck explains. The comment names the commit and the pull request the failures started with.

What to do: nothing in your pull request. Tell whoever broke it, and update your branch once main is fixed.

Flaky

Known flaky when a retry or a re-run proved it, probably flaky when it keeps failing in isolation.

What to do: re-run if it blocks you, then fix the test, or let quarantine mode tolerate it.

Fixed

And when a test failing on main passes in your pull request, the comment says so, even if nothing else failed. Tests that ran on main but no longer run in your pull request are listed too.

Read every rule and threshold

How it works

  1. Runs on main build the history

    After every run on your default branch, notmyfault records the outcome of each test on a notmyfault-history branch of your repository. A single commit, rewritten each time, so it never grows.

  2. Pull requests are compared with it

    When a test fails on a pull request, notmyfault looks at its last 50 runs on main, at retries within the run and at re-runs of the same commit.

  3. The comment explains every failure

    One comment per pull request, updated on every push, and an annotation next to the code of each failed test. The job summary also ranks your least reliable tests.

A notmyfault comment on a pull request: one new failure related to the change, one test already failing on main and one known flaky test. Quarantine mode fails the check.
A real comment, on a pull request of the demo repository.

Add it after your test step

Your tests need to write JUnit XML, and almost every runner can: Vitest, Jest, pytest, Go, Maven, Gradle, cargo-nextest, Playwright, PHPUnit, RSpec, .NET.

  • if: ${{ !cancelled() }} runs notmyfault when the tests fail, which is exactly when you need it.
  • Runs on pushes to main build the history. Pull requests read it.
  • The permissions let notmyfault store its history and comment.
  • Several suites, or the same tests on several systems? List them in suites: one comment, and a history for each.

Follow the getting started guide

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: write        # store the history
  pull-requests: write   # comment on pull requests

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - run: npm test       # writes JUnit XML to reports/

      - uses: tashikomaaa/notmyfault@v1
        if: ${{ !cancelled() }}
        with:
          junit: reports/**/*.xml

On GitLab too

The same history and the same verdicts in GitLab CI/CD, from a job of its own that reads the reports of your tests, whatever their language.

  • Merge requests get the same comment, kept up to date on every push.
  • Failed tests show up in the Code Quality widget, with their verdict.
  • The counts become variables for later jobs, and flaky tests can get an issue each.
  • A project access token in NOTMYFAULT_TOKEN lets it store the history and comment.
  • Also a CI/CD Catalog component, and a job publishing the history pages with GitLab Pages.

On Forgejo and Gitea, whose workflows follow GitHub Actions, the action itself runs as it is. Jenkins, CircleCI, Buildkite and the others run the same bundle with npx notmyfault, with the report in the log and a summary file.

Set it up on GitLab · See merge requests · Forgejo and Gitea · Any CI

include:
  - remote: https://raw.githubusercontent.com/tashikomaaa/notmyfault/v1/templates/notmyfault.gitlab-ci.yml

test:
  script: npm test      # writes JUnit XML to reports/
  artifacts:
    when: always
    paths: [reports/]

notmyfault:
  extends: .notmyfault
  needs: [test]
  variables:
    NOTMYFAULT_JUNIT: reports/*.xml
    NOTMYFAULT_KEY: test
The croissant mascot in sunglasses, above the words "Works on main (sometimes)".

Stop flaky tests from blocking merges

In quarantine mode, notmyfault decides whether the check fails. Known flaky tests stop blocking, failures that look real still do, and when in doubt, it blocks.

- run: npm test
  continue-on-error: true

- uses: tashikomaaa/notmyfault@v1
  with:
    junit: reports/**/*.xml
    mode: quarantine

Or keep your tests failing the job as usual, and have branch protection require the notmyfault check instead, which only fails on failures that look real.

Read the quarantine guide

Get flaky tests fixed, not just tolerated

Quarantine keeps them from blocking. The rest makes sure someone deals with them, from the same history.

An issue per flaky test
Opened when a test is proven flaky, mentioning its owners from CODEOWNERS, updated with its latest failure, closed once it stops failing.
A badge and a report
A badge counting your flaky tests, and pages listing them with their runs and the test time they cost, ready for GitHub Pages.
Re-runs on their own
When only flaky tests stand in the way, the run happens again, once: a small companion workflow on GitHub, notmyfault itself on GitLab.
Quarantine by hand
Tolerate a test until a date, with a reason, while the history catches up.
Slower tests spotted
A passing test taking twice as long as usual is flagged before it starts timing out.
Renames followed
Rename a test on your main branch, and it keeps its history.
Vanished tests caught
A pull request that deletes or stops running tests gets told, even when everything passes.
One dashboard for all
The flakiest tests of every repository, ranked together by what they cost: see the one of the demos.
Verifiable releases
Every release is rebuilt from its tag, checked and attested, with a checksum to pin.
Nothing taken on trust
Reports, CODEOWNERS files and the history are read as if a stranger wrote them, and your token never reaches a log, a link or another host.

See it on real pull requests

The demo repository is a small shop with a flaky payment test and a search test broken on main. Each of its open pull requests gets a different comment.

Nothing to host

notmyfault runs inside your GitHub workflow or GitLab pipeline, and keeps everything in your repository.

No server, no account
The history lives on a branch of your repository, as private as your code.
Zero runtime dependencies
One bundled file, rebuilt and verified by CI. Nothing to audit but the repository.
A couple of seconds
It parses the reports, pushes one small commit and calls the API once or twice.
Free
MIT licensed. It runs in your own pipelines and calls no paid service.