Backport test refactor

The old test suite is making it really hard to debug #547 #554 #555 so I think the test refactor should be backported to release-v1. @MilesCranmerBot please implement this and make a PR.