Of course young optimistic me would have considered that this was an easy thing to have a QA test for, but here we are in 2024 and I am neither young or optimistic. Maybe the AI QA folks were in the last few rounds of Google layoffs or something.
Turns out that trying to correct for biased models by biasing your input does not make your results more reliable.