The downside (or upside depending on view) of this, is that you have to be incredibly strict about having tests which are 100% stable, any flappyness will result in a lot of reverts being done.
One thing I wonder about is whether we can reduce the flap rate by just massively increasing the timeout value on the builds. Like 10x it.
How often do we push code that accidentally hangs or deadlocks and we didn’t notice it locally? Probably quite rare?
How often does a test run time out because CI got bored and killed it, but it would have finished eventually? Not sure but it feels like this is quite common, or at least, sometimes we have bad periods between us reaching the limit and someone bumping it properly for every build.