I've been puzzled by the subjective experience of stutter in games when measurements were showing consistent framerates of 60+. The low not normally dipping below 50, so experience should be perfectly smooth, right?
Wrong. I worked out that the issue must be with the averages. It looks like frames that take unreasonably long to render can create visible stutter without showing up even on the Low FPS value due to the rest of the frames within the target second rendering at a high pace. If the delay between two frames is above 41.6 ms, effectively 24 FPS, you're viewing a slideshow there.
I've had a look into the issue and it turned out the issue was known and well investigated already in this excellent 2011 article: http://techreport.com/review/21516/inside-the-second-a-new-look-at-game-benchmarking
The guys at Techreport actually incorporated their findings into the way they write their GPU reports, which makes their website a very valuable source of additional info you won't find in other places.
This appears to be a particular bad issue with Crossfire and SLI setups, where there seems to be a tendency to throw out two frames in quick succession followed by a larger delay until the next two. Take it to the extreme and you're experiencing half the FPS you're meant to have.
To be perfectly clear, large gaps between individual frames can be a fault with not just the GPU alone, but a range of issues from CPU, storage, network performance and game engine itself. The metric should still allow comparison between individual parts changed within the same system.
Fraps lets you record individual frame timings when in benchmarking mode and this data had been very interesting to dig through. Here's the summary of my GTA V benchmark:
2015-12-16 15:47:07 - GTA5
Frames: 7529 - Time: 101625ms - Avg: 74.086 - Min: 47 - Max: 153
With a min of 47, I was expecting a perfectly smooth run, but it wasn't. Looking into the data, it looks like I had 27 frame changes where the effective rate was below 25. Here's some detailed breakdown of my run:
So at this point I want to call out the Tek guys to consider incorporating this metric into their hardware benchmarks, as I am sure I am not the only one who would reconsider hardware choices based on a GPUs ability to throw out consistent frame rates over a high average that hides bursts and stutter.
I also want to call out to the community to help me pin down how to make the best use of this data. As you can see, I was just poking around, trying to find insights and some of the data is redundant. What values are useful? What FPS ranges are meaningful? How can we quantify the consistency of performance? At what point (percentile) should we make the cut between expected performance and freakish outliers?
I can write a macro to automatically generate a table like this from the raw timing data and share it with the community, but it would be useful to know what's worth displaying and how first.