I'm using Radeon ReLive for recording - it's free, and seems to be doing a fairly good job; nVidia users have Shadowplay, which has largely the same functionality.
At 1920x1200, 59fps and maximum bitrate, the video quality is only very slightly (nearly imperceptibly) worse than what the game outputs, and the resulting footage takes up only a few GB per hour - a far cry from Fraps, which does around 2 orders of magnitude worse!
The performance hit due to recording is only a few %; totally irrelevant in this case, considering that without the limiter I'm normally getting 80-300fps in OOTF, with forced 4xEQ FSAA + 8x anisotropic filtering, on my R7 260x.
But using 8xEQ FSAA + 16x anisotropic causes it to dip as low as 40-50fps sometimes; even though >90% time it's still above 59fps.
I've tested this with dgVoodoo (which only does MSAA, not FSAA) - as well as without dgVoodoo, but with forcing FSAA + anisotropic through Radeon Settings.
Performance-wise, it's about the same.
Quality-wise, it's also very similar for the most part; a careful side-by-side comparison reveals very little, if any, difference - however, dgVoodoo makes the smallest text appear rather blurry, whereas by using Radeon Settings it stays properly sharp.
Out of curiosity, I also tested the 10th Anniversary mod (and also forcing 4xEQ FSAA + 8x anisotropic, on top of it): with the shaders disabled, I'm getting a solid 59fps pretty much everywhere.
After enabling the shaders, I still get 59fps most of the time, but in some areas it can then dip as low as 30fps - still very much playable, but not too great for an HD recording.
Of course the bump mapping has to go for that, and there are still minor graphical glitches (eg. some thin straight white lines forming a "broken frame" around the moons); as well as noticeable straight line patterns on the ground (due to the texture seams?).
Other observations:
- Do not enable lens flare effects - not only they appear to not work in the first place, but also cause weird lag spikes at times.
- Disabling the framerate limiter caused significant microstuttering at such high fps (likely due to frame timing inconsistencies?); effectively degrading the perceived framerate roughly by an order of magnitude.
- AMD ReLive has serious stuttering issues when trying to record 24-bit audio. Don't even bother. 16-bit sounds perfectly fine, too - and at least it records properly.
- ReLive also can't record the desktop (nor a subset of it, such as a game window) under Win7, if a non-Aero theme is being used - and then returns a very misleading error message. Just switch to Aero for doing whatever you want to record, and switch back when done. Fullscreen recording is not affected, either way.