Test System and Procedures
Introducing our Newest Test System and Methodology
When it comes to reviewing a graphics card we previously had a static test bench in which we put every graphics card we reviewed to give comparable results. However, the big flaw in this approach is that every graphics card used the same hardware so that we had stupid scenarios where we ended up testing a HD 7730 or GT 640 with an Intel Core i7 3960X and ASUS Rampage IV Extreme X79 motherboard – hardly a realistic scenario by anyone’s judgement. With our new “low to mid range” test system we’re looking to change that by using a different, and more appropriate, platform. We’re opting for Intel’s Z87 platform and a more modest Core i5 4440, and we’re downsizing from 16GB to 8GB of RAM with a slower frequency of 1866 not 2133MHz. I did contemplate using a Core i3 4330 but I feared this might lead to GPU bottlenecks, especially as we got towards those mid range graphics cards. Based on current pricing (correct as of March 23rd 2014) this combination of the Gigabyte G1 Sniper Z87, Intel Core i5 4440 and 8GB of 1866MHz RAM costs $450 compared to the $1670 of the previous aforementioned parts. We think this is an important change for delivering more realistic and useful content to you, our readers. As this is a “low to mid range” test system we will primarily be focusing on specific GPUs (correct as of March 23rd 2014) which are the AMD R9 270 and below and the Nvidia GTX 660 and below. For clarity this means the following GPUs: For AMD we have the R7 240, R7 250, R7 250X, R7 260, R7 260X and R7 265 while for Nvidia we have the GT 630, GT 640, GTX 650, GTX 650 Ti, GTX 650 Ti Boost, GTX 750 and GTX 750 Ti.
The second major change was the resolutions we test at. Since these are “affordable” graphics cards we will be testing there is no logical reason to test QHD (2560 x 1440), UHD (3840 x 2160) or Triple HD (5760 x 1080). Instead we used Steam’s Hardware Survey to determine the three most popular gaming resolutions as we feel this is what you, our readers, will want to see the most – those resolutions are 1366 x 768, 1600 x 900 and 1920 x 1080.
The third major change is our choice of games. After initiating a poll on our site to determine the most popular games you wanted to see in our graphics card reviews we’ve ditched Alien Vs Predator, Dirt Showdown and Sleeping Dogs in favour of Assassin’s Creed IV Black Flag, Battlefield 4 and Call of Duty Ghosts.
Finally, to address recent developments in cryptocurrency mining we have added scrypt hashing performance of Nvidia and AMD graphics cards as well as different metrics of compute performance for people who use their graphics cards for other non-gaming purposes (e.g. OpenCL and OpenGL accelerated workloads).
Acoustic Testing – Measuring Noise Levels
We take acoustic measurements under three different scenarios: desktop idle, Furmark and Unigine Heaven. We take these measurements at the end of a five minute window of each, this is to ensure the maximum fan speed for the duration is reached to ensure the most accurate acoustic reading is taken. We use a decibel meter to take the measurement approximately 25cm away from the graphics card aligned at the same vertical height as the fan with the sensor pointing towards the graphics card. We disable the CPU fan, use an SSD instead of a HDD and isolate as much background noise as possible to get the most accurate results.
Like with our acoustic tests we take temperature tests under three different scenarios: desktop idle, Unigine Heaven and Furmark. We use TechPowerUp’s GPU-Z utility to record the minimum desktop idle temperature and the maximum temperature during five minute runs of Furmark and Unigine Heaven. We also provide the ambient room temperature alongside those 3 temperature readings as we’ve had extensive user feedback about delta temperatures expressing their dislike and confusion towards the measure. When we test passive graphics cards we target the airflow of an 800 RPM 120mm case fan at the graphics card with an approximate distance of 30cm to simulate the airflow of a case fan, this is because passive graphics cards running on an open-air test bench are adversely affected by the still air of the room compared to inside a case where the channelled airflow of case fans allows them to function more effectively.
Power Consumption Measurements
Power consumption is again similar to acoustic and temperature tests. We use the desktop idle state, Unigine Heaven and Furmark scenarios to acquire power consumption figures. We use a killawatt meter to grab total system power consumption from the socket. During these tests we ensure all background applications are closed to ensure power consumption is influenced as much as possible by the GPU, not other system components or processes. During Unigine and Furmark we take the power consumption figures after 1 minute of the 5 minute run has passed, this is because after an extended period of time graphics card power consumption tends to drop as thermal or power limit throttling kicks in. It is also because we want to take the power consumption readings at the same stage of the Unigine Heaven and Furmark tests to ensure comparability. We acknowledge that measuring power draw from the wall is not the most effective measure but this is the only method available to us at the current point in time.
To formulate our performance metric we use a weighted average method and each benchmark or test gets a weight of 1 out of a total calculation with 10 weights (meaning all resolutions and variations of the same test are only ever one weight). Those 10 weights are 1 for each test we so that means 7 games (listed below) and 3 benchmarks (also listed below). We calculate the relative score for each card in each test by dividing the cumulative frame rate/score of each card by the average cumulative frame rate/score from all the graphics cards in group. We then take those 10 relative scores and average those for each card, before dividing those averages by the average of the card being reviewed in the test so that the value of the card being reviewed is 1 and every other card’s performance metric is in relation to it. The end result are numbers that vary around 1 with 0.5 being half the performance of the tested card and 2 being double the performance of the tested card. We then convert those numbers into percentages to make them more readable so 1 becomes 100% and 0.5 becomes 50%.
- Motherboard – Gigabyte G1 Sniper Z87 (Read our review here)
- Processor – Intel Core i5 4440 (Turbo Enabled)
- RAM – 8GB (2 x 4GB) 1866MHz 9-10-9-27
- CPU Cooler – Corsair H100i with Quiet Fan Profile
- Power Supply – Corsair HX1050W
- Main Storage Drive – Kingston HyperX 240GB SSD
- Chassis – Lian Li T60 Test Bench
- Displays – LG Flatron IPS224
- Operating System – Windows 7 Ultimate 64 Bit
We would like to thank Corsair, Gigabyte, Intel, Kingston, Lian Li and all our other partners who supplied us with test equipment and hardware. Their generosity makes our testing possible and without them we wouldn’t be able to produce the reviews we do, so thank you!
- 3DMark 11 – Performance and Extreme Presets
- 3DMark (2013) – Firestrike and Firestrike Extreme Presets
- Unigine Heaven Benchmark 4.0 – Extreme Preset
- BasemarkCL – Unoptimised, all tests
- ComputeMark – 1920 x 1080, Normal
- Luxmark – GPU test only
- CUDAMiner – auto-detect settings
- CGMiner – manual tuning of settings which varies between the graphics card being tested
- Assassin’s Creed IV Black Flag – High, MSAA2X, SSAO
- Batman Arkham Origins – Normal, MSAA2X
- Battlefield 4 – Ultra Preset
- Bioshock Infinite – Ultra Preset
- Call of Duty Ghosts – Ultra Preset
- Metro Last Light – Very High, AF4X
- Tomb Raider – Ultra Preset
- 1366 x 768
- 1600 x 900
- 1920 x 1080
- Plug “killawatt” style electricity usage meter
- Benetech GM1351 decibel meter
- CPUID HWMonitor
- TechPowerUp GPU-Z
- Windows 7 Professional 64-bit
- Graphs using FPS measurements are sorted by their 1080p value as this is the most popular gaming resolution
- Power consumption, acoustic and temperature graphs are sorted lowest to highest unlike game/benchmark graphs that are sorted highest to lowest.