By : Jeff Nettleton
went through a bit of a torrid time earlier in the year,
an odd situation for the former darlings of the hardware
that much of the dust has settled I wanted to fire a few
questions their way to see what they had to say, and to
use the opportunity to share some of my own thoughts on
the whole benchmarking issue.
sincere thanks go to Nicklas Renqvist and Tero Sarkkinen
for taking time out to provide us with the answers.
Let me start by poking an old wound, though I prefer
to think of it as cleansing it. MadOnion.com and 3DMark
were the gamer's best friend. Even people who didn't know
a GPU from a UPS could quote you their 3DMark score. Then
came the cheating allegations, and as a result the public
brawl which quite badly dented people's faith in 3DMark
as a viable benchmark. You took a real hammering for this
and lost a lot of respect as a result. Was this fair and
do you feel you've done enough to win back that respect?
Whenever you manage to
do something as popular as we did with the brand "3DMark",
you can always expect that something will happen. What happened
after the launch of 3DMark03 and all the allegations against
us was just that. We worked very hard to set the record
straight, and decided to start work even more closely with
the press & media in order to get some balance in the
discussion. I personally think that we have proven our point,
and demonstrated that what we did was necessary and right.
It seems that now couple of months after the launch of 3DMark05,
pretty much everyone has again adopted 3DMark. I think that
everyone has noticed that we consistently follow our benchmarking
guidelines which we set for 3DMark03, and that we stick
to them. Note that these guidelines have stayed essentially
the same since the beginning of our company!
How can you ever hope to convince people that results from
a synthetic benchmark are as valid as those from a "real"
retail games title? The stock quote the keeps appearing
is that "people play games, not benchmarks".
While it is true that people play games and not benchmarks,
it really does not mean that only games can be used as benchmarks!
Both games and synthetic applications can be designed well
or poorly to serve as a benchmark. You really have to determine
these case by case. There are good game benchmarks and there
are bad game benchmarks, similarly, there are good synthetic
benchmarks and then there are bad synthetic benchmarks.
people ask that does benchmark-X (game-benchmark) tell you
at what frames per second they can play game-Z? Of course
the answer is 'No', in the same vein though, you have to
understand that your PC's performance in playing game-Z
does not tell you anything at all about how well your PC
can play game-Y, not even if they have the same game engine
technology. Reason for this is that games differ from each
other in many different ways. Even when two games share
the same engine, they usually have substantial differences
in e.g. the level of DirectX technologies they use, the
amount of polygons and scene complexity they have, differing
fallback paths, etc. What makes it even worse is that these
game-benchmarks are often poorly documented or even undocumented
and thus the user really does not know what the benchmark
on the other hand, is scientifically designed to feature
technologies and workloads that are characteristic of next
generation games. Furthermore, it has code paths that are
meticulously designed to be fair for all hardware from the
performance measurement point of view. To top it off, it
is professionally documented and there are user friendly
controls to turn on or off nearly every imaginable feature
for advanced performance analysis.
short, if you want to know how well your PC plays a certain
game, there is no better benchmark for you than that specific
game. However, if you do not know which games you will be
playing, or you want to know how well in general your PC
can play next generation games, it is a wise decision to
include 3DMark as part of your testing to get that information!
We think it is very important that reviewers use as many
game-benchmarks as possible, along with 3DMark and/or PCMark.
The games should of course be the most popular ones, and
from various genres of game types. To make it even better,
the timedemos could, or perhaps even should, be recorded
in-house. Using 10 different first person shooters in a
graphics card review isn't in my opinion a very reasonable
decision, as everyone is not playing those types of games.
I think that for reviewing a graphics card, the best way
to got is to use both a bunch of different genres of popular
games (FPS, simulators, racing games, RTS etc) and then
professional benchmarks like 3DMark and/or PCMark. To really
emphasize how well the graphics card performs in different
types of games, different situations, how well it stacks
up in raw performance to others and how future proof the
hardware may be.
With all the recent claims of unfair 3DMark driver optimisations,
how difficult would it be to implement some kind of compiler
that randomly juggles code to defeat optimisations without
affecting how a scene is rendered? Are you actively working
on any other forms of optimisation detection or defeating?
We have done some research in how to "fool"
the drivers, but we haven't come up with a 100% working
solution. Juggling shaders works only so far, as at some
point juggling shaders might interfere with the performance.
It wouldn't be fair if someone gets a juggled shader which
is less efficient than the other. We are working on this
matter as we speak, but as I said, so far there is no bullet-proof
way. We are of course up for suggestions & ideas, so
if anyone out there has a rock-solid working solution, don't
hesitate to contact us!