By : Jeff Nettleton
Are we maybe approaching this whole optimisation thing from
the wrong angle? Wouldn't it be fairer to tell the driver
dev teams to optimise all they like so long as it doesn't
impact on image quality? Wouldn't this be a fair way to
see how efficiently the hardware can get the job done and
also how sharp the driver teams are? Drivers are optimised
for most reasonably popular games titles so why not let
them do the same for 3DMark?
There are two reasons why we are so adamantly against 3DMark
(or any benchmark) specific driver optimizations and why
we at the same time are so strong proponents of generic
Products (in this case graphics cards and drivers) should
not be designed to maximize performance on any specific
benchmark since those efforts do not result in benefits
to other applications.
2. Benchmark specific optimizations are against the underlying
reason of the benchmark's existence. For instance, 3DMark
is built to provide an objective gauge of a real-world raw
3DMark can make or break the fortunes of a whole generation
of graphics chip. Considering these guys line your pockets
with considerable sums of money as part of the beta testing
scheme, surely there are times when you have to bow to their
inevitable pressure to write code in a way that shows their
hardware in the best possible light? As an example, I'm
guessing that NVIDIA were very keen that you include SM3.0
specific routines 3DMark05. In short, shouldn't we be a
little sceptical that we're relying on you to dethrone companies
that are actually paying your bills?
We are very well aware of the importance of our products
in the market. That is exactly why we have established a
structured program and detailed processes that enable us
to carry on our mission of developing objective tools. When
you think of it more carefully, we maximize our success
only when we're able to consistently produce benchmarks
that are fair and objective. Any slippage from that and
you can be sure that our reputation and credibility within
the member companies would be lost and that would be pretty
much end of our story.
We have always been very open when it comes to our Benchmark
Development Program (BDP). We detail all the work processes
openly on the web site and we do provide full visibility
to our development to all members. We work very closely
with all our BDP members, and they all have their say in
what they feel about our plans. Whenever a new 3DMark is
being planned, we start by sending out a feedback-form to
the members, and work from there. All members know exactly
what is put into the benchmark, and why.
reasoning behind the BDP is actually very simple - to make
sure that our benchmarks are high quality products, unbiased
and to make sure that we can bring out next generation real-time
3D which works on as many graphics cards as possible. In
other words, we have put a formal program and processes
in place to ensure maximum compatibility and fair benchmarking.
Those who are interested can find more info about our BDP
over here: http://www.futuremark.com/bdp/
It seems at the moment there are two ways to go when it
comes to coding graphics card benchmarks, you either work
to the specific features and architecture of the current
generation of graphics cards or you adhere rigidly to the
DirectX protocol. It seems you've opted for the latter.
What's the reasoning behind this, and how heavily do you
rely on cooperation from Microsoft to achieve this?
We have been relying on DirectX since 3DMark99. It was a
choice we made as DirectX seemed to become a standard in
gaming and a controlled API. OpenGL has evolved a lot in
the last couple of years, but when we made the decision
to go with DirectX, we were confident that it was the right
choice. We work very closely with Microsoft when developing
3DMark and the cooperation has worked out very well.