Written By : Jeff Nettleton
December 2004

4/ 3DV
Are we maybe approaching this whole optimisation thing from the wrong angle? Wouldn't it be fairer to tell the driver dev teams to optimise all they like so long as it doesn't impact on image quality? Wouldn't this be a fair way to see how efficiently the hardware can get the job done and also how sharp the driver teams are? Drivers are optimised for most reasonably popular games titles so why not let them do the same for 3DMark?

Nick: There are two reasons why we are so adamantly against 3DMark (or any benchmark) specific driver optimizations and why we at the same time are so strong proponents of generic performance optimizations:

1. Products (in this case graphics cards and drivers) should not be designed to maximize performance on any specific benchmark since those efforts do not result in benefits to other applications.

2. Benchmark specific optimizations are against the underlying reason of the benchmark's existence. For instance, 3DMark is built to provide an objective gauge of a real-world raw performance.

5/ 3DV
3DMark can make or break the fortunes of a whole generation of graphics chip. Considering these guys line your pockets with considerable sums of money as part of the beta testing scheme, surely there are times when you have to bow to their inevitable pressure to write code in a way that shows their hardware in the best possible light? As an example, I'm guessing that NVIDIA were very keen that you include SM3.0 specific routines 3DMark05. In short, shouldn't we be a little sceptical that we're relying on you to dethrone companies that are actually paying your bills?

Tero: We are very well aware of the importance of our products in the market. That is exactly why we have established a structured program and detailed processes that enable us to carry on our mission of developing objective tools. When you think of it more carefully, we maximize our success only when we're able to consistently produce benchmarks that are fair and objective. Any slippage from that and you can be sure that our reputation and credibility within the member companies would be lost and that would be pretty much end of our story.

Nick: We have always been very open when it comes to our Benchmark Development Program (BDP). We detail all the work processes openly on the web site and we do provide full visibility to our development to all members. We work very closely with all our BDP members, and they all have their say in what they feel about our plans. Whenever a new 3DMark is being planned, we start by sending out a feedback-form to the members, and work from there. All members know exactly what is put into the benchmark, and why.

The reasoning behind the BDP is actually very simple - to make sure that our benchmarks are high quality products, unbiased and to make sure that we can bring out next generation real-time 3D which works on as many graphics cards as possible. In other words, we have put a formal program and processes in place to ensure maximum compatibility and fair benchmarking. Those who are interested can find more info about our BDP over here: http://www.futuremark.com/bdp/

6/ 3DV
It seems at the moment there are two ways to go when it comes to coding graphics card benchmarks, you either work to the specific features and architecture of the current generation of graphics cards or you adhere rigidly to the DirectX protocol. It seems you've opted for the latter. What's the reasoning behind this, and how heavily do you rely on cooperation from Microsoft to achieve this?

Nick: We have been relying on DirectX since 3DMark99. It was a choice we made as DirectX seemed to become a standard in gaming and a controlled API. OpenGL has evolved a lot in the last couple of years, but when we made the decision to go with DirectX, we were confident that it was the right choice. We work very closely with Microsoft when developing 3DMark and the cooperation has worked out very well.