Written By : Jeff Nettleton
December 2004

11/ 3DV
Who decides what the next 3DMark will include and how the game tests will look and feel? Are you usually accurate with your estimations about how hardware will perform at the time of release or is there often a mad scramble near completion to add features or alter its level of complexity to match current hardware.

Nick: We decide everything in-house, but of course with the help of our BDP members and game developers. Stuff like how many game tests, what the content of them will be etc. is basically up to us. We usually spec the next 3DMark at a very early stage, and try to stick to that. As we work very closely with the BDP members, we have a pretty good insight on what any future DirectX features forthcoming hardware will support, so we can plan well ahead what we will use in new 3DMarks. Of course we always first make the tests way too demanding, so when we are closing in on the first beta versions, we have to reduce a bit here and there in order to get the high-end systems to even execute the benchmark.

12/ 3DV
I doubt I'll get an honest answer to this one, but do you feel Microsoft, by virtue of their DirectX API, have just a little too much say over how graphics technology will evolve and who will use it to best advantage? Are they the right people to be calling all the shots?

Nick: In all honesty, I am not really sure how they work when they're developing the DirectX. See, you got an honest answer. :)

13/ 3DV
Have you ever been tempted to plant an Easter egg in 3DMark somewhere, or have you already done it and we've missed it?? On the basis that sex sells, I think a the addition of a voluptuous, beautifully rendered female should feature in there somewhere, purely to show the virtues of a good pixel shader of course!

Nick: We have always had Easter eggs in our products. Not only in 3DMark. I wouldn't mind seeing voluptuous, beautifully rendered women in our products, so let's see what we can make for the next 3DMark, after 3DMark05. :)

14/ 3DV
Assuming you find a little time for gaming, which of the current games gets your juices flowing and which have left you cold? Personally I think that Far Cry would get my vote as game of the year, while Doom3 was probably the game that disappointed me most.

Nick: FarCry and DOOM 3 really were (and still are) breathtaking games, but I am too much of a chicken to play DOOM 3 really. I love what id software has done visually and technically, but it is way too creepy for me. I have never liked to sit in-front of my computer and be sweaty and jump around screaming like a madman. I liked how FarCry started out, but as I got further I got a bit .. well, disappointed. Visually it was top-notch, but the storyline wasn't really at the level as I expected. Oh well.. Still FarCry is a very nice game, and I enjoyed the vast landscapes it has! The game I'm currently hooked on is Half-Life 2, though I need to play it a bit more to make any comments about it.

I prefer to play games on the consoles such as: Burnout3, GTA series, NHL series etc. Playing games on the console is more relaxing for me, than playing on the PC. Why? Well, if I play games on the PC, I have a bad habit to quit the game to check for new emails, messages on the messenger or new interesting topics on various forums. J That is really an annoying habit, but I can't help it!

At the end of the day, playing any game is an experience of its own.

My thanks again to Nick and Tero.

Our Thoughts:::...

I'd like to finish off this interview by sharing with you my personal thoughts about 3DMark and about benchmarking in general. And just for the record FUTUREMARK don't advertise with us, pay us on an afilliateship basis or sponsor us in any way other than for a single license to allow us to use the benchmark in reviews if we wish.

A lot has been said about synthetic benchmarks and about their usefulness as a valid means of indicating hardware performance, and a couple of prestigious websites have exercised their right to cease using 3DMark in their reviews, something they're perfectly entitled to do. While I have no intention of suggesting they're wrong for doing this, I do take a slightly different view on the whole situation.

Graphics hardware has evolved at a scary pace, and modern GPUs/VPUs make even the latest CPUs look quite ordinary. This is great for consumers, but their complexity and programmability makes it increasingly difficult to compare one with another in a fair and unbiased way. In fact it's almost like comparing two different cars. One may have a higher top speed but the other may be quiter. One may be more fuel efficient but carries less luggage. Each has its own strengths and weaknesses and not looking at the whole picture can be misleading. Putting Diesel in a petrol engined car then complining it doesn't run doesn't mean the petrol engined car is inferior, it just does things differently. Different isn't always bad.

With only two major graphics chip vendors and only a handful of high-profile games making it to the shelves each year, it's perhaps not surprising that drivers become heavily optimised for these titles, a practise that adds further to the difficulties of making an unbiased comparison of one against another.

So what would I do if I wanted to write the perfect benchmark? Well it seems fairly clear that we need a 3D engine that's coded not for ATi, nor for NVIDIA, but strictly in accordace with the holy grail of non-OpenGL 3D rendering, Microsoft's DirectX API? We need to include multiple game genres, we need to use as many of the current DirectX features as possible, we need it to look good while also being challenging and we need a mathematically created performance index to compare one result with another. Well, and I don't want to rattle any cages here but, isn't that exactly what 3DMark is?

We can whine all day that it's synthetic in nature, but no moreso than any game on the market that doesn't then go on to spawn future games titles. If FutureMark were to add a few missions and user controlled characters then market 3DMark as a game would that really make it suddenly a "valid" way to test your hardware? Is a control method and a storyline all that's needed to transform all those pages of perfectly legitimate code from a benchmark to a game? Surely it's not that simple.

No single benchmark used alone means a damned thing. I've heard the arguments about why Doom or Half Life are a better indication of hardware performance and I disagree. For sure we'll see dozens of games built around their 3D engine which may suggest it's a perfect benchmarking candidate, but we'll also see disproportionate efforts go into optimising the drivers as a result, so one cancels out the other. In these situations we either need to open the floodgates and let driver teams optimise for benchmarks in the same way they optimise for retail games, or we need someone like FutureMark to police drivers and keep defeating the little routines that unfairly make one piece of hardware look superior to another.

As for those who say a benchmark doesn't reflect how their favourite game will play, you're right, but will Doom 3 tell you how Far Cry will play? certainly from my experience Medal Of Honor framerates seem to have no correlation to those from my old copy of Mercedes Benz Truck Racing? So what's the point you're making?

3DMark was and is a valuable tool for measuring hardware performance, and used in isolation it may well prove to be even more valid than the many "real" game benchmarks simply because we know there are no unfair optimisations going on.

I want optimisations in my game, hell I EXPECT my hardware drivers to be optimised for my favourite games, but when I want to see how much raw horsepower my GPU has under the hood it's DirectX horsepower that seems to matter most so I know that games have limited usefulness, even though I'd still like to see some graphs to see how my new purchase compares to the competition.

FutureMark are a relatively high-profile corporation, and there's always the temptation to assume underhand things are going on. Like Microsoft, and more recently NVIDIA, we don't like a company to get too powerful. As soon as they do we all line up to take it in turns to bring them down a peg or two. Now that's happened let's not cut off our noses to spite our faces. For as long as 3DMark remains a useful and valid tool for HELPING us understand graphics performance it will have a place here. It might be trendy to be anti-establishment but it's not always right.

Disagree? I'd love to hear your thoughts. Head on over to THIS thread if you've anything you'd like to share on this interview, or on your thoughts regarding benchmarking in general.

Thanks for reading.