If you were getting ready to buy a new high-end camcorder or a new car, chances are you’d spend some time reading independent reviews. Maybe you’d pick up a copy of Consumer Reports or Road and Track. Perhaps you’d scan Amazon.com for user evaluations. Whatever. The point is, you probably wouldn’t just make your choice based on the vendor’s marketing claims, right?
Yet that’s exactly what some design engineers do when they choose a processing engine. My colleagues at BDTI and I were discussing this phenomenon last week, and wondering why such crucial technical decisions are often made without any independent input. (Of course the topic is of significant interest to us, since we are, after all, an independent benchmarking and analysis company.) Here are a few of the observations we came up with.
First off, it’s not because the design engineers are stupid, or because they wouldn’t be happy to have additional information with which to make their choice. But many of them are unaware that third-party analysis is an option—or they can’t find independent benchmark results for the specific processors they’re considering.
Part of the problem is cost. Unfortunately, it’s harder to analyze the capabilities of, say, a high-performance processor than those of a Buick. With cars (and most other consumer products) it’s pretty straightforward to do side-by-side testing. Not so with processors, where doing a careful analysis and comparison takes a lot of detailed work, by highly skilled and knowledgeable engineers. And frankly, that costs more than comparing the features of two digital cameras or road-testing a handful of SUVs.
Furthermore, with consumer products like camcorders, the number of users is huge, and the cost of the analysis can be amortized across them. In contrast, the number of chip users is much, much smaller. As a result, organizations that provide benchmarking and analysis of chips typically rely on the chip companies themselves to underwrite the work. Unfortunately, not all chip companies are willing to step up and pay for independent benchmarking—in part, because their customers haven’t insisted on it.
But consider this: Processor choices can go badly wrong in ways that car choices usually don’t. I mean, maybe the car you choose doesn’t get quite the gas mileage you expected, or maybe it’s not as fun to drive as you hoped, but it’s not like you’d buy a car thinking it would seat all five people in your family and then find out a few months later that it only seats three. With processors, an analogous catastrophe can (and does) happen all the time. You think you’re getting a processor that can do real-time H.264 encoding in under three Watts, and whoops, six months into developing the code it turns out that the little silicon engine can do no such thing, and you have to start all over with a new chip. This is the kind of nightmare scenario that good benchmarks can prevent.
The cost of choosing the wrong processor is often a heck of a lot higher than the cost of buying a lame camera or even a not-quite-right car. Making that choice without independent analysis is a risky business.
Jennifer Eyre White contributed to this column.
Add new comment