DxOMark.com levels the camera test playing field
How objective are the camera tests you read in magazines and on web sites? That usually depends on the design of the tests, the accuracy of the test data, and the ability of the editor or lab assistant to interpret the test data. This last fact causes the most problems, especially when “expert” evaluators have little background in color science, optics, lighting, imaging processing, or even as working photographers. (A dead giveaway? If a magazine or web site fails to post biographic information on its editors and writers, it’s likely they want to hide a lack of experience or schooling.) For a time, these reviewers get by with a mix of writing skills and blarney, relying heavily on the old adage: a one-eyed man can always lead the blind!!
Of course, objectivity can also suffer when spineless magazine editors “adjust” product reviews to placate advertisers who threaten to pull the $$ plug if anything bad is written about their products. That’s a more likely scenario these days among print publications facing rising costs and dwindling ad revenue, but even websites that claim to have “higher Ethics” than their magazine competitors bow to financial pressure behind the scenes.
For more commentary on this topic, listen to the November 22, 2008 McNamara Report segment on Inside Digital Photo (slide to the 22min 30sec mark for that segment);
Savvy photographers usually catch on to both scenarios sooner or later, and search elsewhere for their information or check several sources before making a purchasing decision. For those reasons I think the new DxOMark.com website will be a useful tool and a hit among serious photographers. On this site you’ll find in-depth analysis and test data derived from the RAW files produced by nearly every major digital SLR, as well as a few popular compacts and new medium format models. Data includes color accuracy, noise and dynamic range at a variety of ISOs, true ISO sensitivity, low-light ISO performance, and an umbrella DxOMark Sensor rating. There are several interactive tools and graphs that make side-by-side comparisons between a variety of cameras easier than ever. Plus, if you’re a student of digital photography, you will find the library of articles located in the Technology section invaluable resources.
By now you’ve probably heard of DxO, a French company behind the DxOMark.com site that’s known for its popular DxO Optics Pro software. This software is sold with specific support for the camera and lens combinations you own, and performs a variety of complex image processing functions on a DSLR’s RAW file, including distortion correction, noise reduction, exposure optimization, and dust removal. However, you may not be familiar with DxO Analyzer 3.0 test software since it’s not sold to the general public or directly to photographers. Instead, DxO Analyzer 3.0 software and DxO-designed test targets are used by several US-based and International photo magazines, as well as a number of online photo sites, to provide camera and lens test data that is later interpreted by reviewers. Nearly every camera manufacturer also uses DxO Analyzer at some point during the product development cycle to help fine tune image quality.
I can personally vouch for the usefulness of data provided by DxO Analyzer 3.0, as it was part of the test equipment I certified for use in the Popular Photography test lab while I was the magazine’s Executive Technology Editor. In that lab, I used it to help determine lens distortion, chromatic aberration, and light falloff. It was also used to test camera noise levels at a wide variety of ISOs and gray-scale levels. The software’s “Blur rating” is used by a variety of competitors to determine lens sharpness (although it’s not an accurate tool for that use). On the other hand, the Blur Rating is still used in the Pop Photo Lab to determine the efficiency of the image stabilization systems built into a lens or camera body. DxO Analyzer 3.0 could also be used to determine the color accuracy, base ISO, and dynamic range of a camera, but this information either never made its way into Pop test report charts or was provided by other test methods.
While DxOMark.com provides a universal, and extremely useful, tool for comparing the data culled from the RAW files of the most popular DSLRs and medium format cameras, as well as in depth information about camera sensors, it shouldn’t be viewed as the final word. That’s because there is more to a camera than RAW file test data, (including metering and AF performance, handling, and several other parameters) and even RAW camera data is open to interpretation–not only to reviewers trying to determine which camera offers the most for the money, but to different RAW converter algorithms used to develop the data.
If there is a weakness to the data provided on DxOMark.com, it revolves around the “special sauce” that individual manufacturers such as Canon and Nikon claim they include in their RAW conversion software to get the most out of their respective camera’s RAW capture. My extensive experience with RAW conversion utilities has shown that all RAW converters are not created equal–and if DxO Analyzer 3.0 doesn’t access all of Nikon’s “special sauce” it may help to explain why the Dynamic Range rating for the Nikon D90 at DxOmark.com is just slightly higher than the Dynamic Range rating for the Nikon D3 (12.5 vs 12.3, see screen grab below). My experience shooting with both cameras refutes that finding, while I agree with the overall DxOMark Sensor Ratings given for both cameras (the D90: 72.6, the D3: 80.6).
The older full-frame Canon EOS 5D also has a lower Dynamic Range rating than the new EOS 50D with its APS-sized sensor, but that difference could be explained by the advances in sensor technology over a two-year span. Fortunately, controversial ratings such as these are few and far between on this site, and I’d wager that there are far fewer conflicts of logic to be found in the pages of DxOMark.com than there are in the pages of many “venerable” magazines and web sites. Take some time to read the articles and white papers on the site in order to understand the rating system, and I think you’ll find the new DxOMark.com website to be an invaluable tool when it comes to choosing your next digital camera. Now if only the same folks behind DxOMark.com could do the same for printers, scanners, projectors, HDTVs, and memory cards, we’d all be in agreement. Right….