How Technical Camera Reviews Can Sometimes Be Faulty

Photography Tips

Camera technology changes less drastically than it used to in the early days of digital photography. Yet, reviewers and consumers have been focusing on the technical image quality when making buying decisions. Not to forget the human error that creeps in, which can render the test meaningless or misleading. Tony Northrup explains why you should be very careful before blindly believing technical camera reviews:

Inconsistent Lighting Leads to Meaningless Results

We can see a lot of “real world comparisons” between cameras. That’s great. But, what we need to consider is that in the real world, conditions vary by the second. Northrup noticed that on a cloudy day outside, there was a difference of 2 stops of light within a matter of just 10 seconds. And on a clear day, he lost 1/3 of a stop of light within 30 minutes by the movement of the sun. But, with sensor technology improving ever so slightly around 1/10 of a stop, the comparison is rendered useless if the photos are taken in such varying lighting conditions.

Such changes in light intensity are not just limited to outdoor photography. Indoor photography using artificial lighting can have similar issues. If the room isn’t properly sealed from the light outside, variations in light conditions outside will alter results indoors. Similarly, Northrup also explains that the output of artificial lighting (such as LEDs, fluorescents, incandescents) changes with time as they heat up (1/4 stop in 30 minutes). And again, with only about 1/10 stop of improvement in camera sensors, the whole comparison becomes meaningless.

So until and unless the lighting conditions are fairly consistent throughout the testing period, the results of any image quality tests these days are insignificant.

T-stop Matters Too

For those of you who are not familiar, f-stop is the ratio of the focal length to the diameter of the aperture. On the other hand, t-stop measures the amount of light that is actually being transmitted through the lens on to the sensor. Even though theoretically different lenses at the same f-stop should let equal amount of light through them, because of the difference in construction of the lenses, the amount of light being transmitted varies. Hence, two different lenses at the same f-stop value can have different t-stop values.

So, by using different lenses on different camera bodies, reviewers are actually comparing the cameras that in reality are receiving different amount of light. The difference can be around .2 to .3 stops of light which is very significant compared to the .1 stop of improvement in the camera sensor. Unless lenses having the same t-stop values are used, the comparison is flawed.

Sharpness of the Lenses Being Used

Comparing cameras mounted with lenses having different levels of sharpness is another flaw in the review process. It’s important to understand that sharpness is not just determined by how good the sensor is. Sharpness is actually determined by the combination of the lens and the camera. So the next time you see a reviewer compare cameras with a kit-lens in one camera and a $1000 lens on the other, you should understand that the review is pointless.

Numbers Aren’t Everything

Camera geeks love specs that can be measured: the megapixel, the improvement in dynamic range, the improvement in ISO performance. That’s why most of the reviews we see around spend most of the time reading off from the spec sheet. But, there are other subjective things that matter more.

Better focus accuracy is more critical and important than more megapixels. A camera with a good in-body stabilization system pays huge dividends in better image quality than cameras with higher megapixels or crazy ISO numbers

Nailing sharp focus on the subject is critical in photography. But focus cannot be measured in terms of numbers. So, it’s vital to understand how quickly and accurately the camera attains focus. Similarly, features like in-body stabilization allow users to shoot at a lower shutter speed, keeping the ISO low. Hence we can get a much cleaner image. Instead of comparing numbers, it’s also essential to focus on the practical features that camera manufacturers are putting on the camera to make the life of a photographer more productive.

Lens Test Flaws

As much as reviewers tend to have some flaws in judging the actual quality of cameras, they also tend to have flaws in their methodology when it comes to testing lenses. Here’s what Northrup has pointed out in his video:

• Shooting through glass. This is specially true during initial launches of lenses. When reviewers are trapped inside their hotel rooms due to bad weather or if they are not allowed to take the lens with them outside, they tend to shoot through the window glass which tremendously degrades the final quality of the image. Don’t even bother to listen to any comments made on photos that are shot through glass.
• Atmospheric conditions. The quality and sharpness of a lens cannot be judged by looking at a photo taken outdoors. Atmospheric conditions. Haze, especially, adds a lot of interference between the subject and the lens and the photo ends up having less detail. To truly understand the image quality of a lens, it needs to be tested in a controlled environment.
• No focus is perfect. Focusing systems are never precise enough to rely on them completely. If we focus on a subject 10 times, every time the result can come out different. The focal plane is so shallow that if we even miss by an infinitesimal amount, the final image gets a bit off of focus. So, judging the sharpness of an image based on a few random photographs doesn’t make sense as focus tends to vary naturally.
• Depth of field isn’t technically real. As Northrup discusses in the video, the focal plane is a two-dimensional thing and it is very shallow. Anything in front of and behind the focal plane is technically out of focus. Mathematical calculations may say that the subject is within the depth of field but that doesn’t matter.
• Everything shakes. While testing lens sharpness, the reviewers must make sure that the camera is very stable. Things like the wind, cars outside the house, or even people walking in the house can cause the camera to shake and introduce “unsharpness” to the image.  Precautionary measures should be taken by the reviewers, such as using a heavy tripod, testing the camera on a stable surface, and using camera options to reduce camera shake to get the best result possible.

For us as consumers, Northrup suggests that we pay more attention to the focus aspect of a lens than the sharpness. A sharp lens is of no value if it frequently fails to attain focus on the subjects.

Focusing Test Flaws

The focusing system is another important aspect that consumers pay close attention to while making a decision to purchase a new camera. Northrup points out some of the focusing test flaws and warns you not to get fooled by them.

• Performance in unpredictable environment. A focusing system may perform brilliantly in a predictable test environment (e.g. focusing on a subject walking toward or away from a camera), but it may fail drastically when the movement is fast and unpredictable (e.g. a sports scenario). While this factor cannot be expressed in terms of numbers, pay close attention to the behavior of the focusing system before jumping to a conclusion.
• Focus point spread. Northrup gives an example of the Canon 6D Mark II, which has a very good focusing system, but all of its focusing points are concentrated on the center of the sensor. On the other hand, we have the Sony A7III which has its focus point spread almost from edge to edge. Why focus and recompose with the 6D Mark II when you can conveniently focus on your subject on any part of the frame using the Sony A7III? Having a better spread makes your work faster and also increases the chances of having the shot in focus. So pay attention to what the reviewer has to say about the spread of the focus points.
• Not accounting for eye detection. This is also one of those factors that adds more practicality and functionality to the camera. In traditional DSLRs, you need to continuously move the focus point up to the subject’s eye and grab focus. If the composition changes, the photographer will again need to move the focus point and repeat the process. With their brilliant eye auto focus systems, Sony and Fuji have made this entire process possible with the press of a button. It’s important for reviewers to consider such subjective benefits for consumers.
• Not accounting for focus point selection speed. What good is a fast auto focus system if the user is unable to choose the focus point of his/her liking conveniently? Dedicated joysticks are essential to help the user select focus points without fiddling much with the camera.
• Not accounting for subject tracking. It is important to consider how well the camera tracks the movement of a subject across the frame. A good auto focus system must be able to lock on to the subject in a snap and stick with the subject as it moves. Otherwise the shots will end up out of focus.
• Don’t get fooled by the back of the camera. On the back of the camera, the shots may seem to be in focus. The green dots and boxes shown on the back of the camera may seem impressive but sometimes the image may be out of focus. This is especially true while using fast lenses which tend to have really shallow depth of field.

The bottom line is that reviews are not just about reading out specifications. There are many other factors that cannot be measured but are a huge deal either to make the working experience with the equipment better, or to improve the overall workflow. Now we hope that you have a better idea of what to look out for while going through reviews of camera gear.