Benchmarks

A lot of discussions that we have been having in these past few years in the tree climbing community regarding new equipment and techniques being introduced revolve around testing and certification. Essentially it is a discussion about benchmarks, or points of reference, that offer an objective reference point or a defined set of performance criteria that you can measure a given technique or tool against.

That seemingly objective benchmarks may not always be all they seem to be is illustrated in the following example…

As part of an effort to reduce CO₂ emissions, the European Parliament has set a target of 150 grams of CO₂ emitted per kilometer (g/km) by the end of 2015 for all new vehicles. This is a considerable progress from 145 g/km in 2013 or 175 g/km in 2008.

On the face of it, this is a success story in the fight against climate change.

But things are not always as they seem…

A report by the International Council on Clean Transportation (ICCT), an independent research organization, comes to different conclusions. The basis for CO₂ regulations regarding motor vehicles are test values that are assessed under laboratory conditions according to pre-defined driving profiles, the so called New European Driving Cycle (NEDC). This data is then referenced by the automobile manufacturers in their sales pitches.

The ICCT study  «From Laboratory to the Road» concludes that when comparing NEDC data to real-life studies on emissions, based on data from Germany, the Netherlands, the United Kingdom and Switzerland, there were significant discrepancies. The average difference from laboratory to real-life values in 2001 was about 8%, by 2013 the mismatch had risen to a considerable 38%.

The study further remarks upon the fact that differences can vary from manufacturer to manufacturer. As a rule of thumb it can be said that the larger and more powerful the automobile, the more pronounced the difference between theory and practice. According to Spritmonitor.de, in 2006 with certain models of BMW’s 5 series, Mercedes Benz’ E-class and Audi’s A6 the difference between the advertised and actual consumption was 10% – by 2013 this figure had risen to 45%.

One of the reasons for this mismatch is that the test set-up allows for air conditioning units to be turned off, the use of low-friction tyres and high-performance motor oils and for the front radiator vents to be taped off to reduce drag. Also the on-board computer recognizes the test cycle and optises power consumption by turning off the fuel-hungry alternator.

All of this leads to a a significant misrepresentation of true consumption rates.

I stand by the fact that for the end-user, benchmarks are essential to making well-informed choices. We need objective, clearly defined reference points to measure the performance of a given piece of equipment against. Certification is one example for such benchmarks.

But as the example above shows, it is essential that such benchmarks reflect the actual use the piece of equipment is being used for. Benchmarks that only are relevant in a pristine laboratory environment, far away from dirt, sweat and knocks and blows are not very meaningful.

This is one of the reasons that one of the focuses we have had over the years at treemagineers is to consider whether equipment is actually fit for porpoise… no, I must not get into the habit of saying that, fit for purpose and the intended use actually correlates with the use it was designed for.

Benchmarks are a good thing – so long as they assess the intended use of the equipment in a realistic fashion. Should this not be the case, they become hollow, can be misleading and may encourage the drawing of wrong conclusions.