As the spotlight has intensified on the health and sustainability impact of urban life, it’s rare to find cities that aren’t attempting to both reduce their carbon footprint, whilst also striving to make life cleaner and healthier for inhabitants.

Alas, new research from the University of the Basque Country, suggests that many of the metrics used to measure progress on urban stability suffer from a lack of transparency.

“The last two decades have seen significant growth in the spread of tools to classify and measure urban performance (rankings, indexes, etc.) across both the public and private institutions that use them, in response to different types of pressures encouraging uniformity,” the researchers say. “Naturally, all these tools are useful for guiding and assessing the policies implemented by local authorities in various fields of action, and are particularly prolific in the area of sustainability. Yet there is a lack of knowledge about the actual methodological base underpinning them and which is supposed to legitimize their use.”

The researchers assessed a few hundred measuring tools, and identified 21 rankings, indexes and tools to analyze in depth.  The tools were analyzed across four main principles: the aim and target group they were aimed at; the methodology they used; the transparency of any data they collected and processed; and the way they presented the results.

The analysis found that the aim of the tools and the presentation of the results are commonly available, but the methodology and data governance is often much less accessible.

A black box

The methodologies that were available often had clear weaknesses, especially in their neglect of the complex causalities and transparency around their collection and use of data.  Often processes would be biased, which would result in badly ranking regions being ignored, which results in existing stereotypes being reinforced.

“The possibility of ranking and comparing cities of different dimensions may help to spot those that appear to perform better in various urban aspects,” the researchers say. “That is why these tools are used on occasions by urban managers and public decision-makers to develop an action plan, even though one has to have a clear idea about how the ranking or index has been drawn up, and exercise caution when using it, above all if insufficient information is provided about the methodological aspects and the robustness of its results.”

Overcoming these imperfections is important, as the visual display of data often ensures significant interest from the general public, and by communicating complex data in a simple way, it provides an accessible entry point into the issue for lay people.

The researchers urge the developers of these tools to strive to make their methodologies more transparent, however, so that the public can maintain faith in the viability of the results, and more accurate policy measures can be made from the data.