See the full lecture on which this blog is drawn here.

There is much excitement, for good reason, about development data. This comes from a combination of the imperatives for more and better data from the Sustainable Development Goals (the SDGs), from new conceptions of what should be counted, and from new ways and technologies for counting and measuring social life. In this blog, drawn from my inaugural lecture, I explain why each of these elements are indeed exciting. I then outline why we must be modest in our expectations of how those data might inform policy. There have simply been far too many cases where the data are irrelevant to policy. In the second blog I will take the argument a step further to consider the limits even of policies which are profoundly informed by data.

The Sustainable Development Goals were adopted by the United Nations in September 2015. There are 17 goals, covering all the traditional interests (poverty, health, gender equality and education) as well as a whole new set including energy provision, infrastructure, inequality and many more. These 17 goals have, collectively, realized 169 targets that they aim to achieve by 2030, and these targets have produced 230 indicators which will be used to determine whether or not they have been successful.

The SDGs are auspicious for anyone interested in development data because they set up a highly specified vision of what a prosperous world should look like. If these goals are to be successful on their own terms then this will hinge upon using the right sort of data in the right way. This means that getting better data, and understanding the limitations facing these data, becomes all the more important.

It seems that we are now awash with new ways of collecting data that could be useful for the SDGs. Some of these derive from using remote sensing, new algorithms and digital data. The spatial precision of satellite data has improved, as has their temporal frequency, and machine-reading of these data can be used to understand changing land use and household investments. The spread of mobile phones, and the growth of social and economic activity on social media, provide new proxies of human development, and new arenas of eminently measurable activity.

But it is not just the means of measurement which is exciting here – it is what we chose to measure and why. The SDGs are more broad-minded than the MDGs they have superseded. The MDGs were concerned to reduce poverty, and used poverty lines to determine who was poor. The SDGs want to eradicate all forms of poverty, and will be using a broader set of measures of what constitutes wealth and poverty as criteria for success. I find this particularly exciting because if we take a broader notion of what wealth and poverty mean we can end up seeing societies in different parts of the world in new ways.

I can illustrate this with a research project that we are completing in Tanzania, on which I am working with Christine Noe and Moses Mnzava from the University of Dar es Salaam. The project examines rural livelihood change and prosperity in Tanzania, a country which has long been poor, but which has seen dramatic increases in economic growth in the last 30 years or so. The trouble is that there were numerous signs that, according to poverty line data, this wealth was not being shared, particularly in the rural areas where most poor people can be found. Using different measures of poverty, which hinge on controlling and using assets, we are finding families have become wealthier. They have been able to invest the returns from farming activity into better homes, agricultural equipment and education.

Assets matter partly because they feature most prominently in the local definitions of wealth. But they are also centrally important to examine in any study of long term poverty dynamics, because the rural poor invest in assets. Consider this statement from a participant in a focus group in south-east Tanzania which I heard earlier this year.

We get money seasonally. [We] have earned three millions shillings, or two million shillings. Some people when they get money after the harvest they buy a TV, or solar panels, or all manner of things but now if he’s struck by some problem and needs 50,000 shillings he’ll have to wait 5 months.  In July if you ask someone for 500,000 shillings they will give it to you, but go to them in November and ask to borrow 200,000 to deal with a problem and they will tell you I have nothing, I have bought a TV, I’ve bought a plot, I’ve bought bricks.’

Assets matter to poor rural families because they provide a focus of investment that is particularly valuable in the absence of good banking services, and without regular, frequent sources of income.

This graph shows the growth in assets in Mtowisa in south west Tanzania where I once lived for a year in 1999-2000. It shows investment in houses, metal roofing, oxen and so one.

You can see the phenomenal growth in housing in that village in just ten years in this google earth images. This shows the village in 2003.

And this in 2013, and with all the newly built houses with metal roofs in that 10 year period circled in black.

But, and here is the catch, in every single one of the 20 plus village where we have worked on the livelihood change and prosperity project, the precise nature of the change, its drivers, its timing, the gendered distribution of benefits within households, the role of elites, the role of emigration or immigration, the crops and agricultural innovation involved – in every single case there is something different going on. The story here is that there is no single story.

And this is part of the research agenda that new conceptions of poverty and prosperity can unleash.  We can challenge problems of inequality by examining the different understandings of what wealth is and how it is distributed and by understanding the diversity of stories that need to be told to combat central notions and of what constitutes wealth and progress.

A second reason to be excited about development data emerges from an increasing conglomeration of questions about the accuracy and basic validity of data used in development. The work of scholars like Morten Jerven who has combined a quantitative analysis of the flaws and inconsistencies of GDP data, with an ethnography of the construction of those data, to argue that basic notions such as GDP have been mismeasured for years. Governments have not been able to measure their economies, particularly where so much activity is informal and very difficult to observe and count. Famous examples of these failings include the massive increase in wealth in Ghana that occurred overnight when its GDP baseline was recalibrated.

I am particularly pleased to be taking part in projects which extend this sort of critique. We are doing so specifically with respect to agricultural data, which are notoriously inaccurate because so many small-holders conduct their affairs informally and are generally rather suspicious and not necessarily truthful when answering surveys.

In the SAFI project lead by Phil Woodhouse and diverse partners internationally we have been able to show that, conceptually, ideas of what irrigation is are too narrow in these countries. Irrigation appears to be thought of as something requiring large concrete intakes, engineered channels and carefully planned division points. It is not deemed to refer to local practices of rice cultivation which involve small temporary dams in low lying areas, and water harvesting on raised ground. That, as Phil puts it, is just moving water around fields. It is not real irrigation.

But these official definitions are unsatisfactory, they do not seem to explain or describe farmer behaviour. Surveys of farmers in Tanzania suggest that many are growing rice, but that only 5% of them are irrigating it. This is unlikely. We predict that irrigation activity, defined as farmers’ deliberate management of temporary flooding in their fields is much more extensive. Preliminary results from analysis of Sentinel II radar data suggest that this is precisely the case – we can find evidence of irrigation that is one order of magnitude higher than agricultural census data predicts.

This work excites me because it combines all that is new in development data, for it re-conceptualises what we need to examine, challenges existing data, and proposes different data sources. With fresh thinking, interdisciplinary collaboration, and fancy kit like radar and good algorithms, we can shed new light on old problems.

But before we get too entranced by this brave new world of more relevant, accurate and believable facts we must remind ourselves of the limited role that data can play in tackling development problems. I have found repeatedly in my research that facts about things as diverse as basic aspects of environmental change in East Africa, to the role and influence of celebrity advocacy in the UK, seem largely irrelevant to the public debate, and policy discourse, about those topics. Data-informed policy is rare; often policy can be remarkably data free.

As we learn in any basic research methods class, new data and methods present us with a set of insights and then another range of problems that come with them. So, for example, there is a great deal of excitement and interest in the potential of new digital data derived from mobile phone use, and its ability to reveal new facts about social change. But there should be a deep concern for the inequalities and blind spots of these new data, that derive, for example from the unequal gendered access to and use of mobile phones. Fighting inequality requires continual vigilance lest new data renew marginalisation.

And, if we ever get good data, then a different set of problems emerges. Consider, for example, what happens when the data clearly demonstrate that international development goals are not achieved. For example, several of the millennium development goals failed to reach their targets. Reductions in infant mortality of 67% was demanded by 2015; but reductions of only 53% achieved. Reductions in maternal mortality of 75% were demanded by 2015 and only 45% achieved.

We must place these failures in context. David Hulme and Armando Barrientos have argued these standards have made a difference, they have raised interest in poverty alleviation strategies by national governments. In the absence of these goals achievements could have been even lower. But these are not, as one commentator claims, a set of promises that the world makes to itself. For these are promises which can be broken, they are disengaged from political processes. There is no mechanism for holding to account leaders who do not stick to these grand plans. Failure in these contexts is not costly, failure is free.

So, revolutions in development data are absorbing, exciting and reveal all sorts of new aspects about social life and the environment. We need to embrace them as much as possible. But we cannot expect that these new data will lead somehow, to evidence-informed policy. The barriers to that are multiple and enduring. They will be as impervious to new development data as they have been to old.

There is, however, a more profound objection that we have always to remember surrounding development data, and this is best visible not in the weaknesses of development data, but when they are strong – and that is the subject of the next blog.

This blog first appeared on the SIID site here.