Outcome Measurement II
BLOG ARCHIVE
(Originally published 14 August 2010)
My last piece discussed the great progress which has been made in outcome measurement over recent decades. This has been hugely important in providing a better information base for performance management and budgeting. Perhaps the most important single methodological development in this area has been the increased use of surveys.
Take the example of public programs which aim to reduce the discrimination based on race, sex or other criteria. Measuring outcomes in this context requires, as a starting point, measuring the level of discrimination prevalent in the community. In the past, this was done – to the extent that it was measured at all –on the basis of statistics of the formal complaints of discrimination lodged with the police or other public authorities responsible for protecting people from such discrimination (e.g. anti-discrimination commissions). However, complaints measures understate the problem, because many people will experience discrimination without taking formal action. Moreover, the degree of that understatement is variable and unpredictable. A rise in complaint rates might, for example, occur with no underlying increase in the incidence of the problem (due, for example, to , increased awareness of their rights on the part of the victims).
Today, increasing use is being made of surveys to measure the incidence of discrimination and a range of other social problems. To give a concrete example, in the United Kingdom, government has for some years reported a formal indicator of the level of employment-based discrimination – namely, the “differential gaps in perception of employment-based discrimination”, defined as “the composite change in the gap between [discrimination] perception rates of disadvantaged groups by comparison with non-disadvantaged groups”. This measure is based on survey results which report the perceptions of people from defined disadvantaged groups of their experience of discrimination in employment. The survey results are drawn from the Citizenship Survey, a broader biennial social survey conducted by the Home Office (the UK’s interior ministry), which is also used as the basis for a number of other important measures of the state of British society.
The increased use of surveys provides a means of overcoming a number of limitations on performance measures which are largely drawn from the data collected in the course of service delivery by public agencies. Survey data is not only capable of providing a fuller picture of the extent of a social problem government is trying to address. It can also provide a more timely picture, by capturing outcome variables which will only much later manifest themselves in demand for government services. For example, survey-based data on alcohol and tobacco consumption rates provides give a much earlier warning of the extent of the problem which preventative health services need to address than do levels of alcohol and tobacco-related disease, which arises only after years of abuse. Surveys can pick up measure variables relevant to outcomes which take time to manifest themselves. For example, in measuring the outcomes of university education, graduate employment rates, salary levels and similar measures are now widely used. These measures are derived from surveys of graduates at specified time intervals (e.g. six months, one year, two years) after graduation. This is an example of the use of surveys for the purpose of so-called “longitudinal” studies.
Surveys are, of course, the most important means of producing reliable client satisfaction indicators. Following the private sector, leading governments have taken huge strides in developing indicators in which clients are able to tell their own story not only about whether the services they receive are effective (i.e. outcomes) but about service quality.
Of course, surveys are not a flawless performance measurement technique. Issues may, for example, arise about the accuracy of self-reporting. People will tend when surveyed – even when anonymity is guaranteed – to underreport behaviors of which they are ashamed. Particularly problematic are surveys which ask people for their assessment of the magnitude of community problems which they do not necessarily directly experience themselves. For example, surveys of community perceptions of the level of crime and insecurity have shown peaks which results from media hysteria entirely unrelated to the underlying statistical trends in crime rates.
Reliable surveys are, of course, cost money to undertake. At the same time, a badly designed survey can do harm. Surveys are therefore a performance measurement technique which relatively poor countries will only be able to use on a very selective basis. But even in such countries, they should play an important part in moving from a performance indicators based primarily on inputs and activities to indicators based primarily on outcomes and outputs.
Overall, then, it is clear that critics who regard outcomes as largely immeasurable have got it very wrong, and have failed to notice the huge progress which has been made in this area by leading countries over recent decades.