Vignettes@STS.Next.20

STS enables us to understand the consequences of an increasingly numbers-based educational policy.

In 2012, the Organisation for Economic Co-Operation and Development (OECD) – an international organization and policy think-tank comprising 34 of the wealthiest and most developed nations of the world – decided to form a new sub-unit in its Directorate for Education. This new Division for Innovation and Measuring Progress was created by a merger of two long-lived predecessor units, the Centre for Educational Research and Innovation (CERI) and the Division for Indicators of Educational Systems. By many inside and outside the organization, this merger was interpreted as disadvantageous for CERI, renowned for long-term, conceptual, and qualitative policy research, as it arguably subordinates CERI to the stronger quantitative arm of the OECD. In this commentary, I offer a defense of CERI’s important role within an organization such as the OECD, and caution against some risks of this merger. I hope to speak to both the policy-makers in government as well as academic scholars in the qualitative social sciences who frequently lament how rarely they find certain voices represented in transnational policy contexts and organizations.

Over 50 years, the OECD has become a heavyweight of economic analysis and forecasting and emerged as one of the obligatory passage points for collecting cross-country statistical data. Far beyond its economic mission, the OECD has built a strong reputation for quantitative policy analysis in domains including science, technology, and education. For example, in education it has been administering large-scale international surveys such as PISA (the Programme on International Student Assessment), the results of which have had tremendous impact on national education policies.

Though its quantitative policy analysis has primacy, the OECD has also gained a reputation as a forum for conceptual, critical, and forward-looking thought on emerging “big issues.” CERI has been a stronghold of this lesser known side of the organization. In the words of its long-term director Jarl Bengtsson, “CERI was created [in 1968] in part to provide a complement to such [quantitative and manpower] approaches to education through a more qualitative focus on educational research and innovation,” as well as a response to the “challenges to the ways that society had evolved up to then, symbolized by the revolts of Spring 1968.” CERI’s work has, for example, contributed significantly to the notions of “interdisciplinarity” and “transdisciplinarity,” coined prominently by Erich Jantsch at a 1970 CERI symposium. CERI has also broken ground on how to re-design education and teacher training in light of the dawning information age and computerized manufacturing.

Against this background, how can we evaluate CERI’s absorption into a quantitative unit? From an STS perspective, at least four cautionary points are worth making. First, indicators are performative. They tend to create the very worlds they seek to measure by enacting a discourse space in which unmeasured effects do not exist. They focus attention on performance according to some pre-defined axis of achievement, thus sidelining questions of whether an indicator measures the right thing. An overly strong emphasis on quantitative analysis in particular risks foregoing valuable thinking outside the box, sometimes prompting unintended consequences (e.g. the “studying-for-the-test” phenomenon in education).

Second, the crowding-out of qualitative by quantitative analysis follows a certain ideal of rationality that has dominated the international policy landscape for decades. In this ideal numbers are understood to be objective and, like scientific truth, apolitical. Quantitative analysis promises rational solutions, diminishing the need for messy political processes. Yet, this view overlooks the fact that numbers themselves are immensely political. The political process merely gets shifted upstream to decisions about what to measure and how. While this shift does not take the politics out of policy-making, it arguably makes politics more clandestine and less accessible to contestation. By turning into even more of a data production factory, the OECD may reduce its risk of getting tangled in fierce political debates, but it also loses some of its democratic appeal.

Third, it is a common criticism that numbers, while enabling comparison, come with a certain crude oversimplification that doesn’t do justice to the sometimes vastly, sometimes subtly different contexts of individual countries. Rather than numbers, other factors like political culture, national visions, or historically rooted anxieties of a society might be more indicative of whether or not a certain policy will be successful. These are the stuff of “thick” qualitative analysis. Consequently, a sole focus on numbers might well miss some of the unique opportunities and benefits provided by international organizations capable of such cross-country analysis.

Fourth, a possible erosion of CERI will be disproportionally more burdensome for less wealthy OECD member states. Many small countries cannot afford the speculative, long-term, big picture-type of policy research that CERI has been conducting. In contrast, big countries like the US may easily resort to their own agencies and policy think tanks. Qualitative policy research units at international organizations, then, represent unique intellectual resources for small countries, allowing them to voice concerns about important but still intangible issues and to shape policy agendas about global problems. National problems and solutions that might otherwise go by unnoticed by the international community may have a greater chance of being heard as broader problems and “best practices” by a global audience when picked up by international organizations.

To be fair: None of these potential negative consequences necessarily follow from CERI’s merger. Yet, the consequences of the merger should be assessed early, and in qualitative and non-formalistic terms that echo the CERI mission. STS scholars and like-minded policy analysts should care about the fate of CERI (and similar organizational units in mainstream policy organizations), since it directly affects how emerging policy issues are being framed and whose voices are being heard, particularly in an international context. Quantitative policy analysis is today the bread and butter of informed policy-making. Yet, numbers and their resulting policy agendas are incomplete without the context-rich, exploratory and more conceptual type of policy work that CERI provides, which is why a word of caution about the importance of organizational balance seems appropriate.

Keywords: quantification; cross-national comparison; education and innovation policy

Suggested Further Reading:

• Geoffrey Bowker and Susan Leigh Star (1999). Sorting things out: Classification and its consequences. Cambridge, MA: MIT Press.
• Jantsch, Erich (1970). “Interdisciplinary and Transdisciplinary University—systems approach to educations and innovation.” Policy Sciences 1(4), 403–428.
• Sheila Jasanoff (2004). States of knowledge: The co-production of science and social order. London: Routledge.
• OECD (1986). New Information Technologies: A challenge for education. Paris: OECD.
• James Scott (1999). Seeing like a state: How certain conditions to improve the human condition have failed. New Haven, CT: Yale University Press.
• Deborah Stone (2001). Policy paradox: The art of political decision-making. New York, NY: W.W. Norton & Company.