There has been a recent explosion of interest in the effectiveness of education systems around the world, largely driven by international studies that compare the performance of large samples of students from a wide range of countries.
Such comparisons made in these studies such as the Organisation for Economic Co-operation and Development’s Programme for International Student Assessment (PISA) or other international reading and maths tests, have become a key factor in educational policymaking across the world.
Of course, how well children perform in maths and reading tests is not the only important aspect of an education system. In a resource-constrained environment, the costs of a good education cannot be ignored. In that sense, a new report by GEMS Education Solutions, the London-based consultancy wing of Dubai-based company GEMS Education which runs schools around the world, is certainly likely to spark debate around both the efficiency and effectiveness of education systems. But such a ranking is very problematic.
The authors of the Efficiency Index have used scores on the PISA tests and related these to (financial) inputs into the education system, of which they find two – teacher salaries and class size – to be significant.
The index is therefore a measure of how good a country’s pupils score on the PISA test, given how much (or little) a country spends on its teachers. Using these variables, the authors calculate an index for each of the 30 countries they studied, and rank them according to the efficiency index.
Finland, Korea, the Czech Republic, Hungary and Japan do particularly well, and the UK ends up in the top half of the table.
The misuse of league tables
So far, so acceptable, as a starting point for debate. But there are a range of problems with how the data is presented. The authors make three tempting but serious errors that bedevil a lot of the writing on PISA. The first is “league table-itis”. The authors rank countries in an efficiency league table, and there is little doubt that this is one of the main things readers and the press will pick up on.
But there are problems with the rankings. Like all tests, PISA (though technically very well designed) is not perfect. The test score contains what is known as “measurement error”, and so do estimates of things like pupil and teacher ratios.
This means that every point score, like the efficiency index scores, come with a margin of error. As a result, small and even medium-sized differences between countries in rankings like this one are unreliable. In the Efficiency Index, for example, the UK is ranked 11, and France 12. The difference between the two countries is 0.02 on the ranking – this is essentially meaningless. We can possibly distinguish groups of countries, but we can’t actually accurately rank them from one to 30 as is done here (and, to be fair, in many league tables like it).
Not comparing like with like
The second issue is that of comparing like with like. Of course, countries as varied as Brazil and Finland differ on many factors other than their spending on teachers. So we have to be very careful when we compare them.
The authors of the report acknowledge that this is an issue, and write, for example, that cultural differences which may be important could not be accounted for in the data. But then they nevertheless continue to make comparisons as if in fact only the two things they are looking at (class size and salaries) matter.
For example, that Finland is able to attract high quality teachers at relatively low salaries may well have something to do with the esteem in which the profession is held, which may make this model hard to replicate in countries where this is not the case.
Cause, effect and educational policy
The third, and most problematic, error is the way the report plays fast and loose with causal attribution. As is well known, the fact that two things are correlated with one another doesn’t mean one causes the other. Again, the authors acknowledge this problem, but then proceed to ignore it.
This is done most egregiously in the tables in which the authors show what countries should do to reach the same level of efficiency as Finland. In the case of the UK, this means either reducing teacher salaries by 10%, or increasing class sizes by 24%. Other countries, such as Chile or Indonesia for example, would need to reduce class sizes by over 90% to reach Finland’s level or increase teacher salaries by over 200%.
This simplistic conclusion, based on a rather limited set of analyses, seems a particularly dangerous policy recommendation if we want to attract high quality people to the profession.
Efficiency in education is important, and is something both those making policy and researchers should pay more heed to. There is therefore room for high-quality research on this topic. Unfortunately, this isn’t it.
