Wednesday, September 17, 2008

Manufacturing an educational crisis

Here’s an example of Stacking the Deck and Misusing Information from Australian Deputy Prime Minister Julia Gillard (or rather, one of her lackeys) and co., done in order to manufacture a crisis in Australian education to justify the instigation of a pseudo-revolution. (Worse than this, the journalists who reported this story couldn’t be bothered to check the facts for themselves… or didn’t know how to read a graph.)

In this news story and the various versions of it, Gillard, with the help of compliant journalists, make Australia’s “performance” relative to other OECD countries in education look inadequate:

An OECD report ranks Australia 19th out of 28 developed nations on overall education spending and second last to Belgium on public education institutions investment.

The Rudd government and the Australian Education Union (AEU) say the report proves the former coalition government neglected education during its nearly 12 years in office.

"This is a report card that paints a bitter legacy for the nation from the Howard years in education - insufficient investment and falling behind the standards of the rest of the world," Education Minister Julia Gillard told reporters in Melbourne.

… The report shows only Belgium spent less on public education institutions than Australia among the OECD countries.

In 2005, just 0.1 per cent of GDP was spent on pre-primary institutions, compared to the OECD average of 0.4 per cent, ranking Australia equal 24th out of 26 countries in that category.

Tertiary education expenditure was only 1.1 per cent of GDP, also less than the OECD average.

The indicators (and associated statistics) they refer to are all true. However, they’ve only referred to two of the indicators the OECD Education at a Glance 2008 report looked at. As such, by cherry picking these two points, Gillard et al. are guilty of Stacking the Deck and Misusing Statistics.

This is a 500 page document with over 100 indicators… Moreover, on most of the economic indicators that I looked at, Australia comes up pretty well in terms of spending and has increased spending from 1995 (when Labor was in power – losing the 1996 election to the Liberals) to 2005 (the Liberals still in power). Of course, over this period most of the state governments were Labor and the economy as a whole was growing, so to tease out where an increase in funding came from and why is a little beyond what I want to do here. The point I’m making is that to pick out two bits of data and say it’s a legacy of the Howard government is a complete Non Sequitur.

In the finest traditions of political expedience and inept journalism, I’ll cherry-pick some data now. For example, page 231 of the report points out that:

For all levels of education combined, public and private investment in education increased in all countries by at least 8% between 1995 and 2005 in real terms and increased on average by 42% in OECD countries. Australia, Denmark, Finland, the Netherlands, New Zealand, Portugal, the Slovak Republic, Sweden and the United States increased expenditure on educational institutions by 30 to 50%...

Page 237 of the report, Table B2.1. - Expenditure on educational institutions as a percentage of GDP , by level of education (1995, 2000, 2005) tells is the following for the total levels of all education in Australia. In 1995 (Keating’s Labor government) it was 5.3 %, in 2000 (Howard’s Liberal government) it was 5.6 % and 2005 (Howard government) 5.8 %... So, there you go. Howard’s legacy is an increase in overall spending.

The rhetoric from both sides of politics regarding education in Australia is worrying given they seem hell bent on looking to the US for ideas. I won’t go into details about the proposed education revolution as it’s too big a topic. I will point out, however, that Australia consistently out performs the US and UK (another model often cited) on the PISA performance in science. This raises the question as to why Gillard is looking at the school models used in the US and UK? Given the data, the answer is beyond me. Perhaps US and UK students can do maths and write better than Australian students? Nope

I’ll combine these two points now – Australia’s spending on education and Australia’s educational outcomes:
Indicator B7 (page 304): How efficiently are resources used in education? This indicator examines the relationship between resources invested and outcomes achieved in upper secondary education in OECD countries and thus raises questions about the efficiency of education systems.

High spending per student cannot automatically be equated with strong performance by education systems. Spending per student up to the age of 15 in the Czech Republic is roughly one-third of, and in Korea roughly one-half of, spending levels in the United States. However, while both the Czech Republic and Korea are among the top ten performers in the PISA 2006 assessment of science achievement among 15-year-olds, the United States performs below the OECD average. Similarly, Spain and the United States perform almost equally well, but while the United States spends roughly USD 95 600 per student up to the age of 15 years, Spain only spends USD 61 860.
And then on page 306:
Student performance and spending per student. Table B7.1 compares countries’ actual cumulative spending per student between the ages of 6 and 15 in 2005 on average, with their average student performance on the science literacy scale of PISA 2006 and with other economic and social indicators…

Chart B7.2 shows a positive relationship between cumulative spending per student and mean science performance. As cumulative expenditure per student on educational institutions increases, so does a country’s mean PISA performance in science. However, the relationship is not a strong one; cumulative expenditure per student in fact explains merely 15% of the variation in mean performance between countries…. In other words, spending levels per student cannot automatically be equated with the performance of the education system as measured by PISA.
Australia came 6th overall in the PISA performance in science in 2006, behind Finland, Canada, Japan, Estonia and New Zealand. The UK was 11th and the US was 23rd. On page 309 we get one of the most poignant statements:
The fact that similar levels of expenditure between countries can mask a variety of contrasting policy choices made by countries goes some way to explaining why simplistic comparisons of student performance and expenditure levels fail to show strong correlations. It remains for further analysis to examine what influence these different policy choices actually have on quality and equity of learning outcomes.
I point all this out to show that a simpleminded discourse played out in the media does no-one any favours. From reading the popular media over the last few years one gets the impression Australia’s educational system in on the brink of collapse and Australian students are performing badly relative to other countries. It may be politically expedient to further this myth, it may grab headlines, but if it forms the basis of poor policy decisions – like looking to the US where USD 30,000 more is spent per student (from the age of 6 to 15) than Australia and yet they perform extremely poorly and Australia performs very well in the PISA science testing (page 307) - it becomes potentially disastrous. On the other hand, it might simply be a way of justifying an increase in spending on education - in particular in areas of genuine disadvantage. However, as a rule of thumb I'm against humbug, even if in the name of a good cause.

This is not to say all the points Gillard talks about are wrong or bad ideas; it’s just that if you’re right, you have no need to resort to humbug. For example, Gillard wants more accountability and for schools to make achievement data public:

...a significant positive association between schools making their achievement data public and having stronger results. (Page 9 of the Executive Summary.)

Of course, this implies there is little accountability in Australian schools which is simply untrue, but that's a whole other post...

Here's the direct link to the PDF of the OECD report: