Guest Post: The reliability genie is out of the bottle at Ofsted

Professor Colin Richards, himself a former HMI, doesn’t hold back when it comes to criticising Ofsted. Here he looks in detail at Ofsted’s recent double inspection trials.

Let’s give credit when credit’s due. In its report (Do two inspectors inspecting the same school make consistent decisions?) Ofsted has tried to meet head-on one of the major criticisms levelled at the Ofsted inspection process – the inconsistency of inspection judgments. Although the report’s “questionable”  title is a tad misleading, the inquiry on which it is based represents a refreshingly more open attitude than the “no problems, no warts” assumption of previous Ofsted inspection regimes.

The report has a number of commendable features. It is suitably detailed and provides a clear account of how the enquiry was conducted, including the inevitable compromises between the ideal and the practical that had to be made in the real-life, non-laboratory circumstances of “live” inspections. Also, it does not claim too much; its exploratory status and findings are made clear both in the report itself and the Chief Inspector’s comments on it.

A particularly noteworthy feature is its tentative tone: for example, “The small number of inspections and the specific context of the sample mean the results of the study should not be generalised more broadly”. Or again, “there is reasonable security that the 24 completed inspections were carried out independently.”  Gone are the cast-iron certainties that dominate the text of school inspection reports. The latter could benefit by being more cautious about their findings as this report is.

The authors acknowledge the subjectivity inevitable in the inspection process. They also valuably demonstrate the inquiry’s concerns to protect the participating schools from undue pressure, though pressure there inevitably was. Also of note was the use (albeit limited) of outside observers to check on how far the participating inspectors worked independently when arriving their judgments.

Unsurprisingly (but perhaps a little prematurely) the report makes much of the main finding that in just over 90% of cases there was agreement between inspectors on the overall  judgment they had to make ie whether the school remained good or whether more evidence  needed to be collected. But remember this was only a binary choice. That finding of  90% plus is good, almost too good to be true, but then two ten-year olds faced with the same decision would probably come out with at least around 50% agreement!

Appropriately the Chief Inspector recognises that the inquiry is but the start of a more wide-ranging, evidence-informed process. With its focus on short inspections of primary schools of above-average size the study is a relatively easy place to start, though that “easiness” is complex enough, as the report illustrates.

The inquiry’s approach needs to be replicated – initially, in more than 24 primary schools. It’s probably unlikely that that replication will result in another 90% plus figure, but it may and it would be good news if it approximated that percentage.
Ofsted needs to tackle the reliability of inspector judgments in full inspections of both primary, special and secondary schools where more than binary judgments are involved. Such wider replication would be logistically very complex and demanding of Ofsted’s resources but there will be pressure to undertaken these studies now that the reliability genie is out of the inspection bottle!

Rightly or wrongly, there is more concern in schools over the reliability of Ofsted Inspectors’ judgments, rather than those of HMI but such OIs were not used in this initial inquiry. They need to feature prominently in any future reliability studies.

The report attributes the high-level of inter-inspector agreement in part to Ofsted’s own quality assurance procedures and to its training programmes but provides no evidence to support that assertion. Here, the report might be claiming too much.

So, to quote the report’s title, do two inspectors inspecting the same school make consistent decisions?  The tentative answer is positive but only in respect of short inspections of  above-average-size primary schools each undertaken by two HMI who have undergone unspecified training and who are faced with just a binary choice –  the school remains good or more evidence is needed. That’s certainly an endorsement but inevitably a very limited one.

And then there is the even more vexed and problematic issue of the validity of inspection processes – an area which the Chief Inspector would like to explore. She would do well to heed the words of Geoffrey Vickers in his book, The Art of Judgment: value judgments “cannot be proved correct or incorrect; they can only be approved as right or condemned as wrong by the exercise of another value judgment’”. Inspection is value-saturated; its value judgments cannot be proved, only justified.

 

A version of this first appeared in the TES

See previous posts from Professor Richards

 

Please let us know in the comments or via Twitter…

 

Don’t forget you can sign up to receive our daily email bulletin (around 7am) with all the latest schools news stories. Your details will never be given to anyone else and you can unsubscribe at any stage. Just follow this link 

We now have a Facebook page - pls click to like!

 

More recent posts...

0-the-school-leadership-journey

Exam results are prioritised over pupil wellbeing, teachers say
Exercise levels decline 'long before adolescence'
Categories: 1st POST and Guest Post.

Comments

Let us know what you think...