Dr Carmel Kent, EDUCATE Ventures’ Head of Data Science, and Dorothy Lepkowska, Communications Lead, examine the data modelling and analytics lessons that can be learned from this year’s exam chaos.

One of the many ways in which Covid-19 has challenged our national educational system is in assessment. Students interacting digitally, studying asynchronously, and using a variety of media to learn are creating a rich set of “learning footprints”.

But how do we harness this data? We can certainly try to shoehorn it into the education system as it was prior to the school lockdown. But we all know that times have changed. Even if teaching, learning and testing return to traditional norms, there is no getting away from how education has been opened up a new world of technology-assisted working.

Those of us who work in this space might think it is about time. Responding to media questioning about this year’s fiasco, Gavin Williamson, England’s education secretary said: “We’ve got a system that is, I believe, the fairest that we can do; but let’s not forget that we’ve been in a global pandemic — we’ve been in a situation — none of us would have expected to be in.” If this is “fair”, then we need to think ahead to smarter, more evidence-based ways of assessment and how we measure it.

The impact of Covid on this year’s examinations is unprecedented. Instead of students sitting exams to attain a grade, teachers assigned pupils with predicted grades, which were then normalised by exam boards to align with their school’s historical performance.

Although this was done by Ofqual to “ensure fairness across the country”, what it has done is actually the complete opposite: it has potentially tied students’ futures to where they come from and pushed down bright students who come from historically less well-performing schools. “This model was developed by Ofqual and exam boards as the fairest way of assessing students’ progress and to reward years of hard work in these unprecedented times.” was the communication coming from the Department for Education. Rather, it has reduced years of hard work to the mercy of a non-transparent, unaccountable and discriminatory algorithm.

The field of data science, which includes tools and techniques form AI, data modelling and, in the case of education, learning analytics has been developed vastly in recent years and is pushing their way into the mainstream. Analytics and data modelling, when applied correctly to appropriate data, have the potential to transform assessment. However, the fiasco surrounding this year’s results harms not just public trust in the education system, but also undermines the reputation, and therefore the potential, of educational data science as a field, and in particular does great harm to the work of Learning Analytics.

Statistical modelling is neither a “good” nor a “bad” tool. It is just a tool, like many others, that needs to be handled cautiously especially if it is being used to determine young people’s futures. Awarding bodies took upon themselves a near impossible task and got it horribly wrong. Data analytics cannot be reduced simply to statistical methods applied to students’ data. It must adhere to ethical standards regarding transparency, accountability and justice.

Transparency

The International Baccalaureate Organisation has been far from transparent about how this year’s results were calculated. Despite calls to be more transparent from, amongst others, the Norwegian Data Protection Agency — which requested it to provide information regarding the International Baccalaureate (“IB”) May 2020 awarding model and grading system” — it has failed to do so. The IBO has been withholding the requested information about “how the factors are weighted in the model and how the calculation is carried out”. If you cannot trace how a certain result has been calculated, how can you criticise it?

Similarly, Ofqual has refused to say whether it will publish data revealing if deprived pupils have been disproportionately impacted by changes made to teacher-assessed A-level and GCSE grades.

Accountability

England’s Department for Education is expected to release details of its appeals process in a few days’ time. It needs to be seen to be taking accountability seriously. The refusal of the IBO to give students any access to its normalisation model potentially makes any appeal process impossible.

Yet already, the appeals process is a subject of intense controversy. The Secretary of State and the Schools Minister both gave the impression to parents and students that there would be an alternative to the grade that they receive today. This alternative would be their mock exam grade. What Gavin Williamson and Nick Gibb failed to make clear was that this mock grade must be the subject of an appeal. Appeals cost money and are much more likely to be submitted by wealthier schools in areas where historically results have been good, thus adding insult to injury for the students from less affluent communities, who are the same students who are most likely to have been downgraded by the data modelling approach.

Justice

The normalisation process for this year’s examinations was introduced to make sure the grades were aligned with previous years. Ofqual’s formal approach was that “if the spread of grades were significantly different this year, it would undermine their value”, and that “students are not advantaged or disadvantaged if their school or college was more generous or harsh when grades were put forward”. The model was intended to make sure that average grades across the country are not significantly higher or lower than previous or future years.

Aside from raising questions about how well the standardised exams usually do in evaluating the students’ abilities, and whether this year can, in any way, be aligned to previous years, we must ask what safeguards and measures the exam boards used to ensure students were not “advantaged or disadvantaged if their school or college was more generous or harsh when grades were put forward”.

Dr Mary Bousted, joint general secretary of the National Education Union, summed up this year’s chaos thus: “Years of misconceived structures in the exams process have come back to haunt the Westminster government. Students have been downgraded for reasons which to them will be obscure. So not only will the result be devastating, but many will discover it has nothing to do with their own performance and everything to do with the past attainment of their school.”

“Student prospects this year were governed by an algorithm, and the unfairness of that process has been fully exposed.”

Dr Bousted makes a good point. If nothing else, this year’s controversies have provided a harsh lesson in what happens when statistical models are employed without any scrutiny.

But it did not have to be this way. The unique set of circumstances around this year’s examinations provided us with an opportunity to start developing a fairer, more accountable and robust system of assessment and measuring student achievement — especially for young people for whom this process impacts their life chances.

For reasons best known to themselves, regulators, awarding bodies and governments have squandered the chance. Politics, wrapped around an inexplicable mistrust of teachers, have disrupted the process, affecting the university and employment prospect of tens of thousands of students.

At the same time, it has inevitably tarnished the reputation of educational data analytics. When the anger and despair have subsided, many will dig deep into the processes that resulted in this situation and will ask how and why this occurred.

We must have the answers, and ensure there are safeguards in place, so it does not happen again.