Were South Africa’s Matric 2016 Results Faked?
Were South Africa’s Matric 2016 results faked by Umalusi (Council for Quality Assurance in General and Further Education and Training)? 28 out of 58 subjects had their marks adjusted upwards!
Is education likely to improve atop growing unemployment, civil restlessness and community breakdown? A layman would surely look at newspaper headlines boasting about the improvement of South African education but wonder if that’s possible within a South Africa that is constantly degrading.
Gavin Davis, the Democratic Alliance’s Shadow Minister for Basic Education, has written this open letter to Umalusi (Council for Quality Assurance in General and Further Education and Training) over what he believes is an inappropriate upward adjustment in the matric pass mark:
“Dear Dr Rakometsi.
Gavin Davis
I am writing this open letter to you in the spirit of constructive engagement. In doing so, my intention is to safeguard the integrity of our examination system, and to ensure that all learners get a fair deal.
This open letter follows my personal correspondence with you, which contained a series of questions related to the standardisation of the matric marks. In your rather defensive reply marked ‘private and confidential’, you declined to answer my questions.
As I am sure you will appreciate, as a Member of Parliament I have a constitutional obligation to “maintain oversight of the exercise of national executive authority.” The standardisation of our matric marks – as a joint exercise undertaken by Umalusi and the Department of Basic Education (DBE) – is such an exercise of executive authority.I would therefore be failing in my constitutional obligations if I did not raise questions and concerns relating to the standardisation process when they arise. It is worth adding that your refusal to answer these questions is a serious failure on your part. Indeed, it is a contradiction of one of the principles of standardisation as outlined in the preface of Umalusi’s own guidebook entitled “Understanding Statistical Moderation” which states:
“Sharing this information forms part of Umalusi’s commitment to making its processes transparent to all who have an interest in the examinations Umalusi quality assures and certificates.”This open letter is your opportunity to re-affirm Umalusi’s commitment to transparency, and I look forward to your open and honest response. I am going to divide the rest of the letter into three sections to make it easier to read and respond to.
1. THE GENERAL TREND TOWARDS UPWARD ADJUSTMENTS
At the standardisation meeting in Pretoria last Friday, we learned that 32 of the 58 subjects had their marks adjusted (compared to 29 in 2015). Of the 32 adjusted subjects, 28 had their marks adjusted upwards and only four downwards.Some of the subjects saw a dramatic upwards adjustment. The following six subjects saw the biggest upward adjustment from the raw mark:
According to Umalusi and the DBE, adjusting the raw mark upwards is justified if the exam paper was demonstrably more difficult (i.e. more cognitively demanding) than previous years. However, no evidence has been put forward to demonstrate that these papers were of a higher standard.
I noticed at the standardisation meeting that the starting point for adjusting the marks was not the papers themselves, but the results. In cases when the raw mark was worse than last year’s, the DBE went back to the paper and found difficult questions to explain the drop in the raw mark. The DBE then motivated for the raw mark to be adjusted upwards accordingly.
This methodology seems incorrect. It would seem that – as a general principle – the cognitive demand of the papers should be assessed independently of the marks. This should preferably happen before the papers are written so that the need to adjust the marks afterwards is minimised.
The obvious problem of using the raw marks as an indicator of a paper’s cognitive demand is that there may be cases in which the paper was of the appropriate standard, but the learners (for whatever reason) were below the standard of previous years. Adjusting the marks upwards in such cases would therefore mask systemic problems that need to be addressed
I believe that there is an obligation on Umalusi to publish evidence that the exam papers for all 28 subjects adjusted upwards were more cognitively demanding than previous years. The public has a right to know why the marks were adjusted in each case.
2. THE IMPACT OF PROGRESSED LEARNERS
109 400 progressed learners (13.4% of the total enrolment) wrote the National Senior Certificate (NSC) examination in 2016, up from 66 088 in 2015.
It follows, therefore, that that there was a significant increase in the number of weaker students who wrote the NSC this year. This raises the question of whether the inclusion of progressed learners in the standardisation process leads to certain anomalies.
If the raw marks are used as an indicator of an exam paper’s cognitive demand (as described above), then it is likely that the inclusion of progressed learners in the standardisation process will skew the data. This is because, if the raw marks are lower than previous years (when there were no progressed learners), then these low marks could be as a result of the inclusion of weaker students (i.e. progressed learners) and not because the papers were more difficult.
As set out earlier, the only legitimate justification for adjusting the raw mark is if the paper was more or less cognitively demanding than previous years.
Could it be that the inclusion of progressed learners in the standardisation process creates additional impetus to adjust the marks upwards, for reasons not related to the cognitive demand of the papers? And, if so, would this not mean that the marks of non-progressed learners will end up higher than previous years when there were no progressed learners?
3. THE RETENTION OF RAW MARKS WHEN THEY ARE HIGH
A closer look at the 26 subjects that retained their raw score reveals another anomaly that could potentially threaten the integrity of the standardisation process.
I mentioned earlier how, at the standardisation meeting, when the raw mark for a subject was low, DBE and/or Umalusi went back to the exam paper and identified tough questions in the paper to justify an upward adjustment.
Curiously, I did not observe the same methodology being employed when the raw mark was better than the historical mean.
There was little interrogation of why the raw mark was better than last year’s, and whether this could have been because the paper was ‘too easy’.
In some cases, instead of adjusting the marks downwards, the good raw mark was accepted as a welcome sign that the system is improving. Indeed, this logic was evident in your press statement released yesterday:
“It is pleasing to see that the marks of subjects such as Physical Science and History could be left as unadjusted…”
A glance at the data, however, shows that there may have been good reason to adjust these marks downwards to bring them into line with the historical mean. At the very least, there needs to be an explanation (based on an assessment of the cognitive demands of the papers) as to why such a downward adjustment was not made.
Choosing the (higher) raw mark over the (lower) computer-adjusted mark occurred in the following eight cases:
