E-assessment - Film studies

E-assessment

Seeking an online standard

By marking exams online, one board hopes to make assessment open to more teachers and align marking standards.

But it needs to tackle markers' disappointment at the loss of meetings Words Julie Nightingale Recruitment and retention of examiners is an increasingly thorny issue for awarding bodies. One significant sticking point is the difficulty teachers have in taking time out of school to travel to attend the pre-marking meetings at which exam standards for papers are explained by senior examiners. Once set, the meeting dates are immovable, as is the location. But the Assessment and Qualifications Alliance (AQA) exam board believes the solution lies in taking the process online. This enables examiners to do the work from home (or elsewhere, provided the location doesn't compromise security) and can be completed over a longer period, meaning it can be fitted around other commitments. Emma Dawkins, e-standardisation implementation manager for AQA, explains: "Offering training on one day in a defined location presents some problems for a large proportion of our examiners. With online training, we are hoping to give more people the opportunity to widen the pool of potential examiners and perhaps reach new parts of the education community."

How it works

Training is via a secure website. The first thing examiners see is a welcome page from the senior examiner, where the key messages about marking a particular paper are emphasised and important dates are highlighted. Next is a series of scripts, selected by the senior examiner, that illustrate key aspects of the marking process, with the marking scheme displayed alongside. The examiner tackles up to five training scripts but can break off and return at any time within the marking window of two to three days. The examiner allocates a mark to a question in a script and can also make comments to the team leader using an annotation tool. Marks can be amended at any point before the question is submitted. Once the marks have been sent to the team leader, the examiner receives instant feedback on how their marking compares with the senior examiner's. If, for example, the examiner's mark is wildly at odds with the senior examiner's, the senior examiner can call the person to discuss it. "You are building up understanding just as you would in a meeting, and you expect people to become more accurate as they review the instant feedback and progress," says Dawkins. Pilots have been completed for examiners in GCSE health and social care and IT, and A-levels in computing and PE. This summer brings the first phase of its application in a wider range of subjects, including GCSE maths and ICT, and A-levels in French, German and physics. Those involved in the pilots were drawn from AQA's existing examiners and the response has been positive, says Dawkins. Most of the critical feedback has related to the technology skills required to work online. "But it's a user-friendly interface and very much refiects the [marking] method that they have always followed," she says.

Missing meetings

When Edexcel experimented with online training, it found some examiners mourned the loss of the face-to-face contact with other professionals, as well as the chance of a day away from school and the networking opportunity it offers. AQA expects similar complaints when online standardisation is rolled out more widely, says Dawkins. "But we have to balance that against what is in the best interests of the whole training system and what will secure the best, most accurate marking in the long term." In fact, there are mechanisms built into the system that enable colleagues to simulate face-to-face dialogue, she adds. "For example, there is an instant messaging facility so the examiner can

"Messaging provides real-time feedback, very much like aface-to-face meeting"

submit questions to the senior team as he or she is working through the training material and the senior colleague can message back to everyone, not just the individual, so the answer is shared. It's real- time feedback, which is very much what is going on in a face-to-face meeting." There is a further advantage to "virtual" training in terms of confidence-building, adds Dawkins. "Face-to-face meetings can be intimidating for new and less experienced examiners and they may not ask the questions that they want to. Working online removes some of that threat and intimidation."

Content management

Concerns that the system would need most tweaking from the examiner end have not, in fact, come true. Instead, the changes have been concentrated on the content management side - preparing and entering marking schemes, scripts and other material online. "At the moment, getting the content into the system is not as smooth a process as we would like," Dawkins says. On marking schemes, for example, senior examiners have to produce a final version for entry in the system in advance but revisions may still need to be made in the light of examiners' own views. Dawkins says: "If a group of examiners came up with markedly different outcomes from those set by the senior examiner, the system can take account of that and release an updated mark scheme automatically to all examiners. We are trying to reduce the need for that, in terms of improving preparation, but we recognise that getting the scheme watertight could be difficult." Mike Cresswell, AQA director-general, emphasises that e-standardisation will allow more teachers to be examiners. "As with technologies such as e-marking and on-screen assessment, we are introducing e-standardisation on the basis of careful research and development that ensures that we always put the quality of the assessment process, and therefore the candidates, first." The system will be implemented in the UK over the next two or three years. AQA is aiming for 80 per cent of eligible question papers to be standardised online by summer 2010

Software on trial

Mike, a chief examiner for sport and PE, took part in the e-standardisation pilot for A2 sport and PE in summer 2007. He worked with other senior examiners to test the software involved and make any
adjustments well before scripts went out to examiners. On the plus side, the process was less time-consuming than paper submission and the software itself is easy to use, he says.
"The front page was Windows-based with drop-down menus and quickly became intuitive once we were able to 'play' with the system."

Examiners were able to proceed at their own pace and could be in touch with their team leaders by telephone or e-mail to discuss any queries. "Each leader has access to an online summary page showing each of their examiners' progress," he explains.
One negative aspect is the loss of the face-to-face meeting. He says: "I have always said that there are three benefits from examining: financial, technical - finding out information about what's required and passing that on to students - and finally there's the social side, where you meet people of a similar ilk and have an exchange of ideas, even networking. The latter one has gone, and the stimulus of meeting others has also gone."

But, overall, online standardisation is a welcome innovation, he adds. "It's efficient, accurate and well worth making the effort."