Article · 5-minute read
By Sarah Chan – 30th July 2024
I had the pleasure of attending the 2024 International Test Commission (ITC) Conference held in Granada, Spain with a number of colleagues in the R&D team.
Among the sessions I attended, one of the keynote presentations was of particular interest to me, centering around the proctoring system developed for the Duolingo English Test (DET). Dr. Alina von Davier, Chief of Assessment at Duolingo, spoke about ‘Embracing the Future of Remote Proctoring: Ensuring Test Integrity and Accessibility in the Digital Age’.
Remote proctoring allows for the supervision of tests which candidates take remotely, e.g. in their own home or workplace. With the increased demands from working from home and the need for rigorous assessment as technology advances, this was a particularly timely presentation in dealing with a topic many of our clients are interested in.
In the past, many assessments were developed for supervised testing, i.e. in a testing room. The arrival of the internet drove assessment providers to host many of these tests and questionnaires on the internet to be administered remotely (unsupervised). These ‘digital-second’ assessments were not designed for, nor did they take advantage of, the internet’s capabilities.
In contrary, ‘digital-first’ assessments, such as Wave questionnaires and Swift aptitude tests, were designed specifically to take such advantages. For example, interactive features are embedded to enable sophisticated scoring and a better candidate experience.
In remote testing, there is some risk that content could be captured and shared, which has the potential to undermine the efficacy of the assessment. Another potential downside is the uncertainty over the test-taker’s identity: is a clever friend or colleague doing the test for the candidate? This problem can be solved by follow-up supervised testing at the final assessment stage (e.g., interview, assessment center).
‘Digital-first’ assessments can also be integrated with remote proctoring systems to overcome concerns about security and test-taker identity.
Everyday online meeting software (e.g. MS Teams, Zoom, Google Meet) can be used to supervise candidates remotely, where an administrator ‘observes’ the candidate completing a test in the online meeting. However, this approach is only really suitable for one-to-one supervision. Some candidates may also find this form of real-time proctoring intrusive, although in practice it may not be that different from in-person supervision in a traditional exam setting!
With the advancement in technology, many remote proctoring providers now incorporate artificial intelligence (AI) in their solutions. In asynchronized remote proctoring, software is used to record candidates when they complete online assessments in their own time. This proctoring of tests includes recording the candidate’s face as well as their computer screen. AI was trained to analyze the recordings after the test session to detect suspicious behaviors and then judge whether a candidate is likely to have completed the assessment appropriately or if they might have had unfair assistance.
An important point raised at the beginning of the presentation was around the fact that “proctoring is an assessment activity that contributes to test validity”. As the proctoring process is there to determine whether a candidate has completed an assessment appropriately, if a candidate is found to display ‘cheating behaviors’, the test result is unlikely to give an accurate reflection of their ability. Clearly, indicating that a candidate has cheated based on a decision made by AI can carry significant consequences for the individual and organization, particularly in high-stakes selection scenarios.
A key message in the presentation which also aligns with our perspective is that there should be human involvement in AI-based proctoring processes, and both humans and AI need to be trained to achieve the best outcome.
The main thing I took away from the presentation is that the best approach to remote proctoring is to combine the strength of human intelligence and AI. AI can help make the proctoring review process more efficient and keep the human proctor on track, but humans should make the final decision. There needs to be a clear process as to how proctoring decisions are made and how the decisions are reviewed to ensure consistency in the approach for all candidates.
The ultimate goal for proctoring is to be ‘responsible’ – a process that is reliable and valid, secure, minimizes bias, and improves inclusivity and transparency for all stakeholders.
Of all the different Saville assessments, proctoring is most likely to be used with aptitude tests, rather than Wave or Situational Judgment Tests.
Our Swift online aptitude tests were designed with security in mind. They are powered by large item banks to help safeguard content and reduce cheating.
In terms of remote proctoring with Saville’s aptitude tests, we advise that:
Sarah is a Senior Research Psychologist at Saville Assessment, managing our portfolio of aptitude tests.
You can connect with Sarah on LinkedIn here.
© 2024 Saville Assessment. All rights reserved.
