A mixed methods evaluation of the effect of confidence-based versus single best answer multiple-choice questions on student performance and the learning journey

Abstract Background Conventional single best answer tests (SBATs) are a proven and ubiquitous assessment tool. By their very nature, SBATs prompt students to guess a correct outcome when unsure of the answer, which may lead to a reduced ability to reliably assay student knowledge. Confidence-based answer tests (CBATs) are designed to improve reliability because participants are not forced to guess where they cannot choose between two or more alternative answers which they may favour equally. Educators aim to develop assessments for learning rather than assessment of learning; although both SBATs and CBATS have proven benefits, as CBATs provide additional information about student confidence, they may provide additional insights to educators and students alike. However, their use may be limited if they significantly alter student scores or grades compared to SBATs. The aim of this study was to compare performance across these test paradigms, to investigate if there are any systematic biases present and to analyse student perceptions qualitatively. Methods Thirty-four first-year optometry students undertook a multiple-choice test comprising 50 questions. Each question was completed using both CBAT and SBAT approaches. Differences between CBATs and SBATs were analysed using the Bland and Altman method of agreement and linear regressions. Upon test completion, students undertook a survey comprising both Likert scale and free responses regarding their experience and perspectives on the CBAT and SBAT multiple-choice test paradigms. These were analysed thematically. Results There was no significant difference in average test score between paradigms, with a median difference of 1.25% ( p = 0.31, Kruskal-Wallis). The survey indicated that students had no strong preference towards a particular method and qualitative analysis identified themes including confidence, risk-taking behaviours, test-taking motivation and learning journey. Conclusions There was no significant effect of test paradigms on scores and thematic analysis indicated CBAT provided opportunities for reflective practice. As such, CBATs present as a viable equivalent and alternative option to conventional SBATs, with the added potential benefit of providing valuable insight into student confidence that may not otherwise be visible. There is no inherent bias within one method over another.