Examination Assessment Metrics

Note: alpha co-efficients and SEms are calculated including all candidates’ marks

Definitions

Alpha Co-efficient:

Alpha is often described as a measure of test ‘reliability’, but more properly it indicates score ‘reproducibility’: the higher the alpha co-efficient, the more likely a candidate taking this test would perform similarly relative to other candidates in another similar test. When high in an MCQ (say, over 0.85) or an OSCE (say, 0.70 or higher), it suggests that a defensible assessment of candidates’ overall knowledge or skill in the specialty is being made by testing their knowledge or skills in this sample of items or cases, assuming that the exam content is blueprinted appropriately to the curriculum.

SEm:

The SEm is an estimate of the precision of any candidate’s score, the larger it is the less precise the scores. It is often calculated from alpha and the standard deviation of all candidates’ scores on the test. [ SEm% = √ (1 – alpha) x SD% ]. Like many other statistics, controversies surround its meaning and desired value, but in medical exams, approximate maximum values commonly expected are 5% for an OSCE and 3% for an MCQ test.