Article Text

PDF
OC-021 Changes in scoring of direct observation of procedural skills (dops) forms in endoscopy training and their impact on competence assessment
  1. K Siau1,
  2. P Dunckley1,
  3. J Anderson1,
  4. I Beales1,
  5. R Broughton1,
  6. M Feeney1,
  7. N Hawkes1,
  8. B McKaig1,
  9. S Thomas-Gibson1,
  10. R Valori1,
  11. C Wells1,
  12. G Johnson1,
  13. A Haycock2
  1. 1JAG Working Group for the Quality Assurance of Training, JAG, London, UK
  2. 2JAG Working Group for the Qaulity Assurance of Training, London

Abstract

Introduction DOPS are validated tools for assessing competence in endoscopy. Previously, DOPS were scored on a 4-point competence-based scale, with scores of 3 and 4 signifying competence. In July 2016, the DOPS rating scale changed to a supervision-based scale that has been shown to be more reliable, [1] with 4 ratings from maximal supervision, up to competent without supervision. We aimed to assess whether changes to the rating scale have affected distribution of scores and hence demonstrate validity.

Method Scores were collected from DOPS for gastroscopy (1934), sigmoidoscopy (517), colonoscopy (2296) and polypectomy (370) in the 6 months before July 2016 (old DOPS) and after (new DOPS), for trainees at early stages of training (total procedures<100). To allow analysis, the new DOPS rating scales was aligned to a 4-point scale, hence a score of 4 on new DOPS = Scores 3 or 4 on old DOPS, and scores on the new and old DOPS compared using the Mann-Whitney U-test.

Results 5117 DOPS (77.7% new and 22.3% old) were included for analysis. Overall, there were variations in distributions of scores (p<0.001) between forms (Graph). Compared to new DOPS, scores of 1 were underutilised on old DOPS (0.6% vs 3.0%, p<0.001). Frequencies of low scores (pooled scores of 1 and 2) were similar for gastroscopy (p=0.53) and sigmoidoscopy (p=0.34), but not for colonoscopy (new 11.9% vs. old 13.9%, p<0.001) and polypectomy (new 6.8% vs.19.9%, p<0.001). Trainees on old DOPS were more likely to be rated as competent (score 3 or 4) compared to new DOPS (86.4% vs. 55.8%, p<0.001). On subgroup analysis, this was evident for gastroscopy (86.3% vs. 49.1%, p<0.001), colonoscopy (86.1% vs. 58.2%, p<0.001), sigmoidoscopy (90.6% vs. 62.0%, p<0.001), but not polypectomy (80.1% vs. 67.9%, p=0.12).

Conclusion Endoscopy assessors are applying a greater range of scores using a new DOPS rating scale based on degree of supervision, in two cohorts of trainees matched for experience. This indicates better construct validity with the new rating scale. Further work is underway to determine the reliability of the new DOPS to inform summative assessment and certification for UK endoscopy trainees.

Reference

  1. . Crossley Jet al. Med Educ2011Jun;45(6):560–9.

Disclosure of Interest None Declared

  • Assessment
  • certification
  • DOPS
  • endoscopy training

Statistics from Altmetric.com

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.