Introduction Endoscopic Non-Technical Skills (ENTS) are essential to high quality endoscopy. ENTS skills are not routinely assessed in independent practitioners. Endoscopy assistants work as members of the team and observe the performance of endoscopists at close hand. A reliable, validated 360-degree assessment tool based on assistant ratings for ENTS may provide performance enhancing feedback to independent practitioners. We aimed to develop a multi-assessor rating scale (MARS) to provide a global assessment of ENTS skills for endoscopists based.

Method The MARS tool was developed with the four main domains – Communication (COMM), Situational Awareness (SITA), Leadership (LEAD) and Judgement and Decision-making (J and DM). All domains were based on previously validated questionnaire formats tested for factor and item analysis and construct validity with a standardised 7-point scoring scale and 10 items per domain. Experienced endoscopy nurses working regularly with the endoscopists provided a score reflecting their overall performance for each domain question. The internal reliability of the MARS items was assessed using the standardised Cronbach’s alpha statistic. Inter-rater reliability was assessed by calculation of Kappa alpha. Consent was obtained from all endoscopists prior to administering the MARS tool.

Results 9 endoscopists agreed to undergo an assessment of ENTS using the MARS tool. Each was assessed by 10 nurse raters (90 assessments, based on 40 item scores). The mean scores for each item within the 4 domains were calculated for each endoscopist. Cronbach’s alpha co-efficient showed a high level of reliability in each of the 4 questionnaire domains – COMM=0.98, SITA=0.86, Lead=0.97, J&DM=0.98. A performance threshold of 90% each domain was set as a nominal standard. Kappa statistics measuring inter-rater reliability in distinguishing between scores meeting or below this performance threshold for all rater combinations showed moderate to excellent agreement (range 0.54 to 0.8). The MARS tool was practical to administer, all ratings were obtained during a single afternoon. Results were expressed graphically as percentage acceptable ratings in each ENTS domain. Where 20% or more of specific questionnaire item scores were scored as poor or average each was highlighted in the more detailed feedback report given to the endoscopist.

Conclusion The MARS tool is a practical way to measure of ENTS performance designed as a 360 degree feedback tool using 10 raters for administration every 6 months. The internal consistency of the questionnaire is good with acceptable inter-rater reliability. The feedback graphic enables rapid recognition of areas for improvement in ENTS domains.

Disclosure of Interest None Declared