Reliability evaluation of the adapted National Coordinating Council Medication Error Reporting and Prevention (NCC MERP) index

Purpose: Adapted National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) Index criteria were used in a study undertaken to evaluate commercial computerized provider order entry (CPOE) system impact on community hospital medication errors. This axticle describes: (1) adaptation of the Index, (2) classification criteria and processes used to assess the adapted Index, and (3) inter-rater reliability results. Methods: A random sample of 130 (17%) of 2251 medication safety events (MSEs) were classified based on event type, that is, adverse drug event (ADE) or potential ADE (PADE); preventability, that is, 'yes' or 'no,' and outcome severity. Event outcome severity was categorized using adapted Index categories E-I (ADEs) and B-D (PADEs). Decision rules were used for rule-based classification, while an MSE Case Review Panel used judgment-based classification when decision rules did not apply. Inter-rater reliability for both classification approaches was assessed with kappa coefficients, percentage agreement, and confidence intervals (CI). Results: Level of agreement was substantial for both rule-based and judgment-based MSE classification for event type (6 = 0.70-0.90), preventability (6 = 0.67-0.82), and decision rule application (6 = 0.79). Rule-based agreement for ADE and PADE severity was almost perfect for discrete (6 = 0.83-0.84) and combined (6 = 0.87-0.90) Index categories. Judgment-based agreement was substantial for discrete (6 = 0.63-0.67) and combined (6 = 0.66-0.84) Index categories. Conclusions: The adapted Index yielded substantial agreement for event type, preventability, and severity. Adaptation of the Index to support classification of non-preventable ADEs was an important improvement.