Statistical inference, power calculation and software of κ indices

The κ indices can also be used for statistical inference (significance testing, SE estimation and CIs calculation)31 and sample size formulae exist for such purposes.31–33 STATA, for example, provides an extensive list of statistical packages for such purposes, including calculation of CIs (kapci), sample size/power calculation (sskdlg, sskapp) for incomplete designs (such as missing data), and multiple raters –here STEP 3-(κ2). On power estimation the researcher shall consider, as any other variables in medicine, that dichotomisation of a continuous variable may carry an important loss of power, thus requiring cautionary decision-making.34

Interpretation and caveats of κ indices

Across the literature, three major interpretations in which a value of 0.6 or beyond is considered acceptable or fair have been consistently used (table 3).28 It is also possible to obtain negative κ indices when one of the raters reverts accidentally the scale of rating.25 ,33 While they have been widely used, κ indices are far from perfect and face several criticisms. First, the criteria to evaluate κ (table 3) are arbitrary. Second, κ is influenced by the marginal distribution of the ratings (‘the bottom rows and the end columns’ of the cross-tabulation) and the prevalence of the disease. In this sense, several reports have been shown to correct for bias (marginal distribution inequities) and prevalence and, whereas it may be reported, some authors have considered the adjusted κs inaccurate.35 Finally, κ, as a summary statistic, can lose some information when examining the discrepancy between observers, and this information is even more accentuated when the researcher converts a natural continuous variable into a binary one.34 For example, it would be important to see in a hypothetical grading of no cancer, preneoplastic and cancer type of pathology exam what the discordance was between a ‘no cancer’ diagnosis and ‘preneoplastic.’ Consequently, it is important to provide raw data for readers to review as an online supplementary table in order to show the rates of agreement and the reasons for discrepancies.

Weighted κs and matrix of weighted κs

A weighted κ is an extension of the regular κ in which the researcher provides arbitrary weights to penalise the degree of disagreement between observers for multiple categories. Several weighting systems can be used including linear, quadratic and Cicchetti-Allison weights33 (STATA command kapwgt).

The matrix of weighted κ is another alternative method to evaluate agreement in ordinal variables and takes into account κ indices and correlation coefficients.36 For further review, the reader is referred to Roberts.31 ,37 This approach, however, has been rarely used in medicine.

ICC for ordinal variables: perhaps better than weighted κ

Kraemer,38 Streiner and Norman28 favour κ for two-by-two agreement assessment, but recommend using the ICC coefficient when treating anything other than dichotomous variables with more than two observers. This is so, because ICC, as noted per Streiner and Norman,28 provides the following: “(a) the ability to isolate factors affecting reliability; (b) flexibility, in that it is very simple to analyse reliability studies with more than two observers and more than two response options (STEP 3); (c) we can include or exclude systematic differences between raters (bias) as part of the error term; (d) it can handle missing data; (e) it is able to simulate the effect on reliability of increasing or decreasing the number of raters; (f) it is far easier to let the computer do the calculations; and (g) it provides a unifying framework that ties together different ways of measuring inter-rater agreement—all of the designs are just variations on the theme of the ICC”.28 A recent paper published in Gut illustrates how to apply the ICC methodology to examine agreement between ordinal measurements and can be followed as an example.6

Modelling patterns of reliability: log-linear models, latent class models and beyond

Advanced techniques are possible for researchers who are interested in providing more information than a summary statistic, for instance, if an investigator wanted to explore patterns of reliability by different characteristics or compare reliability adjusting for different covariates, there is no gold standard.39 Instead, a researcher may apply latent class models, in which reliability (or agreement as used in the references) is considered an unobservable categorical truth (or latent class) and relate this latent class to the observed data.40–43 A good tutorial can be found in the website of Dr Uebersax44 and, for a more practical application, in the papers by Christensen et al45 and Dunn16 which may be the few studies in gastroenterology using this technique. Other approaches are log-linear models, which examine the association of cell counts on the level of categories (eg, observers) and models’ interaction patterns and associations.26 ,46 ,47 Similar to the latent class models, there are a handful of examples using log-linear models in GI reliability.16 ,48 Again, STATA provides commands for latent class models (gllamm) and log-linear models (ipf).

Selection of the ICC

At least six types of ICCs have been described, and derived from different methodology and, therefore, leading to different inferences.28 ,49 ,50 Thus, when authors report ‘an ICC,’ a thorough explanation shall be provided; however, the question remains which ICC needs to be used and why. Luckily, there are guidelines to assist researchers (figure 2).49 ,51 No matter which ICC is chosen, it is important to provide readers with an online supplementary table of all variance components that can be used for interpretation and correction of measurement error (eg, in the paper by de Vet et al1 in table 2, or as noted in table 5.10 in the de Vet et al52 book, chapter 5). Furthermore, it is also important to provide data to support assumptions of analyses of variance (ANOVA) as noted per McGraw and Wong.49

Statistical inference, power calculation and software

As discussed, several authors have shown different techniques of obtaining statistical inference for different ICCs.49 For sample size calculation, several formulae are available: one relies on hypothesis testing;32 ,53 other on the width of the ICC interval,54 ,55 and assurance of probability for ICC estimation being on a certain value (eg, 90% assurance that the ICC would be 0.95).56 STATA provides support for statistical inferences (icc, including some forms of figure 2 ICCs), and sample size calculation (sampicc).

Interpretation and caveats of the ICC

The value of ICC traditionally ranges from 0 (non reliable) to 1 (perfect reliability). Traditionally, ICC values less than 0.40 are considered poor, values between 0.40 and 0.59 fair, values between 0.60 and 0.74 good, and values between 0.75 and 1.00 excellent.50 ,57 ,58 However, careful observation of the above formula shows that, depending on the relationship between person/observation variation and measurement error, ICC—and reliability parameters in general—can be easily influenced by measurement error. For example, if there is higher variation between study subjects compared with measurement error, the instrument will provide high reliability, that is, the instrument will accurately distinguish between participants. On the other hand, if there is not enough variation between patients, a small measurement error can make a difference and make the instrument not reliable. As a consequence, reliability will depend on the variability of the sample from which the index was derived and is influenced by the measurement error to some extent.

Extra notes on ICCs: other extensions and paired data

Haber et al14 and Muller et al59 provide other types of coefficients (parametrical and non-parametrical) when any of the underlying ICC assumptions are not met (eg, concordance correlation coefficient (CCC), Rothery, etc…). Chen and Barnhart60 provide guidance on CCC and ICC. The CCC is similar to the ICC but that the CCC does not rely on ANOVA assumptions. This would be seen as an advantage, but the CCC is less developed than ICC-based methodology. In the case of repeated measures on the same subjects over time (STEP 4), ICC can be used to take into account such paired design by modelling interaction with time and observation as well as time and observer.

Coefficient of variation

The coefficient of variation expresses the SD as a percentage of the mean, multiplied by 100% and expressed as a percentage. It is calculated for each pair of observations, and a zero value would represent a perfect agreement. One major advantage of the coefficient of variation is that it is unitless, and can consequently be used for variable comparison and in testing models. Its only meaningful interpretation occurs when the scales compared are positive (for instance, not meaningful for negative scales such as Fahrenheit).62 Available commands are in STATA (estat cv).

Correlation coefficients: why they are not in table 1?

In the past, correlation coefficients (such as Spearman or Pearson) had been used to provide a measure of agreement; however, as Bland and Altman point out, the correlation coefficient measures the strength of the relation between two variables, not the agreement63 (table 1). Rather, correlation coefficients measure values in a straight line from −1 to +1 (perfect negative correlation and positive correlation, respectively). The correlation coefficient measures also rely on the accuracy of the assessed variable and on the study variation (eg, population dependent).64 Pearson's correlation coefficient assumes that the variable is normally distributed (in contrast to its non-parametrical counterpart, Spearman's rank correlation). Other misuses of correlation coefficients are described by Altman as forgetting repeating measures over time, restricted sampling of individuals, mixing samples of different characteristics and so on.65

Commands in STATA include the conventional correlation (correlate), variable-adjusted correlation (pcorr), CIs (corrcii), and power/sample size (power onecorrelation).

Proportion of positive agreement and specific agreement

The proportions of positive agreement and specific agreement provide raw estimation of important descriptive data, and these estimations convey the overall agreement between raters and for each particular category (positive or negative agreement). Agreement studies as those described here are used less frequently but should be considered a standard in the reporting of basic descriptive statistics. An excellent resource for agreement studies can be found on Uebersax's website66 and sampling distributions for positive agreement and negative agreement have been described by Samsa.67 It is, however, in the context of criterion validity (ie, diagnostic accuracy studies), when these proportions gain importance as they are required by the Food and Drug Administration to be part of the integral report.68

Standard error measurement and its association with ICC

A thorough summary of the relationship between measures of reliability and agreement is provided by de Vet et al.1 They define measurement error, by SE of measurement (SEM) and equate the square root of the error or , where, can include a systematic error between observers/raters () or not (). De Vet et al1 provide a more in-depth explanation of the formulae, which demonstrates that since ICC uses different types of σ²s (variances), as shown in figure 2, it is possible to derive SEM from ICC. One note of caution is to avoid using the formula . Researchers have used ICC obtained from different studies with different subject variability and, as readers already know, ICC depends on the study participant heterogeneity; thus, only when heterogeneity is similar can the formula be valid.1 ,69

Bland-Altman plot: excellent resource to summarise measurement error

In 1986, Douglas Altman and J Martin Bland proposed a different method to assess systematic error between observers. In order to calculate it, they plotted the mean difference between raters’ scores of the two observers against the difference between the scores of the observers. For instance, a recent paper by Sedwick,70 showed a Bland-Altman plot which investigated the agreement between primary care and daytime ambulatory monitoring in blood pressure. Online supplementary figure S3 shows the mean systematic difference between both readings (d, green line), and two other lines (red, dotted), which represent the ‘limits of agreement’ as estimated by d±1.96x SD of the differences. A further work by de Vet et al,71 shows an association between the limits of agreement and SEM using a 95% CI, the minimal or smallest detectable change (SCD) .

The limit of agreement represents the range within which most differences between measurements by the two methods will lie.72 In other words, the range of the CI will show an indirect magnitude of the measurement error; consequently, if a difference between two observations is noted beyond the interval, it is most likely that the differences are real as opposed to a measurement error. As noted in the online supplementary figure S3, the graph is quite powerful in illustrating the degree of agreement between two raters, allowing readers to see the relationship between extreme values and the degree of disagreement. As noted per Bland and Altman, “the decision about what is acceptable agreement is a clinical one; statistics alone cannot answer the question.” Nevertheless, given the data, the graph shows what the expected systematic and random errors are, making it possible to see the smallest detectable change. In addition, it would be possible to correct using the average difference in the measurement of several techniques.

The plot, however, has several limitations that need to be taken into account. First, the differences between measures are normally distributed (this can be checked using the usual histogram or normal plot.) Second, it is assumed that the sample subjects are representative of the population the study is aimed to extrapolate. Third, the plot can only be used for pairwise observations, so whenever more than two techniques are present, each pairwise Bland-Altman plot needs to be reported separately. Finally, it is assumed that the systematic error (d) is constant across different values. The latter assumption is often criticised.73 Nonetheless, Bland and Altman have proposed several statistical techniques to deal with such criticisms, including log-transformation of the variable74 or running regression techniques to account for this non-uniformity across values.72 STATA also provides command to estimate the Bland-Altman plot.

Repeated measures for measurement errors in continuous variables

In their paper, Bland and Altman describe several methods to account for repeated measurements on the same subjects with similar and different numbers of replications based on variance methodology. Readers are referred to that paper to learn more about implementation.72