Estimating sample sizes for binary, ordered categorical, and continuous outcomes in two group comparisons

Parameter definition

Of all the parameters that have to be specified before the sample size can be determined the most critical is the effect size. Reducing the effect size by half will quadruple the required sample size. The effect size can be interpreted as a “clinically important difference,” but this is often difficult to quantify. A valuable attempt at classification was made by Burnand et al, who reviewed three major medical journals and looked for words such as “impressive difference,” “important difference,” “dramatic increase” and then calculated a standardised effect size.8 This provided a guide to the size of effect regarded as important by other authors. There are several ways of eliciting useful sample sizes: a Bayesian perspective has been given recently,9 along with an economic approach,10 and one based on patients' rather than clinicians' perceptions of benefit.11

In statistical significance tests one sets up a null hypothesis and, given the observed difference of interest, calculates the probability of observing the difference (or a more extreme one) under the null hypothesis. This yields the P value. If the P value is less than some prespecified level then we reject the null hypothesis. This level is known as the significance level (alpha). If we reject the null hypothesis when it is true we make a type I error, and we set (alpha), the significance level, to control the probability of doing this. If the null hypothesis is in fact false but we fail to reject it, we make a type II error, and the probability of a type II error is denoted as ß. The probability of rejecting the null hypothesis when it is false is termed the power and is defined as 1-ß.

Unequal numbers in each group

For a given total sample size the maximum power is achieved by having equal numbers of subjects in the two groups. Often, however, in observational studies an equal number is not expected in each group since the incidence of a particular factor may be higher in one group than in another. In clinical trials, also, the numbers of subjects taking one treatment may have to be limited, so to achieve the necessary power one has to allocate more patients to the other treatment. In this case the sample sizes should be adjusted by a factor dependent on the allocation ratio,12 given as equation 1 in the Appendix.

If one were to maintain the same sample size as calculated for a 1:1 ratio but then allocated in the ratio 2:1 the loss in power would be quite small (around 5%). However, if the allocation ratio is allowed to exceed 2:1 with the same total sample size the power falls very quickly (a loss of around 25% in power for a ratio of 5:1) and consequently a considerably larger total sample size is required to maintain a fixed power with an imbalanced study than with a balanced one.

Continuous data

In a two group comparative study where the outcome measure is a continuous variable which is plausibly normally distributed, such as blood pressure, a two sample t test would be the statistical test used in the final analysis.

To calculate a sample size, in addition to the parameters discussed above, an estimate of the population standard deviation (sigma) must be given. The sample size formula7 is given as equation 2 in the Appendix, and table I gives the sample size required for different values of the standardised difference d, defined as d=(delta)/(sigma), at various levels of power at the two sided 5% significance level.

Alternatively, Lehr gives a quick formula for calculating these sample sizes.13 For a two sided significance level of 5% and power of 80%, the number required in each group is given approximately by m=16/d². This formula overestimates the sample sizes a little for small values of d; otherwise it gives close approximations to the sample size.

WORKED EXAMPLE

In a recent paper, Godfrey et al14 found that 46 people who had no whorls on their fingers had a mean systolic blood pressure of 136 mm Hg compared with 93 patients with at least one whorl for whom the mean blood pressure was 144 mm Hg.

Suppose an experimenter wished to confirm these findings but suspected that the mean difference would be less than that observed, with 5 mm Hg being the clinically minimum difference accepted. The overall standard deviation of blood pressure in each group is assumed to be 17 mm Hg, the same as that published. We find d=5/17=0.294, which is about 0.3, and so from table I the sample size required to detect this difference with a two sided significance level of 5% and with 80% power would be 176 subjects in each group and so 352 subjects in total. Alternatively, from Lehr's quick formula we get m=16/0.2942=185 patients per group. Suppose, like Godfrey et al, we would expect to recruit two people with whorls for every one person with no whorls. With r=2 from equation 1 we find that m'=3x176/4=132 and so rm'=264, giving a modified total sample size of 396. The overall sample size is larger if the groups were unequal because the design has less power than a design of the same size with equal numbers in the two groups.

Binary data

A binary outcome is a response which has just two categories. These categories may be of the form yes/no or presence/absence in relation to a given factor, for example alive/dead. Often an experimenter may wish to compare treatments by testing whether the difference in proportions responding on each treatment could be due to chance. In this case the effect size can be formulated as (delta)=pA-pB, where pA and pB are the proportions expected in the two treatment groups. The statistical test used to test for the association between two binary variables is the Pearson χ² test.

To calculate the number of patients required in each arm of a binary trial use equation 3 in the Appendix. For proportions greater than about 0.1 this simplifies to equation 4. Table II gives the sample sizes required for various values of pA and pB for two sided significance level (alpha) and power 1-ß. Note, however, that for pA in the table only values up to 0.5 are given. This is because having a success rate of 65%, say, is identical to a failure rate of 35% and so the sample sizes for comparing pA to pB are the same as those for comparing 1-pA and 1-pB.

An approximate result similar to Lehr's formula13 for 80% power and two sided 5% significance level is that m=16p(1-p)/(pA-pB)2, where p=(pA + pB)/2. Like Lehr's equation given earlier, this overestimates the sample size a little.

Observational surveys such as case control studies are often summarised by an odds ratio or relative risk, rather than a difference in proportions. If pA is the proportion of cases exposed to a risk factor and pB is the proportion of controls exposed to the same risk factor, then the odds ratio of being a case given the risk factor is odds ratio=pA(1-pB)/{pB(1-pA)}. An approximate sample size formula using the odds ratio (OR) is given by equation 5 in the Appendix.

WORKED EXAMPLE

Tovey and Bonell stated that 52 (19%) out of 281 men found condoms too tight.15 Of these 68% had experienced their condom splitting compared with only 26% of men whose condoms were not tight. Suppose from anecdotal evidence a researcher suspected that the prevalence of reported splitting was nearer 50% in the group finding condoms too tight and wished to conduct a study to show this prevalence still to be significantly higher than in the other group.

The expectation is that the observed ratio of the frequencies of “not tight” (A) to “tight” (B) would be 4:1. Here pA=0.5, pB=0.25 and r=4. From table II the sample size required with equal allocation in each group would be 58, and using equation 2 one derives a modified sample size of just 37 subjects in the group who found condoms too tight and 148 in the other group, giving a total of 185. In the unlikely event of equal group sizes a total of 116 subjects would be required, yielding a saving of 69 subjects. Again, this arises because the equal groups case is more efficient. Note that Lehr's formula for equal sized groups gives approximately 60 per group or a total of 120 subjects required. If we specified the effect size as an odds ratio, then the postulated odds of splitting when the condom is too tight are three times that when it is not. From equation 5, we find in this case that for equal allocation we require 55 subjects per group.

Ordered categorical data

A study may be undertaken where the outcome measure of interest is an ordered scale, such as a Likert scale (strongly disagree, disagree, agree and strongly agree) or a rating scale (better, same, worse). The statistical test used in this instance is the Mann-Whitney U test, with allowance for ties.16 The calculation of sample sizes when the data are ordered is not immediately straightforward. The problem becomes considerably easier, however, if one considers a number of pragmatic steps which will be described later in this section.

As before, we need to specify an effect size, and here it turns out to be easier to use the odds ratio. We must also specify the proportion of subjects expected in each category of the scale for one of the groups. Suppose we have t categories, with the higher ordered categories indicating worse prognosis, and the proportions expected in group A are pA1, pA2,___pAt (where pA1 + pA2+___+pAt=1) with similar notation for group B. Let cA1, cA2,___cAt, be the cumulative probabilities, so cA1=pA1, cA2=pA1 + pA2, etc. The odds ratio is the chance of a subject being in a given category or lower in one group compared with the other. For category 1 it is given by OR1={cA1/(1-cA1)}/{cB1/(1-cB1)} and similarly OR2 for category 2, up to category t-1. As will be shown later, the odds ratio may not necessarily be too difficult to estimate, as the proportions expected for one group may already be known through a pilot study or from previous research. The experimenter may postulate that on the new treatment a patient is only half as likely to have a score above a given level than on the old treatment and so the odds ratio would be estimated as 0.5. Alternatively, an experimenter may know the expected proportions in each category for one group and speculate that, if a proportion, p, were in a particular category or better, then a clinically significant difference would be for the corresponding proportion to be about 20% higher in the other group. From this information an odds ratio can be calculated and hence the other expected proportions and the sample size.

Equation 6 in the Appendix gives the formula for sample size calculations for ordered categorical data. It assumes that the odds ratio is constant for each pair of adjacent categories, that is OR1=OR2=___ORt-1, and this assumption means that the Mann-Whitney U test is the best test to use. It also means that one can estimate the odds ratio from any cumulative proportion from each group. To aid the calculations table III gives values of the numerator from equation 6 for different values of odds ratio and power.

If the number of categories is large it is difficult to postulate the proportion of people who would fall in a given category. However, Whitehead has shown that there is little increase in power (and hence saving in number of subjects recruited) to be gained by increasing the number of groups beyond five.17

WORKED EXAMPLE

In a randomised controlled trial of paracetamol for the treatment of feverish children, Kinmonth et al categorised playfulness as normal or slightly, moderately, or very listless.18 The results for the 43 replies are given in table V, together with the proportions and the cumulative proportions. The first odds ratio in the table is calculated from {0.14/(1-0.14)/(0.27/(1-0.27))} =0.44, and in a similar way we get 0.287 and 0.1625 for the other two pairs. The average is about 0.3.

Suppose a new study was planned in which we wished to replicate these results. The distribution of children in the control group (group A) was expected to be about the same as was found previously and shall be used in the calculation of the sample sizes. If an odds ratio of 0.33 in favour of paracetamol (or equivalently an odds ratio of about 3 against the control) was expected, then from the definition of the odds ratio we can calculate the expected cumulative proportions in the treatment group (group B) from the formula CBi=CAi/(Cai + OR (1-Cai)). Thus the proportion expected in the first category of group B is 0.14/(0.14+0.33 (1-0.14))=0.33 and so on. The cumulative proportions expected in group B are 0.33, 0.65, 0.83, and 1.00, and so the actual proportions expected are 0.33, 0.32=(0.65-0.33), 0.18=(0.83-0.65), and 0.17=(1.00-0.83). The average proportions p are given by 0.235, 0.280, 0.210, and 0.275. Thus (1-(Sigma)p³)=0.935. For 80% power and 5% significance level, from table III, the numerator is 39.02, and so the sample size is 39.02/0.935=41.7, or about 42 patients per group.

The formula is quite complicated and we have a number of suggestions to simplify matters. If the mean proportions (pi's) in each category are roughly equal then the denominator in equation 6 is constant for a given number of categories, and if the number of categories exceeds five it is approximately unity. Thus for 80% power and a two sided significance of 5%, an estimate of the sample size can be obtained from m=47/(log OR).2 If the number of categories is less than or equal to five then multiply this sample size estimate by a correction factor given in table IV. From this table, in the situation of approximately equal proportions, it is evident that having only two categories in your data for analysis may require you to recruit a third more patients than if the data were kept continuous. For our example, the correction factor from table IV is 1.067 and so n=1.067x47/(log 0.33)2=40.8, or 41 patients.

Another simplification occurs if the proportion of subjects in one category for both groups is expected to be large. We can combine categories until there are only two left and use the formula and table given previously for binary data. Combining categories reduces the amount of information available, so one would expect the required sample size to increase.

In the worked example if we had pooled those scoring 1-2 and those scoring 3-4, we would compare proportion pA=0.38 to pB=0.65. Formula 4 shows that this study would require 49.9, or about 50 patients per group. Thus, use of all four categories, rather than simply two, yields a reduction of 16% in the study size, and this might outweigh the benefit of an easier sample size calculation.

Comment

From the equations in the Appendix it is clear that the sample size, significance level, power, and effect size are all interlinked. Given any three parameters, in principle the equations can be solved for the fourth. Thus, if the sample size were limited by resources, and the significance level fixed in advance, one could arbitrarily increase the power of the study by postulating larger effect sizes. In practice, however, the estimate of the effect of an intervention often proves too optimistic, resulting in many trials which are too small. The need for sample size calculations provides an excellent opportunity to involve a statistician early in the planning of a study and not just when the analysis is required. This paper has covered only a limited range of designs, and a statistician could advise on other designs. These include comparison of more than two groups,19 comparison of survival curves,7 20 21 and studies to demonstrate bioequivalence.22 Computer software is available for some of the sample size calculations discussed here,23 24 25 26 and other reviews have been given.27 28