This Article Abstract Full Text (PDF) Services Download to citation manager Citing Articles Citing Articles via HighWire PubMed PubMed Citation

Two-Stage Testing in Microarray Analysis

What Is Gained?

^a Department of Biostatistics, University of Alabama at Birmingham

David B. Allison, Department of Biostatistics, Section on Statistical Genetics, Ryals Bldg, Suite 327, 1665 University Blvd, Birmingham, AL 35294 E-mail: Dallison{at}ms.soph.uab.edu.

Decision Editor: John A. Faulkner, PhD

	Abstract

Top Abstract The Two-Stage Approach Type 1 Error (False-Positive)... Power and False Negatives Quantitative Results Conclusions References

 
Microarray technology for gene expression studies offers powerful new technology for understanding changes in gene expression as a function of other observable or manipulable variables. However, microarrays also pose a number of new challenges. One of the most prominent of these is the difficulty in establishing a procedure for declaring whether a gene's expression level is associated with the independent variable that offers reasonable and specifiable false-positive (type 1 error) and false-negative (type 2 error) rates. A recent article described a two-stage testing procedure to address these goals. However, information was not provided to indicate whether this procedure would or would not meet its objectives. Herein, we show mathematically that the two-stage procedure proposed does not provide benefits in terms of minimizing false-negatives while controlling the false-positive rate relative to standard single-stage testing. Therefore, investigators are encouraged to consider alternative analytic approaches.

THE advent of microarray technology for gene expression measurements opens many exciting opportunities and challenges in aging research ⁽¹⁾⁽²⁾. One of the major challenges involves determining whether a sample numerical difference in gene expression among two or more groups, conditions, or tissues represents a "statistically significant" difference ⁽³⁾. This is challenging in part because microarrays allow one to simultaneously test for differences in thousands of genes, thereby creating a problem of multiple inference if one stays in the frequentist ⁽⁴⁾ null hypothesis testing paradigm ⁽⁵⁾. For example, if differences in each gene expression are compared at the .05 significance level for a microarray containing 10,000 genes and the null hypothesis of no difference in gene expression were true for all genes, we would expect to observe approximately 500 false positives or genes for which a statistically significant difference is observed when there is, in truth, no difference. This issue of multiple comparisons has long been a thorn in the side of researchers. There are many accepted strategies for adjusting the significance level to ensure that the probability of making any false positive is equal to or below the desired significance level. For example, a Bonferroni correction ⁽⁶⁾ divides the desired experimentwise alpha level by the number of tests. Each individual test is then conducted at this Bonferroni adjusted alpha level. A criticism of this and similar approaches is the fact that by controlling the false-positive rate, one becomes more likely to observe false negatives, or differences that fail to reach statistical significance when an actual difference exists. An alternative approach for meeting this challenge was suggested in a recent paper ⁽⁷⁾ which "advocates a two-stage design in which significance testing applied to exploratory data is used to guide a second round of hypothesis-testing experiments conducted in a separate set of experimental studies" (p. B52). Ideally, this method would control the experimentwise alpha level or type 1 error rate (the probability of making any false positives in the study) while making fewer type 2 errors than the single-stage design. However, as previously noted ⁽⁸⁾, evidence has yet to be offered that the two-stage design procedure either controls the experimentwise type 1 error rate or reduces the risk of type 2 errors. The purpose of this brief paper is to examine the type 1 error rate and power of the two-stage design and evaluate how these statistical properties compare with those of a single-stage design.

	The Two-Stage Approach

Top Abstract The Two-Stage Approach Type 1 Error (False-Positive)... Power and False Negatives Quantitative Results Conclusions References

 
The approach by Miller and colleagues ⁽⁷⁾ involves first testing for differences between two groups for all of k genes in one set of data (stage 1) by using a stage 1 alpha level (which we will denote ₁) that is greater than the alpha level that would be required by a Bonferroni correction. No specific mention is given as to how to choose ₁, but Miller and colleagues use ₁ = .001 for an example with k = 10,000, suggesting perhaps that they mean for ₁ to be set somewhat below the more conventional .05. The k hypothesis tests are then performed, yielding some number of genes, m, with significant effects (0 m k). Then, at stage 2, a second independent set of data is gathered and only those m genes found to be significant at stage 1 are tested at level ₂ = .05/m. Any gene significant at level ₂ "can be accepted as age sensitive" (⁽⁷⁾; p. B55). Presumably, although not explicitly stated, a two-tailed hypothesis test is conducted at stage 1, whereas a one-tailed test is conducted at stage 2. The stage 2 test should test only in the direction that the apparent effect was observed at stage 1 because it would make little sense to conclude that there is an effect on the basis of two random samples producing significant results in opposite directions.

This two-stage testing procedure was offered as one way of dealing with the problem of false positives (type 1 errors) that would result from multiple significance testing without correction and false negatives (type 2 errors) that would result from the use of a Bonferroni correction. However, concrete information indicating that this approach will achieve these goals has yet to be presented. First, although Miller and colleagues do not state exactly to what overall alpha level this procedure holds the entire experiment, their use of .05 in determining ₂ suggests that perhaps they intend to achieve an overall experimentwise alpha level of .05. No information has been offered as to whether the proposed design actually controls the experimentwise alpha level. Second, no information has been offered to indicate that this procedure is more powerful than simply testing all data together in a single-stage design with a method that controls the experimentwise alpha level. These questions are further evaluated in the paragraphs that follow. It may be worth pointing out that the two-stage procedure under discussion is aimed at reducing purely stochastic threats to statistical inference and should be seen as distinct from constructive-type replications that have independent value in helping to eliminate nonstochastic threats to valid inference ⁽⁹⁾.

	Type 1 Error (False-Positive) Rate

Top Abstract The Two-Stage Approach Type 1 Error (False-Positive)... Power and False Negatives Quantitative Results Conclusions References

 
Once one has reached stage 2, the probability of making one or more type 1 errors at this stage is equal to 1 - (1 - .05/m)^m. This assumes one uses .05 as Miller and colleagues do in their example ⁽⁷⁾. In fact, one is free to choose any alpha level by simply substituting the desired level for .05 in this formula. It should be noted that this formula is correct when all null hypotheses are true and all gene expression levels are independent. In reality, both of these conditions seem extremely unlikely to be met. To the extent that they are not met, the actual value of the stage 2 type 1 error rate will lie somewhere between zero and 1 - (1 - .05/m)^m. Because of the known conservativeness of the Bonferroni correction, if the stage 2 testing were the only testing involved, this formula would hold the overall alpha rate of the entire experiment (which we will denote _ew) to a value close to (though just slightly less than) .05. However, because we are testing in two stages, a type 1 error will be made only if genes for which no difference truly exists show significant differences (false positives) at both stages. To obtain the experimentwise alpha level, first compute the alpha level at stage 2 given that m tests were significant at stage 1. Then, because the number of genes significant at stage 1 (M) is a random variable, the experimentwise alpha level is obtained by computing the weighted sum over all possible outcomes, m, with the weights representing the probability of that outcome:

_ew=P+

_ew=.

Finally, if the probability of rejecting each test equals ₁, the probability of observing m significant tests out of k independent tests can be described by the binomial distribution with parameters k and ₁. When this is taken into account, the experimentwise alpha level for the two-stage design can be written as

_ew=.

Clearly, _ew is affected by the choice of ₁. To demonstrate this dependence, consider the example used by Miller and colleagues ⁽⁷⁾ where k = 10,000, ₁ = .001, and ₂ = .05/m. Under this circumstance, _ew = .049, which is very close to the level of .05 that might be desired. However, if ₁ were switched to .0001, a value still greater than the Bonferroni corrected value (.05/10000), then the overall type 1 error rate becomes only .0314. Furthermore, when the fact is used that the stage 2 alpha level will be less than or equal to .05, a simple bound on the experimentwise alpha level for the two-stage design is as follows:

This demonstrates that the two-stage design is often conservative, leading to an experimentwise alpha level that is lower than that desired. Furthermore, as the choice of ₁ becomes smaller, the experimentwise alpha level of the two-stage design becomes more conservative.

As the formula above shows and the example illustrates, the two-stage procedure fails to consistently hold the overall alpha level at .05. On the contrary, one can more easily achieve the goal of holding _ew = .05 in a single-stage design by simply setting ₁ = 1 - (1 - .05)^1/k and only conducting a stage 1 analysis. This provides a correction that is nearly equivalent to the Bonferroni correction, but less conservative. Nevertheless, perhaps the two-stage procedure will reduce the type 2 error rate (i.e., increase power) relative to a one-stage procedure with ₁ = 1 - (1 - .05)^1/k.

	Power and False Negatives

Top Abstract The Two-Stage Approach Type 1 Error (False-Positive)... Power and False Negatives Quantitative Results Conclusions References

 
Using the between-groups t test as Miller and colleagues ⁽⁷⁾ discuss and assuming conditions for its use are valid (i.e., normality, homoscedasticity, independence of observations), one can express an effect size for a specific gene as

=,

where µ₁ is the population (not sample) mean level of gene expression for one group of subjects (group 1), µ₂ is the population mean for the second group (group 2), and is the common within-group standard deviation. Then, assuming equal numbers of subjects per group, and denoting the total number of subjects by 2n, one finds that the noncentrality parameter for the t distribution with 2n - 2 degrees of freedom (df) for testing the between-group difference is

₁=

Let t_, represent the value that cuts off the upper 100 percentile of the central t distribution with degrees of freedom and let F(x,,) denote the cumulative distribution function at the point x of a noncentral t distribution with degrees of freedom and noncentrality parameter, . The power for single-stage testing can be written as P₁ = 1 - F (t_1/2,n - 2,₁).

If one splits one's sample into two nonoverlapping subsamples to be used in the two stages, then the power calculation is somewhat more complex. Miller and colleagues ⁽⁷⁾ did not state how the subjects should be divided between the two stages, but for our subsequent calculations we will assume subjects are divided equally between the two stages. Because a significant result will occur only if the gene showed statistically significant differences at both stages, the power for two-stage testing will equal the product of the probability (power) of obtaining a significant result at stage 1 and the probability (power) of getting a significant result at stage 2 given a significant result at stage 1. The probability of getting a significant result at stage 1 can be derived just as above for single-stage testing, with the exception that the sample size will now be half of that used in the single-stage design. As a consequence, the noncentral t distribution will now have

₂=

and df = n - 2. Once a significant result is obtained at stage 1, the conditional probability of getting a significant result at stage 2 depends on how many other genes (C) were declared significant at stage 1. Because C is a random variable, we must then sum these conditional powers over all possible outcomes for C, weighting by the probability of that outcome. Then we can write the power for two-stage testing as

Note that the third term within the square brackets on the right side of the equation is for the one-tailed test at stage 2. For any given value of c, P(C = c) depends on the power of the tests for the other genes that may vary from one data set to the next and will be unknown. Therefore, in order for us to calculate P(C = c) we need to assume some model. For simplicity, we assume that the null hypothesis is true for all genes except the one for which we are calculating power. Were we to assume the null hypothesis is not true for other genes as well, we would increase the probability of declaring a larger number of genes significant at stage 1. As more genes are declared significant at stage 1, the stage 2 alpha level used for each individual test will become smaller, hence reducing the stage 2 power for that test. As a consequence, the overall power will be smaller than it would be under the assumption that the null hypothesis is true for all genes except the one of interest. That is, by assuming that the null hypothesis is true for all genes except the one for which we are calculating power, we are, for any particular effect size and sample size, deriving the maximum possible power for the two-stage procedure given by Miller and colleagues ⁽⁷⁾. Using the same reasoning as for M above, we find that C will follow a binomial distribution with parameters k - 1 and ₁ and we can write the power for the two-stage design as

P₂=

.

	Quantitative Results

Top Abstract The Two-Stage Approach Type 1 Error (False-Positive)... Power and False Negatives Quantitative Results Conclusions References

 
Having derived expressions for power by using both single-stage and two-stage testing, we can begin to compare their relative power. Let us return to the example of 10,000 genes given by Miller and colleagues. To maintain _ew = .05 with single-stage testing of 10,000 genes requires a per test alpha of ₁ = 5.13 x 10^-6. Fig. 1 and Fig. 2 compare the power for single-stage and two-stage testing with two different levels of ₁ (.001 and .0001) across a range of effect sizes. Fig. 1 corresponds to the example of 10 subjects per group ⁽⁷⁾, whereas Fig. 2 demonstrates the same results for a study with 20 subjects per group. The figures clearly demonstrate that the two-stage procedure of Miller and colleagues does not achieve the goal of providing a method that reduces false negatives (i.e., increases power). In fact, it can even exacerbate the very problem it is intended to alleviate. For example, under the scenario considered in that paper (₁ = .001, = 3.0, and n = 10), the power for a single-stage test is .614 (calculations were conducted by using SAS/IML [SAS Institute, Cary, NC]) but the power is only .409 for the two-stage approach.

View larger version (13K): [in this window] [in a new window]   Figure 1. Comparison of the power as a function of the effect size () for a single-stage design vs the Miller two-stage design, using a stage 1 alpha level (1) of .001 and .0001 for an overall study with 10 subjects in each of two groups.

View larger version (13K): [in this window] [in a new window]   Figure 2. Comparison of the power as a function of the effect size () for a single-stage design vs the Miller two-stage design using a stage 1 alpha level (1) of .001 and .0001 for an overall study with 20 subjects in each of two groups.

	Conclusions

Top Abstract The Two-Stage Approach Type 1 Error (False-Positive)... Power and False Negatives Quantitative Results Conclusions References

 
In conclusion, the two-stage procedure offered by Miller and colleagues ⁽⁷⁾ often fails to hold the overall experimentwise type 1 error rate to some desired alpha level. Many choices of ₁, the per test alpha level at stage 1, will result in an overall experimentwise type 1 error rate that is overly conservative. Less conservative, single-stage methods of holding the overall type 1 error rate to any desired level exist ⁽¹⁰⁾. Moreover, this two-stage method can also exacerbate the false-negative (type 2 error) rate; that is, it can decrease power compared with a single-stage method.

Although the two-stage design does not fare well when compared statistically with the single-stage design, there may be nonstatistical concerns that increase the attractiveness of the two-stage design. For example, it is possible that such a two-stage approach could improve power per dollar spent on a study if, at stage 2, one needed only to assay a subset of all genes on the array used in stage 1 and the cost for assaying a subset was less than the cost for assaying the entire set. In such situations, both the costs and required resources per subject in stage 2 (and hence the entire study) might be substantially reduced. Furthermore, the two-stage design proposed by Miller and colleagues represents only one possible type of two-stage (or more generally multistage) design. It is possible that other two-stage designs could be proposed that have better statistical properties and compare more favorably with the single-stage design.

In attempting to interpret these results, one question that may be of primary interest to researchers regards the size of the mean differences represented by the effect sizes shown in Fig. 1 and Fig. 2. To address this issue, we can offer the following information. Writing from the social science perspective, Cohen ⁽¹¹⁾ defined "small," "medium," and "large" effect sizes as values of 0.20, 0.50, and 0.80, respectively. By this standard, an effect size of 3.0, as in the example considered by Miller and colleagues ⁽⁷⁾, is extremely large. However, in basic laboratory research, effect sizes are often much larger. For example, consider a study of a knock-out mouse model of hereditary hemochromatosis ⁽¹²⁾. In that study, when knock-out mice were compared with wild-type mice, the iron concentration in livers were 170 ± 15 µg/g (mean ± standard deviation) in controls and 1010 ± 50 µg/g in beta 2m (-/-) mice. This represents an effect size of 22.6. Unfortunately, because fold change is not a statistic that takes within-group variability into account, there is no way to directly translate an effect size expressed as a standardized mean difference into a specific fold-change value.

Finally, the two-stage procedure discussed operates from a strictly frequentist point of view under the seemingly implausible assumption of the null hypothesis being true for all genes studied. Alternatives to a strict frequentist approach exist (e.g., ^(3)(13)(14)) and are seen by many (e.g., ⁽¹⁵⁾) to be preferable when many tests are conducted and a global null hypothesis seems untenable.

	Acknowledgments

 
This research was supported in part by NSF Grant 0090286 and NIH Grants R01ES09912, P30DK056336, R01DK56366, R01AG018922, P01AG11915, and U24DK058776.

Received July 9, 2001

Accepted January 1, 2002

	References

Top Abstract The Two-Stage Approach Type 1 Error (False-Positive)... Power and False Negatives Quantitative Results Conclusions References

 

1. Windruch R, Kayo T, Lee CK, Prolla TA, 2001. Microarray profiling of gene expression in aging and its alteration by caloric restriction in mice. J Nutr 131:918S-923S. [Abstract/Free Full Text]
2. Pilarsky CP, Schmitt AO, Dahl E, Rosenthal A, 1999. Microarrays—chances and challenges. Curr Opin Mol Ther. 1:727-736. [Medline]
3. Allison DB, Gadbury G, Heo M, et al. Statistical methods for the analysis of microarray gene expression data: a mixture model approach. Comp Stat Data Anal. In press.
4. Neyman J, 1977. Frequentist probability and frequentist statistics. Found Prob Stat I Synth. 36:97-131.
5. Thomas DC, Siemiatycki J, Dewar R, Robins J, Goldberg M, Armstrong BG, 1985. The problem of multiple inference in studies designed to generate hypotheses. Am J Epidemiol. 6:1080-1095.
6. Bland JM, Altman DG, 1995. Multiple significance tests—the Bonferroni method. BMJ 310:170[Free Full Text]
7. Miller RA, Galecki A, Shmookler-Reis RJ, 2001. Interpretation, design, and analysis of gene array expression experiments. J Gerontol Biol Sci. 56A:B52-B57. [Abstract/Free Full Text]
8. Prolla TA, Allison DB, Weindruch R, 2001. Response to: "Interpretation, design and analysis of gene array expression experiments.". J Gerontol Biol Sci. 56A:B327-B329. [Medline]
9. Lykken DT, 1968. Statistical significance in psychological research. Psychol Bull. 70:151-159. [Medline]
10. Hochberg Y, Tamhane AC. Multiple Comparison Procedures. New York: Wiley; 1987.
11. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Erlbaum; 1988.
12. Moos T, Trinder D, Morgan EH, 2000. Cellular distribution of ferric iron, ferritin, transferrin, and divalent metal transporter 1 (DMT1) in substantia nigra and basal ganglia of normal and beta 2-microglobulin deficient mouse brain. Cell Mol Biol. 46:549-561. [Medline]
13. Lee MLT, Kuo FC, Whitmore GA, Sklar J, 2000. Importance of replication in microarray gene expression studies: statistical methods and evidence from repetitive cDNA hybridizations. Proc Natl Acad Sci USA. 97:9834-9839. [Abstract/Free Full Text]
14. Manduchi E, Grant GR, McKenzie SE, Overton GC, Surrey S, Stoeckert CJ, 2000. Generation of patterns from gene expression data by assigning confidence to differentially expressed genes. Bioinformatics 16:685-698. [Abstract/Free Full Text]
15. Steenland K, Bray I, Greenland S, Boffetta P, 2000. Empirical Bayes adjustments for multiple results in hypothesis-generating or surveillance studies. Cancer Epidem Biol. 9:895-903.

This article has been cited by other articles:

				G. L Gadbury, G. P Page, J. Edwards, T. Kayo, T. A Prolla, R. Weindruch, P. A Permana, J. D Mountz, and D. B Allison Power and sample size estimation in high dimensional biology Statistical Methods in Medical Research, August 1, 2004; 13(4): 325 - 338. [Abstract] [PDF]

This Article Abstract Full Text (PDF) Services Download to citation manager Citing Articles Citing Articles via HighWire PubMed PubMed Citation