The question from Mr. Li appears to be the matter of type I and type II error in hypothesis testing. This was elaborated in a book from Robert R. Sokal and F. James Rohlf (1994) Biometry: The Principles and Practices of Statistics in Biological Research.
If your goal is to compare alternative model or multiple models, Byaesian statistics can be very helpful for model checking and model selection. Some information can be found in a book from Ming-Hui Chen/Qi-Man Shao/Joseph G. Ibrahim (2001) Monte Carlo Methods in Bayesian Computation. Shiyun Wen ________________________________ From: Gavin Simpson <[email protected]> To: Sent: Mon, February 8, 2010 3:21:28 AM Subject: Re: [ECOLOG-L] Statistical test about equality On Sat, 2010-02-06 at 21:44 -0800, Li An wrote: > Dear Ecologers, > > In testing ecological models, we often use t-test as a way to compare > our model results with observed data. If they are close enough, we > obtain more confidence about our model. However, in most traditional > situations, we put "no difference" as the null and regarded it as the > default. This means that unless we find substantial evidence, we would > retain the null hypothesis. For instance, we can use this type of test > to examine if a drug has a noticeable effect. > > In our model performance situation (testing observed data = predicted > numbers from a model, assuming data independence), I argue that we > should keep the alternative hypothesis as the default, making every > effort to find substantial evidence to support the null hypothesis (if > unable, we retain the alternative hypothesis related to inequality > between the model predictions and the data). In this case, we can still > use the traditional test statistic such as z or p values, but interpret > the results differently. Rather than using the criterion of p > 0.05 (or > Z<1..96 or t < a big number) to retain the null hypothesis, we should use > a more strict standard--e.g., p > a much larger number (e.g., 0.9) or z > < a much smaller number (e.g.,0.125), to retain the null hypothesis > about equality between the model predictions and the data. This seems > mofrea philosophical issue. Does this make sense? > > Li You might like to look at the field of equivalency testing. Some references cited in the 'equivalence' package by Andrew Robinson for R are: Robinson, A.P., and R.E. Froese. 2004. Model validation using equivalence tests. Ecological Modelling 176, 349–358. Wellek, S. 2003. Testing statistical hypotheses of equivalence. Chapman and Hall/CRC. 284 pp.. Westlake, W.J. 1981. Response to T.B.L. Kirkwood: bioequivalence testing - a need to rethink. Biometrics 37, 589-594. HTH G -- %~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~% Dr. Gavin Simpson [t] +44 (0)20 7679 0522 ECRC, UCL Geography, [f] +44 (0)20 7679 0565 Pearson Building, [e] gavin.simpsonATNOSPAMucl.ac.uk Gower Street, London [w] http://www.ucl.ac.uk/~ucfagls/ UK. WC1E 6BT. [w] http://www.freshwaters..org.uk %~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
