[Apologies - I replied directly to Shiyu Wen as his email arrived separately from his posting to the list - here is my response]
I am aware that Li was phrasing their question in terms of classical hypothesis testing where H_0 is the /Null/ model. However, it is not immediately clear how one uses this to say something about the Null hypothesis with appropriate power. Equivalency testing posits exactly the hypothesis that Li wants to test, namely, H_0 == model differs from observed, H_1 == model is same as observed. We place the emphasis on the model demonstrating that it is similar to the observed. The first reference I suggested deals with exactly the issue Li is tackling; namely one of comparison of model results to observed data: Robinson, A.P., and R.E. Froese. 2004. Model validation using equivalence tests. Ecological Modelling 176, 349–358. The abstract for which states: Model validation that is based on statistical inference seeks to construct a statistical comparison of model predictions against measurements of the target process. Previously, such validation has commonly used the hypothesis of no difference as the null hypothesis, that is, the null hypothesis is that the model is acceptable. This is unsatisfactory, because using this approach tests are more likely to validate a model if they have low power. Here we suggest the usage of tests of equivalence, which use the hypothesis of dissimilarity as the null hypothesis, that is, the null hypothesis is that the model is unacceptable. Thus, they flip the burden of proof back onto the model. We demonstrate the application of equivalence testing to model validation using an empirical forest growth model and an extensive database of field measurements. Finally we provide some simple power analyses to guide future model validation exercises. Which suggests that my suggestion was wholly appropriate. Indeed, Bayesian methods can be useful for model checking and selection. However they bring with them a lot of extra baggage, and furthermore one assumes that the ecological model is in some way /statistical/ and it may well be a process or deterministic model and all one has to work with is the model output. All the best, G On Sun, 2010-02-07 at 17:50 -0800, Shiyun Wen wrote: > The question from Mr. Li appears to be the matter of type I and type II err= > or in hypothesis testing. This was elaborated=C2=A0in a book from Robert R.= > Sokal and F. James Rohlf (1994) Biometry: The Principles and Practices of = > Statistics in Biological Research.=0A=0AIf your goal is to compare alternat= > ive model or multiple models, Byaesian statistics can be very helpful for m= > odel checking and model selection. Some information can be found in a book = > from Ming-Hui Chen/Qi-Man Shao/Joseph G. Ibrahim (2001) Monte Carlo Methods= > in Bayesian Computation. =0A=0AShiyun Wen=0A=0A=0A________________________= > ________=0AFrom: Gavin Simpson <[email protected]>=0ATo: Sent: Mon, F= > ebruary 8, 2010 3:21:28 AM=0ASubject: Re: [ECOLOG-L] Statistical test about= > equality=0A=0AOn Sat, 2010-02-06 at 21:44 -0800, Li An wrote:=0A> Dear Eco= > logers,=0A> =0A> In testing ecological models, we often use t-test as a way= > to compare =0A> our model results with observed data. If they are close en= > ough, we =0A> obtain more confidence about our model. However, in most trad= > itional =0A> situations, we put "no difference" as the null and regarded it= > as the =0A> default. This means that unless we find substantial evidence, = > we would =0A> retain the null hypothesis. For instance, we can use this typ= > e of test =0A> to examine if a drug has a noticeable effect.=0A> =0A> In ou= > r model performance situation (testing observed data =3D predicted =0A> num= > bers from a model, assuming data independence), I argue that we =0A> should= > keep the alternative hypothesis as the default, making every =0A> effort t= > o find substantial evidence to support the null hypothesis (if =0A> unable,= > we retain the alternative hypothesis related to inequality =0A> between th= > e model predictions and the data). In this case, we can still =0A> use the = > traditional test statistic such as z or p values, but interpret =0A> the re= > sults differently. Rather than using the criterion of p > 0.05 (or =0A> Z<1= > ..96 or t < a big number) to retain the null hypothesis, we should use =0A> = > a more strict standard--e.g., p > a much larger number (e.g., 0.9) or z =0A= > > < a much smaller number (e.g.,0.125), to retain the null hypothesis =0A> = > about equality between the model predictions and the data. This seems =0A> = > mofrea philosophical issue. Does this make sense?=0A> =0A> Li=0A=0AYou=C2= > =A0 might like to look at the field of equivalency testing. Some=0Areferenc= > es cited in the 'equivalence' package by Andrew Robinson for R=0Aare:=0A=0A= > Robinson, A.P., and R.E. Froese. 2004. Model validation using=0Aequivalence= > tests. Ecological Modelling 176, 349=E2=80=93358.=0A=0AWellek, S. 2003. Te= > sting statistical hypotheses of equivalence. Chapman=0Aand Hall/CRC. 284 pp= > ..=0A=0AWestlake, W.J. 1981. Response to T.B.L. Kirkwood: bioequivalence tes= > ting=0A- a need to rethink. Biometrics 37, 589-594.=0A=0AHTH=0A=0AG=0A-- = > =0A%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%=0A= > Dr. Gavin Simpson=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 [t] +44 (0)20 76= > 79 0522=0AECRC, UCL Geography,=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 [f] +44 (0= > )20 7679 0565=0APearson Building,=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = > [e] gavin.simpsonATNOSPAMucl.ac.uk=0AGower Street, London=C2=A0 =C2=A0 =C2= > =A0 =C2=A0 =C2=A0 [w] http://www.ucl.ac.uk/~ucfagls/=0AUK. WC1E 6BT.=C2=A0 = > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 [w] http://www.freshwaters= > ..org.uk=0A%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%= > ~%~%=0A=0A=0A=0A -- %~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~% Dr. Gavin Simpson [t] +44 (0)20 7679 0522 ECRC, UCL Geography, [f] +44 (0)20 7679 0565 Pearson Building, [e] gavin.simpsonATNOSPAMucl.ac.uk Gower Street, London [w] http://www.ucl.ac.uk/~ucfagls/ UK. WC1E 6BT. [w] http://www.freshwaters.org.uk %~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
