Hello,
This topic has generated an interesting philosophical discussion. I would
like to comment on a technical detail from an earlier post that might cause
some confusion. An earlier poster on this topic wrote:
>The likelihood of a model given the data
>is always between 0 and 1, and thus
>log likelihood is always a negative
>number (it could be zero; but it's
>always non-positive).
This is not a correct statement. The most simplistic illustration of this is
to examine the likelihood of a normal distribution:
L(mu,sigma | x) = Prod_{i=1:n} 1/[sigma*sqrt(2*pi)] exp[-(x_i - mu)^2 / (2 *
sigma^2)]
where x is a data vector of length n, sigma is the standard deviation and mu
is the mean of the proposed distribution (i.e. the model).
If a data point is equal to the mean, and sigma is < 1/sqrt(2*pi), then the
likelihood for that point will be greater than 1. Indeed, with a very small
sigma, you can generate data that will yield a very large, positive log
likelihood -- and thus a negative AIC value -- if the proposed values for mu
and sigma approximate “truth” (see an example from R below).
Using a small variance term to illustrate my point may seem contrived;
however, this issue is especially important when considering other
distributions (such as the beta) that have probability density functions
that routinely exceed 1. That said, nearly every model I have ever fit has
had a positive AIC value. Based on that experience, if my model output
yielded a negative AIC I would certainly double check the calculations. My
point is simply to show that AIC is not constrained to be positive (except
in cases where the pdf of interest has a maximum <= 1).
Cheers,
James
James D. Forester
Research Associate
Center for Integrating Statistical and Environmental Science
University of Chicago
#A brief example from R
mu <- 0
sigma <- 0.2
k<-2 #i.e. two parameters for mu and simga
x<-rnorm(1000,mu,sigma)
Lik<-prod(dnorm(x,mu,sigma))
#Because with very small sigmas, Lik will approach Inf, calcualte the LogLik
LogLik<-sum(log(dnorm(x,mu,sigma)))
AIC.1<- -2*LogLik + 2*k
print(log(Lik))
print(LogLik)
print(AIC.1)
