On 12/08/2011 08:24, Petr Baudis wrote:
On Thu, Aug 04, 2011 at 08:21:27PM +0200, Kahn Jonas wrote:
And I'm not even taking into account the fact that you want to continue
testing till you reach significance. That would again require you take a
lower level.
I have seen this claim multiple times and I would be interested in some
more detailed argument - could someone elaborate, please? I tried
looking over the wikipedia and other pages but couldn't find anything
and I'm not sure how this could break things and more importantly, how
much it would break things in practice.
Suppose I have a coin which I believe is biassed in favour of heads. I
decide "I will toss it 100 times, count the number of heads, and look it
up in a statistical table. If it is significant at the 5% level, I will
assume it is biassed and tell people about it; otherwise, I will shut up
and forget about it." That might be a reasonable decision.
But suppose instead I decided "I will toss it many times, counting the
number of heads. After each observation of heads, I will look the
results so far up in a statistical table. If it is significant at the 5%
level, I will assume it is biassed and tell people about it; otherwise,
I will carry on until I get bored." Eventually, you announce "I have
tossed my coin 1573 times, and I find that the excess of heads is
significant at the 5% level." People should be sceptical. The
statistical test you used is based on the assumption that you decided at
the start to toss your coin 1573 times, but that is not the procedure
you actually used. Maybe your coin is biassed, maybe it isn't; but the
"5% level" you are claiming is based on a false assumption.
If you don't see why that it is false, consider this more extreme
example. "I will toss it 1000 times, look for the run of 100 tosses that
has the most heads, and look up the results for that run in a
ststistical table and announce its significance level."
Nick
(I admit that my probability/statistics background is very very sketchy.
One of the things I want to fix during my PhD studies. ;-)
--
Nick Wedd
[email protected]
_______________________________________________
Computer-go mailing list
[email protected]
http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
