On 3/21/18 10:53 AM, tom...@gmail.com wrote:
On Wednesday, March 21, 2018 at 3:30:48 PM UTC+1, Boris Zbarsky wrote:
The point is to gather data on how this behaves in the wild. If the
study is opt-in, then you have to try to figure out what part of the
effect you're seeing (if any) is just selection effects.
From my understanding of Patrick's original message, the control in this study isn't
"the 50% of Nightly users who don't have DoH enabled", but instead the existing
DNS calls on each and every resolve
Sure. But the point is that you have to assume that the relationship
between DoH and the existing DNS calls among the study sample is somehow
representative of that relationship in the overall userbase, right?
While yes, self-selection bias could still influence the results, it isn't
obvous to me at least that it would be significant.
My point is that to evaluate whether it's significant one needs to know
about who is self-selecting into the study and how they differ from the
people who are not self-selecting.
How would you propose evaluating that?
So this doesn't _need_ to be opt-out, as long as you're willing to not
believe any data it produces. But then what's the point?
Thats seems like an exageration.
OK, fair. If we have a large-enough group self-select in and BoH turns
out to not work well enough for a large enough percentage of it, that
can be used to place a lower bound on the fraction of users it won't
work for in general.
But conclusions other than that seem a bit hard to come by...
-Boris
P.S. There is, of course, the problem that the nightly audience is
_already_ subject to serious selection effects.
P.P.S. Note that I am not explicitly advocating for this study; just
presenting an argument for why making sense of opt-in data might be ...
complicated at best.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
