On Wed, Feb 22, 2017 at 2:33 PM, Vijay Budhram <[email protected]> wrote:
- Connect another device (CAD) has been ramped up to 50% of users but due > to the way experiments and control groups were setup it was not exposed to > that many users. > Slight correction to this. Connect another device is in front of a magnitude of order more users, but the number of users reporting to DataDog has not jumped. When we first released the feature, the new screens were displayed to 1.25% of users who were signing up or signing in, all of these users had their metrics reported to DataDog. Two changes kept the number of users reporting to DataDog the same: * We no longer present CAD to users who are signing in. Half of the users that were eligible to see the new screens have been eliminated. * DataDog metrics are sampled at 10%. When we first rolled out the feature, *every* user who saw CAD reported their metrics, now only 10% of them do. Using Able, we select 50% of users and say "they are tentatively eligible" for the experiment. Of that 50%, we eliminate all users who are signing in, so we are down to 25%. Of that 25%, half are in the treatment, half are in the control group, now down to 12.5% of users. This 12.5% is from the general population, only 10% of the general population report metrics to DD. We are back to 1.25%, exactly the same % of users that reported when we first released the feature. Shane
_______________________________________________ Dev-fxacct mailing list [email protected] https://mail.mozilla.org/listinfo/dev-fxacct
