Very cool! This will be super helpful for scaling up more experiments.
-- Alex Davis // Mountain View Product Manager // FxA & Sync (415) 769-9247 IRC & Slack: adavis On Tue, Jan 31, 2017 at 5:55 AM, Shane Tomlinson <[email protected]> wrote: > Using the infrastructure that was built into the content server, a user > could only be part of 1 A/B test at a time, even if tests were sufficiently > "independent". The mentioned change is to allow the user to be part of more > than 1 test. If two tests have the possibility of interfering with each > other, we still have the capability to ensure the user is a member of at > most one of those tests, so we haven't lost any functionality either. > > Shane > > On Mon, Jan 30, 2017 at 6:23 PM, Alex Davis <[email protected]> wrote: > >> The last minute train-79 activity to roll out "Connect another device" to >>> 25% of people has the side effect of allowing us to roll out multiple >>> independent A/B tests with minimal fuss. >> >> >> Can you guys elaborate on this? I'd love to learn more :) >> >> -- >> Alex Davis // Mountain View >> Product Manager // FxA & Sync >> (415) 769-9247 >> IRC & Slack: adavis >> >> On Fri, Jan 27, 2017 at 3:09 PM, Shane Tomlinson <[email protected]> >> wrote: >> >>> >>> >>> *Theme: Train-80 has begun* >>> This was a very dense and technical meeting, so the notes are a bit >>> longer than usual. >>> >>> Action items from last time: >>> >>> - Train-79 was tagged once, twice, then a third time on the content >>> server. The last minute flurry of activity was to handle two things: >>> >>> - Bump the number of users who would see the new "connect another >>> device" flow to 25%. >>> >>> - "Docker all the things" was a bit rocky. Circle would report >>> multiple test runs as fully successful, while run against a branch, >>> and >>> then fail on master. jbuck and vlad wrangled the Docker build into >>> shape. >>> >>> New >>> >>> - Ryan Feeley has lots of new DataDog graphs related to connect >>> another device. Results continue to be positive, and now we have a better >>> idea of ow many people see each particular screen. A surprising number of >>> people seem to be verifying in a 2nd, disconnected, Firefox instance. >>> These >>> users appear to be highly motivated to sign in on that 2nd device. The >>> numbers also reveal that most users sign up and verify on the same device >>> (not too surprising), meaning the forthcoming "Send a Firefox Mobile >>> install link via SMS" feature should make it significantly easier for >>> these >>> users to connect their mobile devices. >>> >>> - Phil, Vijay and I had a short pow-wow about the "Send SMS" >>> feature. I'm stretched thin, and Vijay has offered to help for the next >>> week. We figured out the major components we'll need, and which are >>> self-contained enough that Vijay could handle in a short amount of time. >>> Vijay is going to dive in on customs-server and fxa-js-client work, Phil >>> is >>> going to handle the auth-server, and I've successfully removed myself >>> from >>> active development for at least a week. >>> >>> - Phil has been diving into the content server to re-generate >>> flow-id's and flow start times when a user signs out. I think with [1], >>> he >>> has a winner of a PR that ends up simplifying several pieces of internal >>> state management. >>> >>> - Divya's work to show OAuth scopes in "Devices and Apps" is ~2/3 of >>> the way done and should be ready to go with train-80. >>> >>> - Vlad has received feedback on his Web Sessions work and is going >>> to update the "Sessions API" on the auth server to also return associated >>> device info. This effort is to reduce the number of XHR calls that need >>> to >>> be made to show a list of connected devices, OAuth apps, and normal web >>> sessions. >>> >>> - The last minute train-79 activity to roll out "Connect another >>> device" to 25% of people has the side effect of allowing us to roll out >>> multiple independent A/B tests with minimal fuss. >>> >>> - Phil is also working on "Functional Testing All The Things All The >>> Time" - a recurring problem is a checkin to the auth/oauth/profile >>> servers >>> or js-client sometimes cause an unexpected test failure on the content >>> server. It's always a drag when it happens. Phil thinks so too, so is >>> experimenting with running a subset of content server functional tests on >>> every checkin to the auth server. From what I saw in IRC, he has managed >>> to >>> get the test run to complete in under 10 minutes. >>> >>> - In Hawaii, we spent a few hours with Mark H discussing how we >>> could get around [2]. There are several ways a user can have an active >>> Sync >>> session and have the FxA front-end forget about their session. This >>> happens >>> when the browser has the user's sessionToken but the user's FxA >>> localStorage gets wiped. There are several ways this can happen - either >>> a >>> profile reset, or by modifying history settings, or an explicit "forget >>> history", and even e10s plays a part (though this may have been >>> resolved). >>> When this happens, users who click "Manage Account" have to sign in >>> again. >>> It's less than awesome. The decided upon approach is do finally do the >>> WebChannel handshake we've been discussing for nearly 2 years where when >>> FxA loads, it'd request from the browser it's view of the FxA world. The >>> browser would respond with a sessionToken, email, and user id. The FxA >>> front end would use that as the canonical source of truth. I have 2 >>> patches, one to the browser, one to FxA, to make this happen. We got >>> stuck >>> on how to approach private browsing mode, discussion in [3]. Since >>> Hawaii, >>> nothing has been done and we'd like to finish this before forgetting >>> about >>> it again. >>> >>> - Vlad has opened an auth-server PR [4] that should give us more >>> insights into why the DB connections on fxa-dev boxes sometimes go >>> completely haywire. >>> >>> - New auth-server loads tests are in the works using [5], when tests >>> are run, results will be automatically sent to DataDog. This effort is to >>> see how changes in the auth-server affect load and responsiveness. >>> >>> - The question came up in IRC the other day,"What would an ideal >>> dynamic configuration system look like?" Able has proven itself valuable, >>> but we are using it in ways it's not meant to be used and that has >>> down-sides. We are asking ourselves what a better dynamic configuration >>> system would look like, and leave Able to do what it's good at - >>> bucketing >>> users into experiments. >>> >>> Action items: >>> >>> - Vlad to tag content-server train-79 >>> - stomlinson to bug "someone" about moving forward with [3]. >>> - stomlinson to update fxa-ci w/ newest servers. >>> - vlad to look at never-ending shrinkwrap problem in auth-server#1612 >>> - rfeeley to start new feature doc for [6] - sending sign-in >>> confirmation to users via sync tab. >>> >>> Shane >>> >>> [1] - https://github.com/mozilla/fxa-content-server/pull/4676 >>> [2] - https://github.com/mozilla/fxa-content-server/issues/4252 >>> [3] - https://bugzilla.mozilla.org/show_bug.cgi?id=1323853 >>> [4] - https://github.com/mozilla/fxa-auth-server/pull/1627 >>> [5] - https://artillery.io/ >>> [6] - https://github.com/mozilla/fxa-content-server/issues/4667 >>> >>> _______________________________________________ >>> Dev-fxacct mailing list >>> [email protected] >>> https://mail.mozilla.org/listinfo/dev-fxacct >>> >>> >> >
_______________________________________________ Dev-fxacct mailing list [email protected] https://mail.mozilla.org/listinfo/dev-fxacct
