There are various factors that determine which statistics methodology we should apply to a given use case. Uber’s Experimentation Platform conducts both randomized experiments and observational studies. On average, over 1,000 experiments are running on Uber’s Experimentation Platform at any given time.īelow is a chart outlining the types of experimentation methodologies that the Experimentation Platform team uses: Figure 2. The platform also lets users configure the universal holdout, used to measure the long-term effects of all experiments for a specific domain.
SEQUENTIAL TESTING PRACTICE PROBLEMS DRIVER
For example, before Uber launched our new driver app, completely redesigned with our driver-partners in mind, it went through extensive hypothesis testings through a series of experiments conducted with our XP.Īt a high level, Uber’s XP allows engineers and data scientists to monitor treatment effects to ensure they do not cause regressions of any key metrics. There are over 1,000 experiments running on our platform at any given time. The platform supports experiments across our driver, rider, Uber Eats, and Uber Freight apps and is widely used to run A/B/N, causal inference, and multi-armed bandit (MAB)-based continuous experiments. Uber’s Experimentation Platform (XP) plays an important role in this process, enabling us to launch, debug, measure, and monitor the effects of new ideas, product features, marketing campaigns, promotions, and even machine learning models. Uber applies several experimental methodologies to use cases as diverse as testing out a new feature to enhancing our app design.
SEQUENTIAL TESTING PRACTICE PROBLEMS HOW TO
Finally, we show how to compute confidence sequences for the difference between quantiles of two arms in an A/B test, along with corresponding always-valid p-values.Experimentation is at the core of how Uber improves the customer experience. Simulations demonstrate that our method stops with fewer samples than existing methods by a factor of five to fifty. We apply our results to the problem of selecting an arm with an approximately best quantile in a multi-armed bandit framework, proving a state-of-the-art sample complexity bound for a novel allocation strategy. This inequality directly yields sequential analogues of the one- and two-sample Kolmogorov-Smirnov tests, and a test of stochastic dominance. Specifically, we provide explicit expressions with small constants for intervals whose widths shrink at the fastest possible rate, as determined by the law of the iterated logarithm (LIL).Īs a byproduct, we give a non-asymptotic concentration inequality for the empirical distribution function which holds uniformly over time with the LIL rate, thus strengthening Smirnov’s asymptotic empirical process LIL, and extending the famed Dvoretzky-Kiefer-Wolfowitz (DKW) inequality to hold uniformly over all sample sizes while only being about twice as wide in practice. We give two methods for tracking a fixed quantile and two methods for tracking all quantiles simultaneously. We propose new, theoretically sound and practically tight confidence sequences for quantiles, that is, sequences of confidence intervals which are valid uniformly over time. Consider the problem of sequentially estimating quantiles of any distribution over a complete, fully-ordered set, based on a stream of i.i.d.
0 kommentar(er)
