For companies that want to learn and iterate quickly there is no substitution for scalable experimentation practices — and as machine learning techniques have become increasingly democratized the burden of rigorous work is and should increasingly shift to how effectively models are tested and validated. In the pure online publisher space this problem is relatively easy, and A/B testing practices are common enough that much of this problem has largely been “solved” — but what about when A/B testing doesn’t work? Sometimes real world release mechanics or marketplace effects make traditional online experimentation practices impractical or, worse, can lead teams to conclude the wrong results. 

This session will cover the importance of good experimentation practices, and some of the tools and techniques that Uber uses to run experimentation at scale.