Events
Colloquium
Reducing Symbiosis Bias Through Better A/B Tests of Recommendation Algorithms
 |
Speaker: David Holtz (Columbia) Assistant Professor in the Decisions, Risk, and Operations (DRO) Division Columbia University Business School Wednesday, February 18, 2026 11:30AM - 1:00PM Lunch at 11:30am in 1307
Talk 12:00-1:00pm in 1327 Location: Yale Institute for Foundations of Data Science & Webcast, 219 Prospect Street, 13th Floor, New Haven, CT 06511 and via Webcast: https://yale.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=d2e4e1ac-7408-4568-91a5-b3ca01469bbb |
Abstract: It is increasingly common in digital environments to use A/B tests to compare the performance of recommendation algorithms. However, such experiments often violate the stable unit treatment value assumption (SUTVA), particularly SUTVA’s “no hidden treatments” assumption, due to the shared data between algorithms being compared. This results in a novel form of bias, which we term “symbiosis bias,” where the performance of each algorithm is influenced by the training data generated by its competitor. In this paper, we investigate three experimental designs–cluster-randomized, data-diverted, and user-corpus co-diverted experiments–aimed at mitigating symbiosis bias. We present a theoretical model of symbiosis bias and simulate the impact of each design in dynamic recommendation environments. Our results show that while each design reduces symbiosis bias to some extent, they also introduce new challenges, such as reduced training data in data-diverted experiments. We further validate the existence of symbiosis bias using data from a large-scale A/B test conducted on a global recommender system, demonstrating that symbiosis bias affects treatment effect estimates in the field. Our findings provide actionable insights for researchers and practitioners seeking to design experiments that accurately capture algorithmic performance without bias in treatment effect estimates introduced by shared data.
Joint work with Jennifer Brennan, Yahu Cong, Yiwei Yu, Lina Lin, Yajun Peng, Changping Meng, Ningren Han, and Jean Pouget-Abadie.
Speaker bio: David Holtz is an Assistant Professor in the Decision, Risk, and Operations Division at Columbia Business School. Previously, he was an Assistant Professor at UC Berkeley’s Haas School of Business. He earned his Ph.D. from the MIT Sloan School of Management in the Information Technology group, and also holds an M.A. in Physics and Astronomy from Johns Hopkins University and a B.A. in Physics from Princeton University.
Holtz studies online marketplaces, digital platforms, and AI-enabled systems using large-scale field experiments and rich digital trace data. His research agenda spans topics including the economic and organizational impacts of artificial intelligence, online trust and reputation system design, personalized recommendations, and the evaluation of interventions in digital environments with interdependence and spillovers. His work has appeared in leading journals including Management Science, Marketing Science, Nature Human Behaviour, Science Advances, and Proceedings of the National Academy of Sciences, as well as top computer science conferences such as ACM CHI, the ACM Conference on Economics and Computation, and the ACM Web Conference. His research has been covered by outlets including The Washington Post, The Economist, MSNBC, and The Boston Globe, and was cited in the 2025 White House Economic Report of the President.
Before returning to academia, Holtz worked in the Bay Area as a data scientist and product manager at several technology firms, including Airbnb, where he was a founding member of the company’s algorithmic pricing team, and TrialPay (acquired by Visa). In carrying out his research agenda, he continues to work closely with leading technology firms, including Airbnb, Microsoft, Spotify, and Meta. He is a Research Affiliate at the MIT Initiative on the Digital Economy, a member of the Columbia Data Science Institute, and a co-organizer of the Conference on Digital Experimentation (CODE@MIT). He is also currently a Visiting Researcher at OpenAI.
Add To: Google Calendar | Outlook | iCal File
- Colloquium
Submit an Event
Interested in creating your own event, or have an event to share? Please fill the form if you’d like to send us an event you’d like to have added to the calendar.