BEGIN:VCALENDAR VERSION:2.0 PRODID:-//wp-events-plugin.com//7.2.3.1//EN TZID:America/New_York X-WR-TIMEZONE:America/New_York BEGIN:VEVENT UID:909@fds.yale.edu DTSTART;TZID=America/New_York:20260218T113000 DTEND;TZID=America/New_York:20260218T130000 DTSTAMP:20260129T204825Z URL:https://fds.yale.edu/events/fds-colloquium-david-holtz-columbia/ SUMMARY:FDS Colloquium: David Holtz (Columbia)\, "Reducing Symbiosis Bias T hrough Better A/B Tests of Recommendation Algorithms" DESCRIPTION:\nAbstract: It is increasingly common in digital environments to use A/B tests to compare the performance of recommendation algorithms. However\, such experiments often violate the stable unit treatment value a ssumption (SUTVA)\, particularly SUTVA’s “no hidden treatments” assu mption\, due to the shared data between algorithms being compared. This re sults in a novel form of bias\, which we term “symbiosis bias\,” where the performance of each algorithm is influenced by the training data gene rated by its competitor. In this paper\, we investigate three experimental designs–cluster-randomized\, data-diverted\, and user-corpus co-diverte d experiments–aimed at mitigating symbiosis bias. We present a theoretic al model of symbiosis bias and simulate the impact of each design in dynam ic recommendation environments. Our results show that while each design re duces symbiosis bias to some extent\, they also introduce new challenges\, such as reduced training data in data-diverted experiments. We further va lidate the existence of symbiosis bias using data from a large-scale A/B t est conducted on a global recommender system\, demonstrating that symbiosi s bias affects treatment effect estimates in the field. Our findings provi de actionable insights for researchers and practitioners seeking to design experiments that accurately capture algorithmic performance without bias in treatment effect estimates introduced by shared data.\n\n\n\n Joint wo rk with Jennifer Brennan\, Yahu Cong\, Yiwei Yu\, Lina Lin\, Yajun Peng\, Changping Meng\, Ningren Han\, and Jean Pouget-Abadie.\n\n\n\nSpeaker bio : David Holtz is an Assistant Professor in the Decision\, Risk\, and Oper ations Division at Columbia Business School. Previously\, he was an Assist ant Professor at UC Berkeley’s Haas School of Business. He earned his Ph .D. from the MIT Sloan School of Management in the Information Technology group\, and also holds an M.A. in Physics and Astronomy from Johns Hopkins University and a B.A. in Physics from Princeton University.\n\n\n\nHoltz studies online marketplaces\, digital platforms\, and AI-enabled systems u sing large-scale field experiments and rich digital trace data. His resear ch agenda spans topics including the economic and organizational impacts o f artificial intelligence\, online trust and reputation system design\, pe rsonalized recommendations\, and the evaluation of interventions in digita l environments with interdependence and spillovers. His work has appeared in leading journals including \;Management Science\, \;Marketing S cience\, \;Nature Human Behaviour\, \;Science Advances\, and \ ;Proceedings of the National Academy of Sciences\, as well as top computer science conferences such as ACM CHI\, the ACM Conference on Economics and Computation\, and the ACM Web Conference. His research has been covered b y outlets including \;The Washington Post\, \;The Economist\, MSNB C\, and \;The Boston Globe\, and was cited in the 2025 White House&nbs p\;Economic Report of the President.\n\n\n\nBefore returning to academia\, Holtz worked in the Bay Area as a data scientist and product manager at s everal technology firms\, including Airbnb\, where he was a founding membe r of the company’s algorithmic pricing team\, and TrialPay (acquired by Visa). In carrying out his research agenda\, he continues to work closely with leading technology firms\, including Airbnb\, Microsoft\, Spotify\, a nd Meta. He is a Research Affiliate at the MIT Initiative on the Digital E conomy\, a member of the Columbia Data Science Institute\, and a co-organi zer of the Conference on Digital Experimentation (CODE@MIT). He is also cu rrently a Visiting Researcher at OpenAI.\n CATEGORIES:FDS Events,Colloquium LOCATION:Yale Institute for Foundations of Data Science & Webcast\, 219 Pro spect Street\, 13th Floor\, New Haven\, CT\, 06511\, United States X-APPLE-STRUCTURED-LOCATION;VALUE=URI;X-ADDRESS=219 Prospect Street\, 13th Floor\, New Haven\, CT\, 06511\, United States;X-APPLE-RADIUS=100;X-TITLE= Yale Institute for Foundations of Data Science & Webcast:geo:0,0 END:VEVENT BEGIN:VTIMEZONE TZID:America/New_York X-LIC-LOCATION:America/New_York BEGIN:STANDARD DTSTART:20251102T010000 TZOFFSETFROM:-0400 TZOFFSETTO:-0500 TZNAME:EST END:STANDARD END:VTIMEZONE END:VCALENDAR