AB Testing Workshop Service
DataSF offers AB testing design workshops to departments interested in comparing the effectiveness of different forms of communications like letters or emails.
The redesign of your communication rests on using the principles of Make it Easy, Attractive, Social and Timely or EAST, a framework developed by the Behavioral Insights Team.
Once we design a new letter or communication using the EAST principles, we conduct an experiment (AB test it). The way it works is that:
- We identify the target population
- Split it randomly into two groups: treatment and control
- Run the text
- Compare the results
If you are interested in an AB Testing Design Workshop, contact us for more details.
On this page we will add AB experiments as they are complete.
Back to topTreasurer & Tax Collector: A better letter
The Office of the Treasurer & Tax Collector (TTX) collects City taxes on a range of things, including
- Taxes on land and buildings (real property) and
- Taxes on office furniture, boats, equipment, etc. (personal property)
The key difference between real and personal property is that the City has the right to seize land and buildings if taxes go unpaid. This gives the City a bit more leverage. But when it comes to personal property, the City relies on self-reported values and cannot simply seize the property if taxes are not paid. This is the difference between a “secured” and “unsecured” tax.
That’s why the Office thought behavioral science insights could help make their tax payment letter more compelling.
It is harder to collect taxes on unsecured property like office furniture, boats, or equipment. Unlike land or structures, such as secured property, the City can't seize unsecured property in the event of nonpayment.
Back to topBefore / After
To improve the letter, the Office partnered with DataSF to conduct a workshop where five teams competed to design a better letter. The winning letter design used the key concepts of making it easy, attractive, social, and timely.
The image below provides before and after with the behavioral insights used. In particular, we leveraged concepts of making it easy, using social comparison and norms and making the action items more attractive.
The new letter, on the righthand side includes the EAST principles.
Back to topExperiment and Results
To test the new letter, the Office randomly split the 3,495 delinquent taxpayers into 2 groups:
- 1,732 received the old notice (control)
- 1,763 received the new notice (treatment)
Responses were due December 15, 2017. Based on a response rate of 16.3% for the treatment and 14.0% for the control, we can say with 90% confidence that the new letter improved the response rate. In this case, it improved it by 17%.
TTX is now looking to run experiments on all its tax letters!
The rate of response to the new letter (treatment) was 17% higher than the response rate for the old letter. We are 90% confidant that our new letter improved response rates with p=.0515
Back to topClient Details
Client Team
- Gilbert Jeung, IT, Treasurer & Tax Collector
- Greg Kato, Compliance Director, Treasurer & Tax Collector
- John Krump, Bureau of Delinquent Revenue, Treasurer & Tax Collector
- Lauren Philibosian, Property Tax, Treasurer & Tax Collector
- Margarita Rodriguez, Bureau of Delinquent Revenue, Treasurer & Tax Collector
- Darryl Yee, IT, Treasurer & Tax Collector
Testimonial
Back to top"Conducting an A/B Test with DataScienceSF was a great experience for the Office of the Treasurer & Tax Collector. They connected us with the Behavioral Insights Team through the What Works Cities initiative. We had training from the state of the industry in using behavioral insights for government work. They expertly facilitated a process where our team quickly learned the major elements of the EAST principles and rapidly put them into practice. We are incorporating A/B testing and behavioral insights into our future communications with taxpayers and look forward to making more evidence-based decisions."
Greg Kato, Compliance Director, Treasurer & Tax Collector