Experimentation is a key tool for how companies build successful products, user experiences, and businesses. It’s also a topic that is dear to our hearts here at Signific, and something we’ve helped both small and large companies with over the years.
There have never been better conditions for experimenting in many ways, but it’s also an area that is constantly evolving. Here are some of our favorite content pieces that we came across in 2023. Articles, talks and guides we’ve read and discussed internally that hopefully helps you along the experimentation journey. We’ve also gathered our own content around experimentation. Enjoy!
If you have more articles or talks to recommend feel free to ping us! Sharing is caring 💚
Have a question about product experimentation? Feel free to reach out on hej@signific.se. We have senior consultants and have worked with experimentation at companies like SVT, Polestar, King, Spotify and many more.
Intro to experimentation
New to experimentation? Don’t worry, you are not the only one. This is one article and one webinar we believe give a good introduction.
Running experimentation teams / programs
Through out the last years we’ve helped several companies start and scale experimentation within teams and whole organisations. One key take away that it is as much a cultural and organizational challenge as it is technical and statistical. Here we gathered some of our favorite talks and articles together with some shameless plugs of our own content.
Talks
- The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)
- The UX Research reckoning is here | Judd Antin (Airbnb, Meta)
Articles
- How Volvo on Demand (with some help from Signific) ran their experimentation program
- What Can Be Learned From 1,001 A/B Tests?
Signific content on this topic
- Webinar: Learnings from starting experimentation team at SVT (Swedish)
- Article: How to score your hypothesis for experiments (Swedish)
- Webinar: How to do user centered experiments (Swedish)
- Webinar: Cold starting experimentation at Polestar (English)
Tech and statistics to enable experimentation
Sure, doing a few experiments is where you have to start, but to really get the flywheel going common barriers are often caused by technical and statistical aspects. This is something we’ve experienced first hand from companies like SVT, Spotify, Polestar and many more. Here you have talks and articles we have read, shared and discussed during our knowledge sharing sessions and in our internal slack through out last year.
Talks
- Talk by Erik Giertz (now at Signific) on how Spotify analyzes thousands of experiments
- Outstanding research challenges within experimentation by Martin Tingley (Netflix)
Articles / Posts
- Spotifys Experimentation platform
- How SVT uses data sprints to improve data quality
- A/B Experiment infra is AI Infra
- Data scientists biggest fear: Statistical theater
- Statistics: Are you Bayesian or Frequentist?
- IKEA.com — the problems with static content for A/B testing
- What is a growth engineer?
Signific content on this topic
- Article: How many users do you need to A/B test (Swedish)
- Article: Why use Bayesian methods for A/B testing (Swedish)
- Article: How Machine learning and Experimentation works together (Swedish)
- Webinar: Barriers to scale experimentation in your organization (English)
Reach out
Head of Sales Martina Fritjofsson: martina@signific.se | Linkedin
Co-founder Max Dyrhage: max@signific.se | Linkedin