July 2, 2020
Growth

A Complete Guide to Growth Experimentation

Dan Kwarcinski
|

Look at growth as a mindset, not a playbook.

Growth is a combination of clean data infrastructure, excellent customer experiences, and rapid experimentation. Achieving sustainable growth therefore requires an understanding of all stages of the customer journey and how to leverage them. Whilst there are best-practice approaches that are applicable across businesses, there is no out-of-the-box solution, or gung-ho approach that will get you there.

When push comes to shove, growth is about enabling the right team, with the right culture and the right tools. It takes time and effort to do this, as you have to centralise processes, keep teams aligned and build a culture that values experimentation. In our experience, businesses who are in need of better growth methodology observe one of the following mentalities internally:

  • A siloed approach to teams. Sales is purely sales, marketing is purely marketing, and there is little collaboration across disciplines. In this set-up, companies compete with themselves internally, setting up internal KPIs where departments compete (for who achieves best sales, margins, best outcome).
  • Fear of failure. Where avoiding poor results is more important than success, leading them to traditional marketing and generic growth hacks, risk-averse choices and less meaningful takeaways.
Regardless of company size or industry, growth experimentation is held back by two common problems.

This is a natural product of bigger companies, as rigid structure and process is often necessary for order. But it can also appear in scale-ups too, who can become over-invested in some of the legacy processes they set up in their formative days. With this in mind, let’s talk about where there is some more specific room for improvement:

  1. Businesses often mistake A/B testing on their website for growth experiments. This is not the case majority of the time, as the majority of businesses don’t have sufficient traffic to do micro optimisations in a meaningful timeframe. Instead, they should focus on high-impact experiments.
  2. Businesses think experiments must be successful at all costs. Often, it even leads to the ‘shifting of goal-posts’ mid-campaign in order to show success; this minimises an organisation's opportunity for learning and improvement.
  3. Businesses think they need long planning cycles for one-off campaigns, with high production values. High cost campaigns lead to high production values which lead to higher ad spend to justify the investment—a vicious cycle.
  4. Businesses suffer paralysis by analysis. Afraid of making mistakes, employees pass decisions up the chain, slowing down the speed with which they can be made.
  5. Businesses have insufficient analytics infrastructure to measure test performance. The inability to measure the performance of experiments effectively slows down any experimentation culture.

Companies held back by these hurdles struggle to implement a build-measure-learn cycle that let’s them grow their understanding of customers, markets and trends at high speed. On one end, start-ups don’t have the luxury of relying on institution knowledge that has been created over years or even decades—to survive they need to learn much faster. On the other end, the business landscape for established corporates is changing rapidly—whether it’s marketing channels, the competitive environment, or consumer preferences, they need to stay on-top of these changes.

We believe three factors are critical to a business’ successful growth: Culture, Process, and Skills/Tools. 

Culture

A growth culture should exist across all areas of the business—from product and engineering through to sales and customer service. However, the growth team needs to have the authority and capability to prioritize and implement experiments across the business. Several building blocks can help with this.

Growth is everyone’s job. Growth is not only about customer acquisition, but also (or even mostly) involves customer retention, various stages of activation (e.g. from trial to paying customer and later to advocate), pricing strategy and of course product development. This means everyone can and should contribute their ideas. Within the growth team itself, all areas of the business should be represented as otherwise you may miss the critical insight that is obvious to one but not the other. The growth team can then prioritise, derive and run experiments from these ideas and coordinate implementation of validated.

Celebrate ‘failure’ as a learning opportunity. Fear of failure and being wrong stifles ideas and leads to teams shifting goalposts to demonstrate success, robbing the team and business of the ability to do better next time. In order to accomplish this, ‘framing’ is important. As long as a growth experiment generates knowledge it did not fail. To encourage this thinking we suggest to categorise experiments as ‘accepted’ and ‘rejected’ upon their conclusion instead of ‘success’ and ‘failure.'

Ask for forgiveness, not permission. Speed is fundamental as the faster you are able to validate or reject your hypotheses the faster you’ll generate the insight to drive further growth. One of the major barriers in larger organisations is red-tape and fear of making decisions without coverage from ‘above’ (this goes hand-in-hand with fear of failure). This slows down the experimentation process significantly. In order to address this, the growth team should have clearly defined authority—and testing budget without return expectations—within which they can run experiments.

Keep in mind: If you make 100 decisions a day and get only 50% right instead of making 10 decisions and getting 100% right you’re still moving at 5x the speed of your competitor. And this assumes you’re not even learning from those incorrect decisions.

Solve problems. Be creative. Collaborate. Growth hacking is about solving problems. Each experiment needs to minimise cost—in both time and money—and maximise the insight generated by the experiment. At the same time, identifying high-impact tests (more about this in the process section below) requires creativity and collaboration across team boundaries.

Building culture is a slow process. Buy-in and leading by example from senior management is critical in making it possible at all. At the same time constraining the culture shift to the growth team while the rest of the business continues on as usual can lead to counterproductive friction between teams. 

Processes

There are a number of different schools of thought around team structure and processes. We’ve found the following building blocks to work well.

A full-time growth team to manage the processes, infrastructure, testing backlog, implementation and analysis. This should cover growth, marketing, product, engineering  and analytics experts. Sales, customer service, operations, and other teams should be included in addition to their primary role. For these ‘part-time growth members,’ it is important they have sufficient time carved out of their primary role to actively contribute to the growth team.

A Kanban to outline and track your experiments: those in the backlog, planned, in-progress & completed. We like to use Asana or JIRA, but this works just as well in Trello or even Google Sheets when you’re starting out. There are some specialised solutions, but these are usually overkill. We’d generally recommend using the same todo/project management solution your teams are already using.

Within the Kanban each experiment should consist of six sections:

  • Hypothesis: What do you think will happen? A clear, short statement without justification or explanation. At the end of the experiment you’ll need to be able to accept or reject this hypothesis.
  • Rationale: Why do you think this? Provide an explanation of the hypotheses— what is the supporting material that makes you think this hypothesis is correct?
  • Implementation: Describes the quickest and cheapest way to test the hypothesis
  • Acceptance Criteria: The KPIs and threshold to be reached before the hypothesis is counted as accepted. These KPIs can be relative (e.g. an A/B test), or absolute (e.g. we need to achieve a certain CAC to consider this channel effective)
  • Results: The quantitative outcome of the experiment based on the KPIs defined in the acceptance criteria and whether the threshold was reached.
  • Learning: A qualitative interpretation of the result, key takeaways and new questions/hypotheses the experiment has resulted in.

In order to prioritize the experiments we recommend using a scoring model like ICE: Impact, Confidence, Effort. For each experiment, give a rating for these factors 1-10, average them, and you have your aggregate ICE score. This gives you a rough priority list from highest to lowest. It isn’t perfect, generally we believe it makes sense to split effort into two components: Effort to test and effort to scale, so you can separate your quick wins from your long-term projects. Something that may be easy to test manually can be much harder to implement across the business. 

Once an experiment has reached the ‘in-progress’ phase, your hypothesis, rationale, implementation and acceptance criteria should be locked and unchangeable. This structured approach will ensure your experiment is always run with a clear learning outcome and documents its approach outcomes and new questions, making it easier to reference old experiments and get new team members up to speed.

Experiments should be clearly time-bound, whether based on duration or KPI (e.g. number of conversions). This ensures experiments don’t run forever and gives you a clear point at which to evaluate the experiment (whether via significance testing or Bayes).

Sprints are also very helpful in focusing the team on high testing velocity. However, the sprint duration is highly dependent on how quickly you are able to generate a sufficiently large sample for meaningful statistical analysis. Some experiments may have to run through multiple sprints, but your sprint duration should be set to allow the majority of your experiments to complete within a single sprint. At the end of each sprint, the team should come back together for a retrospective to discuss not only the experiment outcome and learnings, but also what improvements can be made to the experiment and implementation processes.

Skills and Tools

Finally, running successful growth experiments requires the ability to measure experiment results accurately. This is best achieved with a solid analytics infrastructure.

Growth teams need to combine a wide array of skill sets, from marketing, copy-writing and psychology to analytics and engineering. These skills don’t need to be combined at expert level in a single individual, but all team members should have a good technical and analytical understanding.

Many growth experiments consist of combining existing systems in novel ways to create value—take for example the early-days Airbnb example of posting their listings on Craigslist. This required ‘hacking’ together a solution that was able to convert Airbnb’s listings into a craigslist post and then post it on the platform. In order to be truly creative, growth teams need to understand what is technically possible—and what 3rd party solutions that are out there.

These golden ticket growth hacks are only possible with a robust growth process, culture and infrastructure. They result from shifting away from the traditional approach, moving away from campaign-driven promotions, and looking towards layered, automated strategies. Throw in some hustle, and at the very least you are sure to achieve fantastic, sustainable, long-term growth. Throw in some out-of-the-box thinking, good timing and some good-luck for measure, you are in the perfect position to pull off your one-in-a-million growth hack.

Happy hacking!

You May Also Like

How to set intentions that energize you 2

Consumers are inundated with ads, so it’s vital that your ad catches the eye and immediately grabs interest.

May 22, 2020

Motivational sayings ten great ones

Consumers are inundated with ads, so it’s vital that your ad catches the eye and immediately grabs interest.

May 22, 2020

Believing is the absence of doubt 2

Consumers are inundated with ads, so it’s vital that your ad catches the eye and immediately grabs interest.

May 22, 2020

Master Your Growth Trajectory. Trust Your Navigator.

Data-driven growth and analytics consulting for scale-ups & established corporates.

Let's Chat
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.