We’re constantly seeking ways to optimize our PPC campaigns and maximize impact.
Testing is critical to this process, but traditional methods like A/B tests, incrementality evaluations and geo experiments often have significant limitations.
Large data requirements, extensive planning and reliance on ad platform functionality can make it challenging to get clear, reliable insights.
When these constraints come into play, we may find ourselves making important decisions based on incomplete or misleading data – wasting budget or missing out on scaling opportunities.
This article explores a powerful but often overlooked testing technique: causal impact studies. Discover how they work, when to use them and how they can transform your approach to optimization and decision-making.
What are causal impact studies?
Causal impact studies accurately measure the true effects of changes in your campaigns by estimating a counterfactual (i.e., What would have happened without the implemented change?).
Understanding the difference between correlation and causation is crucial.
For example, if the number of Aperol Spritzes I drink in summer increases alongside my complaints about the heat, one isn’t causing the other; both are influenced by the sun being out more.
Causal impact studies help you determine whether a change in your paid media campaigns directly caused a shift in a specific KPI or if that shift would have occurred anyway.
The study takes a set of observed data and estimates this counterfactual scenario – essentially asking what would have happened without the change.
The difference between this counterfactual data and the observed data reveals the causal effect of your intervention.
Dig deeper: 3 steps for effective PPC reporting and analysis
How do they work?
In an A/B test, two groups of users are involved: one exposed to a test condition and the other under control conditions.
You can observe the outcomes for both groups – what happens with the test condition and what happens without any changes.
However, you cannot see the outcome for the test group if no changes had been made, nor can you determine how the control group would have performed if the test condition had been applied.
In a causal impact study, the goal is to estimate the outcome for the test group if no changes were made (in this diagram, test group 2):
To build this estimate, you need to find another data set from the same time period that is correlated with your KPI but not affected by the campaign change. This could be data from a similar campaign that wasn’t impacted by the test or something broader like brand searches or overall category demand.
When you run the model on these two data sets – your observed data and the correlated data set – it will first examine the relationship between them. Then, it will estimate what would have happened to the observed data if it had followed that relationship beyond the point of implementation.
If this estimate matches your observed data, it indicates that your change had no impact. However, if the estimate shows significantly different results, you can identify a meaningful causal effect.
The study runs many iterations of the model to generate a distribution of estimated results from which a confidence interval can be built.
To validate your results, you could always go back to your A/B tests.
If you run an A/B test using the same test conditions, does your control group come out with the same data trend as your counterfactual estimate? If so, then you can confidently say that your model is accurate.
Full information and implementation guides on the package created by Kay H. Brodersen and Alain Hauser can be found on GitHub. I also highly recommend watching Brodersen’s talk on the subject on YouTube.
Dig deeper: Advanced analytics techniques to measure PPC
When to use causal impact studies
When is it appropriate to use a causal impact study? To answer this, consider the following pros and cons.
Pros
Clear understanding: You can gain a clear insight into the impact of a specific change.
Flexibility: There is flexibility in the test setup, and you have control over confounding variables, such as seasonality, as long as you choose the right data set for comparison.
Retrospective analysis: These tests can be conducted in retrospect. If an A/B test was not possible or wasn’t implemented, you can still analyze a past change to determine whether it had an impact or if other factors influenced the results.
Cons
Technical expertise required: Implementing the test requires a certain degree of technical know-how. While I have support from my team at Google and my data solutions team, not everyone has that luxury.
Resource intensive: If a hypothesis can be adequately answered using an A/B test, that approach is generally easier to implement and less resource-heavy.
Data dependency: The strength of the model heavily depends on the data set you use to train it. If you select a data set that does not closely relate to your test KPI, your model may not be accurate, leading to unmeaningful results.
If you have the technical ability (or the willingness to learn), an appropriate data set for comparison, and your hypothesis cannot be answered by a simpler test like A/B, then a causal impact study is a valuable tool to accurately determine the true impact of an intervention.
For example, my team is currently running two analyses for a client: one where we turned off their GDN activity and reallocated that budget to Demand Generation and another in which we’re testing the impact of adding assets back into a feed-only Performance Max campaign. The causal impact studies will help us determine whether these changes significantly affected our overall Google Ads performance.
My next test?
Validating whether my Aperol Spritz intake is caused by the sun being out more or whether it has something to do with the increasing length of my to-do list!
Measuring true campaign effectiveness with causal impact studies
Causal impact studies are a powerful tool for paid media marketers seeking to understand the true effects of their campaign changes.
By accurately estimating counterfactual scenarios, these studies help you discern whether observed outcomes result from your actions or other factors.
While they require some technical expertise and careful data selection, their ability to provide clear insights makes them invaluable for optimizing marketing strategies.
Embracing causal impact studies can lead to more informed decisions and ultimately improve the effectiveness of your campaigns.
Dig deeper: How to evolve your PPC measurement strategy for a privacy-first future
