The past year has been extraordinary, and hope is now rising that the worst effects of the pandemic will soon be behind us.
When greater perspective becomes possible, there will be all sorts of lessons to learn -many of them far more important than those offered below (and I acknowledge the imbalance of comparing a devastating pandemic to our relatively sheltered profession of Paid Search…)
But – partly for my own sanity – I have been noting topics that shed some interesting light on common issues we face in the world of PPC.
Here are four of them.
Lesson #1 – Don’t zoom in too far
If you’ve been following the course of pandemic closely, you may be very familiar with charts showing case numbers detected by day.
As the numbers can jump about considerably from day to day, the best of these charts come with a smoothing line, clarifying the longer-term trend.
This chart from worldometers.info for example, takes some of the interpretation off our plate by showing a three-day-average trend line.
While smoothing out some of the day-to-day variability, this trend line shows a marked weekly cycle, whether it’s on the way up or down. Namely, the number is dipping around Monday/Tuesday of each week.
Now overlay the 7-day trend line, and the longer-term pattern is clearer (a case where viewing data in ‘lower resolution’ actually gives a truer picture).
In this case, we happen to know that the weekly variability comes from a consistent delay in the reporting of data over weekends…
But even when short-term variability is genuine, it is misleading to take it for a ‘real change’. You may have experienced this in the form of questions like ‘why are conversions down this morning?’… usually the result of concern caused by zooming in too far.
There are subtler pitfalls with analysis by date range too.
In this tweet, Dan Barker calls out the misleading claim that the data (below) showed workplace visits rising at a certain point during lockdown – based on a classic ‘zoom-level’ mistake.
Before calling a rise or fall, look at the ‘bigger picture’. A 7-day trend line would have saved the original Tweeter from his mistake.
Lesson #2 – Calibrate risk tolerance to the context
In August 2020, multiple research teams from around the world were in the tantalising position of believing that they had developed a safe, effective vaccination against COVID-19.
While almost all governments waited for the painstaking process of proving beyond any reasonable doubt that a vaccine was suitable for use, one bucked the trend.
On 11th August, months before its phase 3 trials had been completed, the Sputnik V vaccine developed by the Gamaleya institute was declared safe and effective, and approved for use by the Russian government.
Why declare a vaccine safe and effective after a trial of 76 people, while all other governments continued to follow the accepted procedure of awaiting results from phase 3 trials, involving 1000s?
It’s not necessarily based on a different conclusion about the probability of success… it’s a calculation made by different actors, with different weightings on the costs and benefits of success, failure and inaction.
If those making the decision believed that they and/or Russia had a disproportionate amount to gain from making the call early, then the risk of backfire may well have been worth taking for them, while it wasn’t for other countries, looking to other vaccines.
So it is with PPC.
We often have to consider how much data we need, to justify decisions.
As with Sputnik V vs the Oxford or Pfizer vaccines, the answer depends on the context.
The more badly and quickly we need or want an improvement, the more risk we can tolerate of getting it wrong… and the lower we should set our confidence threshold.
Risk tolerance should be calibrated to the context, just as performance targets themselves are.
Lesson #3 – Doubling up vs doubling down
When vaccinations became more widely available, from late 2020, the question arose of how to assign our resources to make the most effective use of them.
One of the key questions in this was how to balance the need for each vaccinee to receive (in most cases) two injections, vs the need to spread some vaccine benefit across the greatest number of people.
i.e. How long to delay the second jab, in favour of maximising first-jab coverage.
Of course, the answer should depend largely on the data… how much benefit does one jab offer vs two? For how long? How much slower will the population coverage be if we stick rigidly to a two-week interval?
But it also depends on prioritisation, and on the level of tolerance both of the well-understood risks, and of uncertainty.
It is interesting that policy-makers in different countries reached quite substantially different conclusions on this question*.
This dilemma – when and how far to concentrate one’s resources in making sure of capturing those most in reach (lower funnel first) vs spreading coverage to make some impact across the largest number (upper funnel expansion) – is one that we face often.
To see a similar question played out among some of the finest minds, when the stakes are life and death (x1000s) is a sobering and powerful lens to view it through. And their reasoning and conclusions are worth paying attention to.
*It is likely that – as with lesson #2 above – politics played a role in some of the conclusions reached. This is also a situation well-known to PPC managers… as unfortunately most of us have taken actions and seen (and made) decisions with more political than actual performance benefit.
Lesson #4 – The power of simplicity
Another – equally controversial – aspect of this question, has been how to prioritise different groups of people for vaccination.
Many groups have had very reasonable claims to be in particularly urgent need of vaccination, both from the heightened risk they face, and the importance of protecting their work from disruption (teachers, police, and other key workers being the most obvious cases).
The problem with making this kind of prioritisation, is the complexity it introduces – especially given a large operation that crucially needs to run at full tilt.
That complexity would add more steps in the process; more work required to fulfil it successfully; more potential wrong turns in the journey from factory to arm, and it would be certain to slow the rollout down as a result.
For this reason, the decision was made to continue prioritising the rollout along just one variable: age.
It was a case of prioritising the good over (attempts at) the perfect. While the decision was debatable (and hotly debated) this is a useful principle to learn from in our work. Fine tuning may be ideal in theory, but in practice the benefits of simplicity are often greater.
Most PPC managers have also experienced these kinds of competing priorities from clients…
Client: 1, Bring us as many conversions as possible
Client: 2, Keep CPA under £25
PPCer: We’ll modify #1 with this criterion. OK
Client: 3, Keep a reasonable balance of conversions from each location
PPCer: This will compromise #1 and #2… but OK
Client: 4, Please keep overall impression share above 70%
With each new layer of prioritisation, the essential work of maximising conversions at a good CPA is compromised further…
These kinds of conflicting aims are one of the areas where it is most important to push back against client requests.
So these are the four lessons for PPC:
• Watch out for pitfalls in pattern analysis
• Be flexible about how much data you require for action
• Reflect on growth vs consolidation, and
• Know when to keep it simple
Here’s hoping the next batch of insights comes from a happier source.