You might assume that it’s the stuff of science fiction…
Runaway robots taking our poorly-specified instructions ‘too literally’ – with predictably disastrous results.
We just need to be very clear about what we do and don’t want our machines to do, and take proper care not to leave any ‘Asimovian’ loopholes open – right?
Well 🤔
One thing I’ve taken from a recent shallow-dive into the topic, is that it’s really not so simple…
The AI alignment problem comes in several varieties… In this post, we’ll take a look at three, which have direct implications for our relationship with Google Ads automation.
Understanding these pitfalls sets us up a little better to decide when to make use of marketing automation… how… and with how much faith…
Tim Harford: How to Make the World Add Up
1 WHAT TO OPTIMISE FOR?
First, telling the algorithm what you really want it to do isn’t necessarily straightforward.
In our case – for smart bidding algorithms – the primary guidance mechanism is conversions, to mark the clicks we want, as opposed to the clicks we don’t.
We could see conversions as a ‘reward function’ for the algorithm – telling it what to ‘value’ according to our own priorities.
Conversions are coarse-grained, though… a crude measure of value… so we graduate to conversion value in order to differentiate and prioritise between those ‘positive clicks’ (something that’s well worth doing in lead gen with estimated values, as well as its obvious application in ecommerce)
But trouble awaits us here.
The first issue is that conversion value tends to have a dual role in our accounts.
Usually we want and expect conversion value to represent actual gross revenue, and there are good reasons to use it in this role…
It’s intuitive to clients, it’s a valuable indicator to see what’s coming in for cash flow purposes and immediate term ROAS.
But then conversion value becomes compromised in its role as the algorithm’s perfect guide to value.
Our perfect evaluation of an action would also take account of future value… the effects of product returns, returning custom and LTV, customer referrals…
This is impossible to do accurately (we know that IRL attribution is messy and largely hidden, so one inevitable difficulty in stating the value of an action is that we can’t really know it ourselves…)
But we could try.
We could tinker with the conversion value figure to take these into account… but then conversion value no longer equals revenue.
Second, to guide the algorithm more faithfully to our priorities, we’ll also want to apply a value to those ‘micro conversions’ – newsletter signup etc – which hold a value to us, but don’t carry any tangible revenue.
Those are actions whose value we would like the smart bidding algorithm to acknowledge. But again, we’re then forced to choose between conversion value-as-revenue and conversion value as a well-defined reward function.
This all limits how accurately we can specify our values – and therefore align the values incorporated by the algorithms.
2 HOW TO OPTIMISE FOR IT?
Then, assuming that we can clearly state our values – the question arises of HOW the algorithm goes about optimising for them.
"Any measure that becomes a target loses its value as a measure."
There are many sensational allegories of the ‘one-track-minded’ optimiser in cautionary tales about AI safety.
The best-known is Nick Bostrum’s paperclip maximiser – but there are plenty of real-world examples of AIs achieving their given objective the ‘wrong way’.
Victoria Krakovna of Deepmind has compiled a list of such ‘specification gaming’ examples, in each of which an AI generated “a solution that literally satisfies the stated objective but fails to solve the problem according to the human designer’s intent”.
(See also Robert Miles’ excellent video detailing nine of the most interesting examples).
How does this relate to us in PPC?
Take our well-intended conversion value – complete with appropriate adjustments for the long-term view. The algorithm – we hope – will seek to find ways to maximise that value.
If those ways are novel – such that we wouldn’t have easily devised them with our own observations and manual tools – so much the better…
And that is one of the reasons why – in lead generation – it’s so hard to shake the ‘wrong kind’ of enquiry.
If the algorithm finds an easy source of conversions – or conversion value – it will exploit it as far as it can. And as with so much in life, the easier way is often less rewarding.
For example, if there’s a source of job-seekers ready to fill in your form for employment enquiries rather than for its intended purpose, the algorithm will gladly keep seeking them out, and paying (with your money) for those users to visit your site.
Not because it’s trying to deliver lower-value conversions… but because it’s trying – as hard as it can* – to generate as much of exactly what you say you want, as possible.
but then…
3 *HOW HARD IS IT REALLY TRYING?
And here’s the third issue… to what extent is your conversion value really the ‘ultimate good’ in the algorithm’s eyes?
In the end, maximising advertiser-specified value isn’t the only thing that matters to Google – or, therefore, Google’s algorithms. Not by a long shot…
There’s a wide range of attitudes to this topic (and I’m certainly not the most cynical PPCer you’ll find) but it’s clear that our priorities as advertisers and those of Google aren’t identical.
We shouldn’t expect them to be.
We ideally want to spend as much as it’s profitable to spend. Google ideally wants us to spend as much as we are willing to spend.
Consider how seriously (or lightly) the algorithms take ROAS targets.
Of course, it would be unrealistic to expect a ROAS target to be treated as a ‘hard limit’. That would be virtually impossible, given that Google can’t know in advance how much – if any – value will result from a given click. But it is in Google’s hands to decide how much leeway (or liberty) the algorithm takes with it.
• What will it sacrifice to reach the target…
• (or to get at least within X% of it over time period Y… )
• And what values should X and Y have…
• If it’s failing to reach the target (failing being defined how…)
• then how long will it give itself before it becomes more conservative with bidding and with overall spend…
• and how radically / at what pace…
• How much and for how long will it spend while failing to meet your target (by what %…) before it stops spending altogether?
These and many questions like them are all choices that Google makes, and all could be made
• more conservatively (prioritising short-term cost-effectiveness for the advertiser)
or
• more aggressively (upweighting spend / prolonging ‘hopeful’ experimentation / making the assumption that performance ‘close to’ target is ‘good enough’)
We will never know just how Google calibrates all of this (and I for one wouldn’t understand the answers if I asked…)
But we know for sure that our campaigns are allowed to keep on spending quite a bit, for quite some time, at quite some way below target.
It’s not obvious where the balance should lie.
We shouldn’t wish for an algorithm to play the penny-pinching accountant who won’t endorse any plan of action… but we don’t want it to play the incorrigible maverick either…
and for those determining the ideal path between those extremes, there is (at the least…) plenty of incentive for ‘motivated reasoning’.
So if algorithms don’t do exactly what you want, it may be a feature rather than a bug. They are – in part – optimising for outcomes that are not entirely aligned with yours, however well you specify what yours are.
So we have our triple alignment problem with smart bidding:
1) The difficulty in telling the algorithm precisely what you really want (the conversion value issue).
2) The problem of trusting the algorithm to maximise your stated value in the right way (admittedly, this can be seen as a subset of problem #1. State the value perfectly and #2 is solved… but AI has a knack of uncovering those sneaky solutions that no-one foresaw to take into account).
3) How fully the algorithm takes on your priorities as its own. i.e. how it weights your values versus Google’s values where the two don’t fully align.
Keep all of these in mind when setting goals, and conversion values, and keep an eye on just how (and how hard…) the algorithms are trying to optimise for them.
