Sign in Subscribe

Marketing measurement is in crisis

Marketing measurement is in crisis

The rapid adoption of AI by major advertising platforms has created a unique paradox: the better a platform gets at optimising the ads, the less we know about how it is done. We are trading visibility for efficiency. We are handing over the keys to autopilot systems that refuse to tell us the route it took. The information and knowledge loss is staggering. We no longer see the clear link between a specific creative lever, a granular audience segment and the final sale. The “black box” is no longer a caution. But it has rather become a standard operating procedure.

The first time I felt I was losing control and traded it to gain time was when Google launched Broad Match Modifier in 2013 for Google AdWords. While this supported the philosophy of doing more with less, it came at the cost of learned knowledge. Fast forward to today’s discussions with the new, young and next generation of marketers, I’m often presented with this remark: Trust the algorithms to deliver the business goals. We have officially stopped asking how these “conveniences” deliver our business goals.

While advancements have been made in how marketing is operated and executed, measuring it is still trailing behind. So, if you are using traditional measurement methods or techniques to judge AI-driven performances, you are essentially measuring a ghost.

How does AI optimisation actually work? (The Black Box)

To understand the measurement gap, we must first look at what’s happening inside the engine.

AI optimisation isn’t all about “bidding smarter”. It is a fundamental restructuring of how media is bought and delivered. Platforms like Meta (Advantage+), Google (PMax), and TikTok (Smart Performance Campaigns) are making millions of micro-decisions per second that no human trader could ever replicate. Here are some of the real-world applications of AI optimisations where a marketer has no means to review logs, records or get information on how these things happen:

  • Creative Optimisation: The AI takes your images, videos and copy, and dynamically reassembles them. It might show a different headline or variations of video to User A than to User B. 
  • Audience Expansion: You might set a target audience, but the AI is actively ignoring it when it sees a better opportunity to deliver your business goal. It finds "lookalikes" based on signals you can’t see, browsing history, ad interactions, dwell time or predicted LTV.
  • Location Fluidity: It’s not just "Target New York or Paris". The AI optimises based on where the conversion probability is highest at that exact second, shifting budget between postal codes or even specific retailer proximities dynamically.
  • Placement & Format: The AI decides if your ad serves on a Feed, a Reel, a Search result, or a partner network site. It chases the cheapest inventory that drives the conversion event, regardless of brand safety or context (unless strictly guard-railed).

Time-of-Day: Rigid schedules are gone. The AI might spend 80% of your daily budget in a 2-hour window if its predictive model spots a trend.

The result? Performance usually improves. But the "Why?" disappears.

The "Good Old Days" of Pre-AI Optimisation

It feels strange to call the pre-AI era "simple," considering we were already battling new features such as Google's BMM, Meta's DABA or iOS14 privacy changes and GDPR regulations. But compared to today, we had a luxury that is now extinct: Granularity.

In the pre-AI optimisation world, Marketing Mix Modeling (MMM) was a robust source of truth because the inputs were stable.

  • Deterministic Control: If we ran a campaign targeting "Females, 25-34, Interest in handbags," we knew exactly who saw the ad.
  • Trackable Creative Diffusion: We knew that Creative A ran in Placement B. We could export reports showing exactly how much budget went to specific assets.
  • Stable Variables: Campaigns didn’t mutate daily. If we set a budget for a specific region, it stayed there.

Even with signal loss from Apple’s ATT (App Tracking Transparency), MMM worked well because we could still aggregate data at a meaningful level. We could regress sales against "Instagram - Prospecting - Static Image." The privacy laws hid the user ID, but they didn't hide the tactic.

We had the granular data required to build models that explained variance. We could look at a lift in sales and confidently say, "That was the result of the aggressive push in the Shanghai geo-market using the 'Discount' creative message."

The New Reality: MMM in the AI World

Today, you feed the same data to an identical MMM solution, and what do you get? Noise

Because AI optimisation handles so many variables dynamically, the "inputs" for your MMM have collapsed. You cannot model what you cannot track.

  • Loss of Creative Nuance: You can no longer model the impact of "Video vs. Static" easily because the AI might have served the static image as a slideshow in a video placement. The line is blurred.
  • Audience Homogenisation: Since AI broadens audiences automatically, "Prospecting" and "Retargeting" are often mixed into a single bucket. You can’t build a model to tell you if your Retargeting budget is efficient because the platform likely spent it on Prospecting to find cheaper conversions.
  • Constrained to Big-picture Thinking: Modern MMM is forced to zoom out. We are limited to measuring at the Channel (e.g., Social), Platform (e.g., TikTok), or Timeframe level.

We are back to answering basic questions like "Did spending more on Google help?" rather than sophisticated ones like "Did the dynamic creative optimisation in the Northeast region drive incremental lift?"

The granularity that allowed us to optimise tactically is now gone.

How Marketers Must Navigate This Complexity

So, do we give up? Do we just trust the algorithm and hope for the best? Absolutely not.

I believe that we need to evolve. We need to stop trying to measure AI with tools built for manual buying. Here is how I believe we should navigate this new complexity:

Shift from "Tactical" to "Strategic" Measurement:

Stop trying to measure which button colour worked. The AI does that now (it measures, optimises and delivers). Our job now is to measure the Strategy. Measure the impact of Creative Concepts rather than individual assets. Tag campaigns by "Emotional Hook" vs. "Rational Benefit" and let the MMM measure those high-level themes. Focus on scenario planning and strategy optimisation.

Embrace Incrementality Testing:

Since MMM is losing granularity, you must validate the AI’s claims with lift studies. Run as many geo-lift tests. Turn off the AI in one region and see what happens to the baseline. If the platform claims a ROAS of 15, but your order management system doesn’t show that, the AI is cannibalising organic traffic. Incrementality is the only remaining lie detector left.

Feed the AI Better Data:

The AI is hungry for more data signals. If you limit it, it hallucinates. Invest heavily in Conversions API (CAPI) and server-side tracking. The better the business data you feed the AI, the better it optimises toward actual revenue rather than proxy metrics like clicks or add-to-carts.

Context is King:

I’ve said it before: Quantitative data fills the painting with colours; qualitative data narrates the story of the painting. Leverage AI compute and use human insights to answer strategic questions and make critical optimisation decisions. The machine is great at delivering results, but it is terrible at explaining the results. That is still our job.

We are in a transition period. The old models are breaking down and the new ones are still being built. But those who can balance the efficiency of AI with the rigour of strategic measurement will win.

What is your team doing to validate the performance of "Black Box" campaigns? Are you seeing a disconnect between platform metrics and real revenue?