The advent of large language models (LLMs) marks the most significant shift in the search experience since the rise of mobile technology.
Google’s SERPs are evolving beyond the traditional 10 blue links, introducing AI Overviews and other enhanced features. The full impact on brands and consumers is still unfolding
This article explores the hypothesis that the integration of LLMs into everyday life will fundamentally change how people search.
With LLMs capable of providing detailed answers, consumers are likely to move away from simple queries (e.g., “car insurance,” “running shoes”) toward more complex, conversational searches (e.g., “How much is car insurance for a 2022 Accord?”).
Hypothesis: The shift in consumer search behavior
Since the introduction of LLM models, consumers have started to shift their search patterns for items containing longer queries.
Methodology: Analyzing Google Ads data
Since search data from various LLMs, including Google’s Gemini and AI Overviews, isn’t available yet, I used search query data from Google Ads.
I pulled every search query from Jan. 1, 2022 to May 31, 2024, which resulted in 41 million search queries and over 83 million impressions.
I broke the data down into three buckets:
Pre-ChatGPT (Jan. 1, 2022–Nov. 30, 2022): No such thing yet as consumer-facing LLMs. Consumers would still be searching “traditionally.”
12 months Post-ChatGPT (Dec. 1, 2022–Dec. 31, 2023): During this data range, consumers are now exposed to LLMs and may have begun altering the way they interact with search.
2024 YTD (Jan. 1, 2024–May 31, 2024): Gemini launched on Dec. 6, 2023. This bucket helps keep the months clean, but now with Google’s product fully launched, this would be the most “advanced” they could be given the date range.
Results: How search term lengths and metrics have changed
After looking at 41 million search terms and their corresponding impression, click and cost data over the past 29 months, what can we conclude?
Search term length has increased since the launch of ChatGPT, but it has been slow. CPCs are up, and CTR is down. LLMs and AI Overviews seem to be driving fewer clicks to traditional ads as consumers searching for longer queries may find what they are looking for through the AI Overviews.
Let’s start by looking at how the data has changed over time by query length. Looking at the data from these three datasets, you can see some small shifts over time.
Most notably, there was a shift between keywords with 5-6 words and keywords with 7-8 words. Search terms with 7-8 words in them have nearly doubled since the launch of ChatGPT and have taken that volume from the 5-6 search term bucket. While the rest have stayed fairly consistent, only moving by <15 in either direction.
What could this mean? I would say that those consumers who had previously searched for slightly more complex terms have started to be even more specific with additional conversational modifiers to their queries.
Overall, we are still only talking about 10% of keywords with more than seven words. The majority still have less than four search terms in the query.
Now, let’s examine some of the throughput metrics and how they have changed over time. Starting with CPC, we can see a big spike in CPC costs over the last six months.
All search term buckets have felt this CPC increase, with the biggest jumps being in the 3-6 keyword length buckets. CPCs are up by 60% after being fairly flat over the last 18 months.
Something is certainly happening here, especially when you examine the click-through rate on the same data set.
The hypothesis on click-through rate with AI Overviews would be that ads get fewer clicks. So far, that seems to be true, with click-through rates down across the board. Again, this is most notable in the 5-7 search term length buckets.
This would align with the thought that the more detail someone enters into the query, the more likely Google surfaces an AI Overview due to the more defined query.
The search volume is there, as impressions have not decreased. However, it’s important to note that while CPCs have been rising, CTRs have declined over the past six months, coinciding with the rollout of the AI Overviews.
Action items: Adapting to the new search environment
Monitoring search term data
Begin by tracking your search term data. This allows you to understand how your specific product and keyword sets are being influenced and identify any shifts or trends that might indicate changes in consumer behavior.
Analyzing consumer behavior
If you notice a shift or evolution in how consumers are searching, it’s time to dig deeper. Analyze the data to uncover insights that can influence the types of landing pages, ad copy and content you create.
As consumers move from traditional search queries to more conversational searches, they provide more direct insights into their intent, which can significantly improve conversion rates.
Monitoring LLMs
We’ve started tracking various LLMs (such as ChatGPT, Perplexity and Gemini) to observe how frequently they display certain brands and how these results fluctuate daily.
For example, over a week, the presence score (how often a brand appears across different LLMs) can vary by as much as 20% daily.
Understanding and monitoring this data in these early stages can provide a competitive edge as the pace of change in search technology continues to accelerate.
Random finding
I had no idea that consumers search only with emojis. This is not something I had considered doing myself.
However, after looking through the search query results, I found many emoji-only searches.
The example above shows consumers looking for the movie “Teenage Mutant Ninja Turtles.” Google is smart enough to interpret and map these searches correctly.
Depending on your vertical and brand, this could be an untapped opportunity worth exploring.
