A Better Understanding of Personalized Search

Over the years, search results have become increasingly complex and that trend is likely to continue. The traditional model of 10 blue links and rank checking is no longer accurate as users are receiving results that are increasingly customized to them. As results are becoming more personalized, it’s valuable to better understand how personalized search results are being presented to users. A group of researchers at Northeastern University set out to answer this question and found some interesting results.

How Broad of an Impact Does Personalization Have?

By comparing an aggregate set of searches from logged in users against a control group, they found that 11.7% of results show differences due to personalization, but that this varies widely by search query and result ranking position. Of various factors tested, the two that had measurable impact were 1) being logged in with a Google account and 2) the IP address of the searching users.

The Carry-Over Effect

When measuring personalization, it’s important to eliminate background noise. One type of background noise is the carry-over effect. This is something you have likely experienced. When you perform a search and follow it with a subsequent search, the results of the second search is influenced by the first search. This is the carry-over effect. It is a style of personalization, based on recent search history, but it is not a consistent element of personalization because it times out.

Carry-Over Effect

Above is an example of the carry-over effect. Prior to this search, there was a search “urban outfitters in Hawaii” performed. The subsequent search shows a result relevant to Hawaii, followed by a locally relevant result for a location in Boston.

To determine the timeout period of the carry-over effect, a set of searches were repeated with increasing distance, in time, between the two searches.

Carry-Over Effect Time Graph

It was discovered that the carry-over effect expires after 10 minutes. At the 11 minute mark, the effect was no longer present.

Personalization by Rank Distribution

While the aggregate impact of personalization appears to be 11.7%, its impact is not consistent by rank position. When looking at variability by rank position, an interesting trend is exposed.

Search personalization by rank

The graph above shows variability by ranking position, comparing a set of logged in Amazon’s Mechanical Turk (AMT) users vs. a control group. The variability of the control group is the background noise and the difference between the two is attributed to personalization.

The top ranked URLs are less likely to change based off personalization, with most personalization occurring at the lower ranks.

Where Does Personalization Take Place?

Not only does personalization vary by rank position, but also it varies by query category. During the test, these 20 queries were found as the top 10 most and least personalized search queries.

Most personalized search queries

Looking at the terms that are personalized, as well as those that aren’t, a few trends start to appear. Results with high degrees of personalization include companies and politics. One of the factors driving personalization is localization of results, with company queries showing store locations relevant to the location of the user. Queries with the least amount of personalization include factual queries (what is) and health.

The graph below helps demonstrate these differences by query category.

Personalization change by category

In summary, the graph above shows that some query categories are personalized more than others. . For example, 82% of results for “what is” queries are identical, while only 43% of results for “gadgets” are identical. Overall, “politics” is the most personalized query category, followed by “places” and “gadgets.”

What Happens When Rank #1 Does Change?

Results consistently showed that URLs with top ranks saw less variability due to personalization. The raises the question, when does personalization change the #1 ranking and where does the replacement come from? The study found a number of interesting conclusions.

In summary:
• 73% of new rank 1 results originate from rank #2
• 93% of new rank 1 results come from the first page
• 82% of old rank 1 results remain somewhere on the first page

Looks like, even in these turbulent SEO times, ranking #1 still means something.

Leave a Comment