However, the advantages brought by the modern way of tracking users have always outweighed the drawbacks of missing a couple of users or some of their interactions. After all, server logs were only able to provide you with the user’s IP Address, User-Agent, date and time, as well as the file that had been requested – not of much value to a marketer in 2015. Especially when compared to the rich data that web analytics tools like Google Analytics provide you with.
The setup of our experiment
So, how do ad-blockers affect web analytics data? In order to answer that question, we set up an experiment with our own analytics data. Our hypothesis for this experiment is that the release of iOS 9 had a significant effect on the number of recorded pageviews in Google Analytics. This means that the difference of recorded pageviews by Google Analytics is significantly decreasing compared to the number recorded by the server log after the release.
In order to test the hypothesis, we compared our data from Google Analytics with the actual server logfile. For this comparison, we took both mobile and desktop users into consideration. We monitored if there had been a change in the difference between these two data sets, when comparing with a time span of 3 weeks before and 3 weeks after the release of iOS 9.
Here’s how we prepared the two data sets in order to conduct our analysis:
If you would like to reconstruct our experiment with your own data, you can find more detailed instructions in the appendix of this blog post.
The effect of ad-blockers on web analytics data
As explained, server logs will always show more traffic than Google Analytics. Hence we display this difference as a % of page views that were captured on the server log, but not in Google Analytics:
First and foremost, our hypothesis that the release of iOS 9 led to a large increase in blocked analytics data cannot be confirmed statistically. In more technical terms, the difference of Safari before and after was t(29.96) = 1.78, p = 0.09. Therefore, the hypothesis cannot be verified.
However, there is already a notable and statistically significant difference between Safari and Chrome in our benchmark period from before the launch of iOS9: t(33.18) = 2.38, p < 0.05.
Thus, the data suggests that Safari users already blocked the Google analytics code before the release of the iOS 9 much more frequently than Chrome users.
What can we learn from these numbers? First of all, it may be more common for Safari users to block Google Analytics. Additionally, it may also be that Safari is not able to execute the Analytics snippet in general, at least when compared to Chrome.
At a first glance, it seems that there indeed is an effect of the rise of ad-blockers on analytics data: Even before the release of iOS 9, many users downloaded ad-blockers in the app store from 3rd party providers. We suggest at least part of the difference between Chrome users and Safari users can be explained by this fact. A weighted average of the two browsers leads us to believe that about 8% of our website users have installed an ad-blocker that blocks Google Analytics.
However, one limitation of this study is the time interval used, which was limited to only three weeks before and after the launch. After detecting an already high percentage of blocked calls to Google Analytics (which was surprising) before the release of iOS 9, the results definitely warrant a follow up study with a time interval that looks back at least one year, back to when ad-blockers were not a very widespread topic. Moreover, the analysis would optimally consist of a bigger data set consisting of websites from different industries and markets as well.
What marketers can do
It’s important to note that there will always be a difference between what we can see in Google Analytics, in our server logfiles, and what actually happens on our website. Still, there are several simple measures webmasters and marketers can take in order to raise the accuracy of their data and analysis:
- Make sure your Analytics Code is placed and triggered correctly
- Check your Google Analytics settings and filters
- Keep your website code healthy
- Know what you don’t know: make regular checks to ensure the quality of your data
- Analyze your available data sets using metrics that outline the underlying business value instead of measuring the size of your audience (Link)
If the rise in plugin downloads seen in the last few months has ignited a trend that will eventually result in a majority of users blocking web analytics tools however, the industry will likely find ways to still gather accurate data.
However, the debate concerning what user data can and should be tracked and aggregated should be held first. This discussion needs to address questions like: How can publishers still finance their services through ads? How can marketers target and tailor relevant ads to the appropriate audience? And how can the industry achieve all of this, without infringing on the user’s privacy or user experience?
In the worst case scenario, marketers and publishers will start collecting data in even more personal ways (keyword «browser fingerprinting»), while simultaneously bombing users with flashy and irrelevant ads that slow down the site and negatively affect user experience. As a result, users will then either stop being economically viable by using ad-blockers (bad for the entire industry) or switch to a competitor (bad for the individual publisher).
To counteract these developments, marketers and publishers are advised to use the already available data in a meaningful way, to create more relevant advertising that takes both the user’s needs and the context into account (more here).
Appendix: How to reconstruct our experiment
For our experiment, we removed some of the pages that are irrelevant or inconsistently tracked from both data sets.
The list includes:
We also considered the following: There were many requests by bots still in our Server Log, some in plain sight, others deceivingly hiding their identity (more on spoofing).
Since we were using an unfiltered View in Google Analytics (which had «Bot Filtering» deactivated as well), some, but not all Bots, may have even shown up in our Google Analytics data set. Since we were primarily interested in the changes in traffic coming from Safari, we filtered our Google Analytics data set for the Safari versions 5-9, and filtered our server log file for the user agent of these versions accordingly.
Since we feared a systematic bias in Safari data, we wanted to compare it to the changes to another browser in the same time frame, in order to have a valid benchmark. We chose to use Google Chrome (versions 44.x, 45.x), since it was the most popular browser version used on our website within the defined timespan.