Make Google Analytics stats more accurate by watching X-PURPOSE and X-MOZ headers

I just found a video on YouTube that shows the two things I'm talking about http://www.youtube.com/watch?v=XnX9bbg1dYU

Notice when the user clicks the new tab button the browser displays thumbnails of websites the user has viewed in the past. The browser keeps these thumbnails up to date so they display the latest content on the site by regularly visiting the website in the background (without the users knowledge). This requires the browser to render the entire page including the Google Analytics code but the user isn't actually visiting the site as this is happening in the background.

Then watch what happens as the user starts typing a URL, the browser tries to guess the URL being typed and loads that page before the user hits enter. The problem is it will often guess the wrong URL and load a website that the user didn't actually want to go to. Lets use this issues URL as an example : http://drupal.org/node/1096478

The user starts to type : drup
The browser guesses and loads : drupal.com
The user continues to type : drupal.org
The browser loads : drupal.org
The user continues to type : drupal.org/nod
The browser loads : drupal.org/nod and gets a 404 error
The user types : drupal.org/node
The browser loads : drupal.org/node as this page actually exists
The user types : drupal.org/node/1
The browser loads : drupal.org/node/1 as this page exists
The user types : drupal.org/node/10
The browser loads : drupal.org/node/10
The user types : drupal.org/node/109
The browser loads : drupal.org/node/109
...
And so on until the browser finally loads http://drupal.org/node/1096478

This is a bit of an exaggeration because if the user typed fast enough the browser wouldn't have time to load all of these pages but it'd get a few loads in before the user finished typing. And as you can see above the user didn't want to go to many of these pages but they're now displayed in Google Analytics with a high bounce rate.

As I mentioned in my first post the browser sends "X-Purpose: preview" or "X-MOZ: prefetech" when it makes these preview requests to the server.

@hass if you need any more information or would like me to show you how it works and how the headers are sent I can setup screen sharing.

Yeah, it's really rude sending all this extra traffic, but on the other hand I turned Instant on in my browser and find it really useful :| catch 22. I don't believe Google Chrome currently turns this feature on by default but it wouldn't surprise if they turn it on in the future so it'd be good to prepare.

Google does have a page about blocking the Instant feature but it only works per browser session.

While searching for some "official" information for these headers I turned up an interesting Mozilla article about specifying links that a browser should prefetch. Of course this is what we do when we want to prefetch images so it certainly isn't a new concept as this also adds the "X-MOZ: prefetech" header.

In the Chromium issue queue there isn't much mentioned:
https://code.google.com/p/chromium/issues/list?can=1&q=%22X-Purpose%22&c...

I really can't find much official information about these headers but I can certainly see it in my server logs. To add the headers to your Apache logs you'll need to setup a custom log format like the following:

# LogFormat reference can be found at http://httpd.apache.org/docs/2.0/mod/mod_log_config.html 
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" \"%{X-Purpose}i\" \"%{X-Moz}i\"" combined

@hass Sorry I should of elaborated, if you block the headers your blocking the thumbnails as well!!! This is obviously a big negative as the thumbnails can be really useful for users, so I think most sites will want to keep Instant and just stop Google Analytics from loading when "X-Purpose: preview" and "X-MOZ: prefetech" are sent.

I'm interested to know if you think this is in scope for the GA module? As I don't see any other use for this functionality outside of the GA module. Other analytics modules might like the functionality but I don't think it warrants a separate module as the code sharing would be minimal. Yet I think its a feature everyone with the GA module needs by default.

Yeah I totally agree, it may happen once they think it through. What they need is a separate header for Instant and Thumbnails so you can block one and not the other.

Lets keep it open because we need to work out a solution, I'll look into it soon and post a patch.

Sorry I should of read back through the conversation, I just wanted to clarify our conversation so far as I'm a little confused.

Just wanted to reply to :

Safari 5.0.4: I have just traced with Wireshark and it does not send the X-PURPOSE header at all if you reload the TOP SITES tab...

Chrome 10.x: I see nothing helpful (except a request to /dictionary/js/api_loader.js) in the network trace if the browser is opened first and presents me the thumbnails. Additionally it seems to be as I guessed first. The images are locally cached. If you reload the thumbnail page you see nothing in the network trace. I'm not sure in what situation the images are generated nor how I'm able to rebuild them.

Yes, these are cached so refreshing the page doesn't update them, however at regular intervals in the background Safari and I presume Chrome updates this cache to keep the thumbnails fresh. When safari updates them it sends the "X-Purpose: Preview" header.

Additionally I have read somewhere that Google only sends a HEAD while instant is active to the server to check if the URL exists and it gets an 200 OK back or not.

I have Google Chrome Instant on in my browser and sometimes when I start typing URLs it displays the site before I even finish typing, sometimes it displays the wrong page because I haven't typed the full URL and it is definitely pulling in javascript.

Just so we're on the same page, what I'm talking about is adding some code to the Google Analytics module that checks if "X-Purpose: preview" or "X-MOZ: prefetech" is sent by the browser and if it is it still loads drupal but doesn't load the Google Analytics code. This way the user still gets their thumbnails or previews but we don't have to deal with inaccurate stats in Google Analytics such as high bounce rates.

Here's some quick example code just for demonstration purposes:

if ($_SERVER["HTTP_X_PURPOSE"] == "preview" || $_SERVER['HTTP_X_MOZ'] == "prefetech") {
  // Don't send the Google Analytics javascript code to the browser
} else {
  // Send Google Analytics javascript code to the browser
}