TL;DR: User/visitor-based KPIs have never reflected actual users. They’ve only ever reflected clients (IOW, the combination of device and browser you visited the site from), making reporting on them a waste of pixels.
Continuing my series on dimensions in Google Analytics, I want to address the most misunderstood of them all: the User Type dimension. This dimension is supposed to measure new vs returning visitors** to your site. But, in reality, this dimension provides your report with a steaming pile of poo. I’ll demonstrate why in this post.
But I’m not just going to look at the User Type dimension. Since the issues that plague the User Type dimension apply to metrics as well, I’m going to broaden the scope of my investigation to all user-based key performance indicators (KPIs). This includes the Users and New Users metrics, which are accessible via custom reports, as well as the Overview reports (under Audience and Acquisition, respectively).
**Google still vacillates between using “user” and “visitor” in its report nomenclature.
Faulty Assumption
Website owners, marketers, and analysts have had a long-standing love affair with these KPIs. But they were ill-conceived from the beginning. They always operated under the assumption that you use only one device and one browser or app on that device to surf the web. Even in the early days of the web that wasn’t true for many who had a computer at home and work or who bounced back and forth between Internet Explorer and Netscape. #flashback
Everyday Analogy
Calling unique clients visitors or users is kind of like saying your social security number (or whatever number your country uses to identify its citizens) is interchangeable with your license plate. A single person may have multiple vehicles s/he uses to get around. But my car != me. Therefore, my girls can borrow my car without risk of being arrested for identity theft.
Likewise, your client is a vehicle you, the user, use to get to a site. It’s not you. The problem with user-based KPIs in Google Analytics is they measure license plates, not social security numbers.
Let’s Do an Experiment
My devices: I have two laptops (one Mac and one PC), a desktop (an iMac), an iPad, and an iPhone.
My browsers: On my laptops and desktop, I have a combination of four browsers: Chrome, Firefox, Safari, and Internet Explorer.
My Chrome habit: To complicate matters, I use multiple instances of Chrome (which I show you how to do in this post) because I have an affinity for Chrome.
My apps: I use a variety of apps between my iPad and iPhone.
My Quest for Redneck Wine Glasses (Phase 1)
Watch what happens when I search for redneck wine glasses in two different instances of Chrome. With the Google Analytics Debugger extension active, I clicked through to the One Hundred Dollars a Month blog, which had a post featuring redneck wine glasses. And then I looked at the data Chrome was capturing about me in the console. Notice that it indicates that it’s my first session (formerly known as a visit) and assigned me a Visitor ID of 712640405. This is accurate. I’ve never been to this site before. Or sipped wine from a redneck wine glass, regrettably. (But it’s on my bucket list.)
Note: This site is running the classic version of Google Analytics. If it had been running Universal, you would see clientId instead of Visitor ID, like you do with my site.
And why do you think Goog changed the nomenclature from Visitor ID to clientId? You got it! Google Analytics is measuring clients, not visitors. Unfortunately, this shift isn’t reflected in Google Analytics reports, as they still give you the ability (although just an illusion, IMO) to report on users.
Google Analytics does give us the ability to measure actual visitors now with Universal, but there are quite a few hoops you have to jump through to set this up, which we’ll get to.
Rinse and Repeat in a Different Instance of Chrome (Phase 2)
Now let’s see what happens when I go to the site using a different instance of Chrome from my same laptop.
This time I was assigned a Visitor ID of 1140255071. At the risk of stating the obvious 1140255071 != 712640405.
It also recorded it as my first session. Liar. You’re my witness. I just visited this site.
But it actually makes sense if you trade in your Warby Parkers for a pair of taped-up horn rims since, according to my cookie data, I’m a totally different visitor. This poor, unsuspecting site has no idea that I’m the same person who just found the site via a Google search.
It gets worse.
If I came back via Firefox I’d be a third visitor. Safari? Yet another. My iPad? Yep. Another. My iPhone browser? You guessed it. And an app on my iPhone or iPad? Throw in a couple more.
When you consider that I use different instances of Chrome on each of my computers, I could realistically be 10+ different visitors for sites I visit regularly.
For example, I bank from all of my devices, and I use my bank’s app on my phone and tablet. Because I use the same instance of Chrome for my banking, on any given day I could show up as coming from one of five Client IDs: my two laptops, my desktop, my iPhone app, or my iPad app.
Why I’m Calling BS
If I can’t be identified as the same person while using the same browser and device, how is Google Analytics going to associate me as the same user when I actually switch devices? It can’t. So I’m only a returning visitor if I come back to your site using the same instance of the same browser on the same device. And I’m a <air quote> new visitor </air quote> every time I visit your site with a different device, browser (or browser instance), app, or kiosk.
So when you add a KPI like new visitors to your monthly reports, you’re not exclusively celebrating acquiring, you know, new visitors (although I’m sure there are bonafide new visitors in the mix). You’re more realistically celebrating diversity. Some of your visitors want to visit your site using different devices and browsers. Woohoo! Drinks on me!
Who Can Safely Use User-Based KPIs
You can use user-based KPIs if you satisfy all of the following requirements:
1. Your site has a login option. This is the most critical component. There has to be some way to authenticate your unique visitors. A site like Facebook would have no problem identifying unique users because most people using the site are going to be logged in. So it doesn’t matter what device or browser or app I use, Facebook knows who I am. Going back to our car analogy, they know my social security number, not just my license plate.
2. Visitors actually log in to use it. My site has a login for subscribers to my dashboard course. However, the overwhelming majority of visitor visit the site logged out. My content isn’t behind a subscriber wall, so there’s no reason for visitors to log in just to read my blog posts. So guess how many effs I give about my new vs. returning visitors? My site isn’t a good candidate for these KPIs.
3. You are using Universal Analytics. You have to update to Universal to take advantage of Google Analytics’ new feature that actually enables you to track unique users. (More on that in the next point.)
4. You have modified your Google Analytics tracking code to capture your members’ User ID. You can use customer ID, login ID, member ID … whatever your CRM uses to identify your users. But it can’t identify the user to Google, so names, email addresses, and the like are out. Justin Cutroni wrote a great post about using the User ID feature if you want to come up to speed. But you will still need to read the help files on this one. That’s the only way I was able to really wrap my mind around it.
5. You have set up a view that uses User-ID to determine new/returning visitors instead of clients. After you’ve added the extra code to capture your site’s User ID, you’ll need to create a new view to enable this feature. Google doesn’t reprocess data like tools such as Kissmetrics does, so you start from scratch whenever you enable a feature like this. You can read more about this in the Configuration section of the User ID reference.
This new view will only track the visits of your logged-in visitors. A switch is flipped, and Google Analytics no longer uses the device to determine if a visitor has been to the site before; it now looks at that User ID. Because of this switch Google is able to provide you with new Cross-Device reports not found in views that don’t have User ID enabled.
Better But Still Not Perfect
Sooo if you are doing all of these things and using a User ID-enabled view, your user-based KPIs are infinitely more reliable. Why? Because now you have a way to identify me across different devices and browsers. I’m no longer just a visitor coming from my personal Gmail instance of Chrome on my 15-inch MacBook Pro; I’m Annie Cushing … a real human being with a social security number and proclivities for data, shoes, and lime green and aversions to country music, Brussels sprouts, and the color pink.
It’s still not foolproof. I mean, I could go to my bank’s website to research how to send a wire transfer without logging in. Also, if I’m a member of a site and have logged in with my current browser but am not logged in when I visit it, that session will not be routed to this view. Kissmetrics has it all over Google Analytics in its ability to identify unique users even if they’re not logged in. But it’s at least a step in the right direction.
If I had my druthers, Google Analytics would align its report nomenclature to reflect that these are clients, not users. I think it would help marketer assess if it’s worth reporting on.
Bottom Line
If you’re doing analytics for a social networking site, bank, airline, online tool, or any other site where your visitors have incentive to log in — and you’re taking full advantage of Google Analytics’ newer User ID feature — you’re golden. And you should be rocking those user-based KPIs.
For everyone else, I’d shift your focus from Google Analytics’ traditional user/visitor KPIs and put the lion’s share of emphasis on session-based KPIs. There’s plenty of good data in them that’s not based on false assumptions.
Learn More
If you want to become crazy marketable with advanced analytics skills, check out my Analtyics Audit Template, a self-guided, 147-page audit template that is regularly updated and will teach you how to do detailed analytics audits like a pro.
Great post and interesting discovery…I guess we have to accept that Google is not all-seeing…at least not yet 😉 We’re currently working with a client to convince him of the benefits of transforming his site into a membership site – so thanks for providing another little reason 🙂
Happy to help in some way! I’m a huge fan of Google Analytics, and the Google Analytics team is fabulous. I think this is one of those clusters that Google inherited from Urchin (like another one I’ll be publishing eventually) that they just never really got in front of. At minimum, my hope is that marketers will have a better idea of what they’re actually reporting on when they include these KPIs in their reports. It pains me to think marketers may have been fired (or agencies and consultants dismissed) because they’re not bring in enough new visitors or they’re not making the site engaging enough to attract return visitors. That’s why I finally spoke up.
Great explanation of the current state of the tracking challenges, Annie.
Your focus was on not being able to trust the ‘New Visitor’ numbers, and I agree, but the other batch of “Returning Visitor”s are worthy of some consideration. At a minimum, these people came back, using the same device they did before. Maybe even from multiple same devices. If that number is high, you have a bunch of habitually addicted followers….yummy! That is good to know!
Also, even if you had 14 devices, it is only the first visit from the device that gets tagged as ‘new’. After that, it would tend to level off the New Visitor duplication over time. If you weren’t bringing in any new eyeballs, your numbers would drop quickly.
It all just reinforces that numbers alone are meaningless, and that is why analysts need an understanding of what the numbers actually measure. Article like this are just what people need – keep it up! I look forward to the next one.
Maybe it’s my science background, but I find marketers’ tendency to gloss over glaring leaps of logic as concerning. What it boils down to for me is we’re calling these KPIs users and visitors, but they’re not. And yes, although there is a modicum of value in noting that you have some returning visitors, businesses will naturally want to compare those numbers month over month and year over year. But you can’t do that because they’re slippery rocks on a slippery slope of bad data, from a statistics standpoint.
It reminds me of when marketers were taking organic keyword data, as it kept shrinking and shrinking, and were analyzing whatever was left, until Google finally just took all of them away.
So my opinion is if the data can’t stand up to scrutiny, marketers are taking risks reporting on it. Marketers are relying on these KPIs to communicate how many unique users, new visitors, and returning visitors a site is getting. But the technology doesn’t support the claim. That’s an issue. And my experience has been that if you try to circle back and explain why you, as a marketer, can’t be held to performance expectations because the data isn’t accurate, you’re going to have a difficult time gaining credibility, especially if you bring it up when you’re not meeting expectations.
Well done and much needed post Annie.
I think the exact-ness of Analytics (those metrics down to the hundreths!) lulls a lot of marketers into thinking that their Analytics are hyper-accurate.
There’s a lot of nuance and caveats that go into something as seemingly basic as New Visitors – not to mention engagement metrics. Thanks for taking a close look at it!
Very true, Nate. We have to think about what we’re actually reporting on to make sure we’re not being held accountable to KPIs that rely on bad data.
Hi,
Great post on tracking users, I stopped using visitors as a metric quite a few years ago.
In relation to the new USER ID setting in Universal Analytics I set it up the following way to help combat just a little bit more.
When a user logs into my website I drop a cookie onto their machine purely with the User ID of the customer, Then if they come back and are not logged in I still pick this up in Universal Analytics.
This is sheer brilliance. If you ever write a post on how you set this up I will link to it from this post.
Although doesn’t this break GA terms of service? I thought they say you should only record a user ID when visitors are logged in? Having said that, I can’t find a reference to this immediately.
I just read through the Google Analytics ToS and didn’t see any such restriction. It just says that you need to disclose how you use cookie data:
“You must post a Privacy Policy and that Privacy Policy must provide notice of Your use of cookies that are used to collect data. ”
So it would be incumbent on the site owner to disclose that they’re using cookie data to identify members/customers. But beyond that, you just can’t send personally identifiable information to Google, and a User ID doesn’t.
You can read the ToS (for the US) here: http://www.google.com/analytics/terms/us.html.
Great post Annie!
For ecommerce sites, even partial availability of User-ID enabled insights with session unification would be a goldmine to identify the high value TG segments and leverage that to target more users like them 🙂
Can be used to create User based remarketing lists rather than visits based segments..
Absolutely! But I think you’d get way more value from capturing these User IDs in a custom dimension. That way you can marry your GA reports with your CRM data and also build out some VIP email lists (based off smoking metrics like revenue earned per customer). That’s way more valuable than just knowing they came back to the site x number of times.
Great post Annie. As you say it depends on the site. One site I analyse is used by librarians who are on a restricted network. This means they use the same machine and that only has one browser. Granted that scenario is a one off. Given for other sites new vs returning isn’t the best metric. What else could we use to measure loyalty?
There are lots of metrics that aren’t visitor-based that are better options, such as Page Depth and Session Duration (both under Audience > Behavior > Engagement). Although Session Duration is less reliable on blogs where visitors often don’t click a second page. Page Depth is rock solid.
But also, if you’re using the AddThis or ShareThis WordPress plugins, you can really get some compelling engagement metrics.
Fantastic post Annie! I have always struggled with users of digital analytics mis-interpreting person-based metrics like these, and I’m ecstatic that you’ve covered this (and in your usual inimitable style)!
Much like @John, I’ve set up our User ID reporting to cookie users with their User ID so that return (non-logged in) visits can be associated with the same user – this is relatively straightforward to implement (just set up some logic in your tag manager to read the cookie value if it exists, and insert it as the User ID – otherwise when the user logs in use that value as usual, and drop a cookie).
What I’ve done that may differ from John’s implementation is that I’ve restricted this to users of mobile and tablet devices, in order to minimise the risk that the browser might be used by multiple users (the other side of the inaccuracy of these standard ‘user’ or ‘visitor’ metrics). I figure there’s a better chance of mobile/tablet devices being used by a single person.
Over the years, I’ve seen the advertising world obsess about using panel-based data because of the need to represent people (and their demographic information) and not device/browser instances – this also driven by the fear of cookie deletion/blocking, etc.
Interestingly, since Google acquired DoubleClick (2007) and AdMob (2009), and released their attribution modelling tool (2011), advertising agencies have seemingly been less concerned about ‘user or cookie -based’ metric inaccuracies, and have now embraced attribution modelling, which of course relies on cookies and inherits all the multi-browser issues you mention here.
This is really fascinating, Christopher. Thank you.
As a non-developer, my question is how do you set up your tag manager to grab the user ID from either an authentication OR the cookie? Is there some kind of data layering built in that tells the TMS to first look for a login; if it doesn’t find that, get the user ID from the cookie? What I’m thinking about is those instances where a customer visits the site but isn’t logged in the first three pages but then logs in and browses some more or checks out. Then you would have the User ID from both the cookie (we’ll assume this visitor has been to the site before) and the login. In such a case, how does the user ID data get sent to Google?
I’m not really a developer either 🙂 I can write a bit of javascript, and I understand the concepts. Sorry – reading back over my post, it wasn’t very clear.
Our TMS allows us to set cookies, and to look for values set in those cookies, so it’s a simple matter of setting a load rule that looks for the cookie value if the user isn’t logged in. If the user logs in, then the logic reads the value from the login details. The source of the User ID value doesn’t matter to Google – it is set explicitly in either case, and in this case will be the same value anyway.
Pity about the Brussels sprouts, BTW – I think they’re a widely mis-appreciated vegetable (and often poorly cooked) – they’re magic when pan-fried with bacon!
This is fantastic. Thank you!
Excellent post, thanks Annie!
As for setting up the cookie / login lookup in a TMS – thanks for the perfect idea for the next #GTMtips article 🙂
Excellent, Simo. Ping me on Twitter when it’s finished! https://twitter.com/anniecushing
Hi Annie,
Such an important truth about really all web analytics platforms today; they’re cookie-dependent, and therefore browser-dependent.
Another source of analysis confusion you mentioned was native applications; this is actually worse than you imagine. Most applications make use ‘web views’, which essentially instantiate a basic browser within the context of the app when you click an outbound leading link. The greatest hits list includes Gmail, Facebook, Twitter… it goes on. I looked at all of the applications I regularly used this functionality on and came to realize that I could represent 41 different clients overall. Chances are better than not I represent 3-5 on any given online transaction I make.
It’s a wild world out there. Thanks for sharing this.
Dan
Hey Dan,
I agree about this being a cookie issue. My main issue comes down to labeling. By labeling KPIs as user/visitor you’re setting an expectation that the data can’t support.
Could you clarify your point about the apps? What do you mean by “outbound leading link”? Or “The greatest hits list includes Gmail, Facebook, Twitter… ” I didn’t follow how these could represent 41 different clients.
Always a pleasure to see your name pop up! 🙂
Hi Annie,
I mean an external link leading to a 3rd party site that you click from within a native application, like a news story that someone shares on Facebook; I believe internal links are opened as App Screens, but I could be wrong on that. When you click those links, the app instantiates it’s own browser, called a web view (IIRC). Here’s a screenshot of what I’m talking about:
http://www.insidefacebook.com/wp-content/uploads/2009/08/application-webview.jpg
This instantiated browser behaves just like Safari or Chrome on your phone; it accepts cookies, renders HTML and CSS, and executes JavaScript, etc.
For example, if you shop for shoes on Amazon, email yourself a link to check out later, then click that link AND THEN click the ‘Open in Safari’ button, you’ll have created three different ‘clients’ inside Analytics. Every app’s web view is it’s own browser, so if you open a lot of links within applications, you’re generating a ton of new clients. My 41 number comes from this:
– 2 different browsers on work and home desktop computer
– Safari on iPad, iPhone
– 35 different apps that I regularly use web views in
I figured this out for a talk I was giving on direct traffic, but it impacts both. It’s a nightmare.
Dan
Also, just to clarify: each web view maintains its own set of cookies. They don’t get shared with other web views or core browsers.
Oh okay, I thought you were saying you could show up as 41 “users” to a single site you visit. You’re talking about all of the sites you visit from all of your devices. Totally agreed.
Wow, you hit it over the fence! “Social Sec vs Lic Plates” analogy is so perfect even I got it! 😉
Question:
Has there ever been a user habit study that looks at how many cars a typical user has? (i.e. user study of # of devices and multi-browser use).
In a perfect world, the study would require context to include classes of the use case of the sites (e.g. banking, blog, ecommerce, social, news, catalogue, education, etc…).
thoughts?
Joe
#NewAAFan!
Hey, Joe! Thank you for the kind words! I don’t know of a study like this, but it would be a fascinating piece of research. I’d love to know myself.
Hi Annie!
Great post! A thing that everyone has to give thought before opening GA and start “mining” for insights. My question is: Do you consider the standard “User” dimension totally useles from an analyst standpoint, or a change in it’s related metrics could still indicate something useful, if we are counscious of what it really means? As I understand the so called user dimension can cover 4 types of users a new one, a returning user in a new browser type (recorded as new), a returning user on a new device(recorded new) or a returning user on a device-browser combination that has already been recorded in GA(thats the returning visitor dimension). Is it right to make the assumption that most of the new visitors(i mean humans) will use more devices and browsers to visit a site in a selected timeframe, so by dividing the new visitors in a calculated metric by an arbitrary average at least can get us closer to a real number of new people who came to the site? Or you suggest simply to ignore theese dimensions if user id is not an option?
Thanks,
David
Hi, David! I don’t consider user-based dimensions and metrics completely useless — more an outdated concept than anything. There are probably sites people primarily visit on one device, so you really can’t apply a general assumption to user-based metrics. I’m probably not going to visit my health insurance site on my mobile devices too often (if at all), but that’s more because they’re generally not very usable on mobile. That could change over time.
Hi Annie,
We have GA on our login based site, and I am wondering how I differ from the site that doesnt require a login. There is nothing preventing a client from logging in from various devices or browsers, so my user data will be equally skewed. I understand I can track my login data on the backend, but how does having a login make the user data more accurate?
Cheers
Dan
It depends on how compelled visitors are to log in. If your site is like mine, where logging in is only required if you’re trying to view content behind a pay wall, your UserID-enabled view is going to have limited data. It’s still good to capture customer ID as a custom dimension, as well as a boolean value of logged in or logged out, but you’re not going to much beyond that.