Can you trust stats?

With Hitwise getting acquired for millions and comScore going for an IPO, every company in the web stats area has that million dollar twinkle in the eye. This includes, Quantcast, and the awful Alexa. While Quantcast and Alexa don’t really try to push and pull companies, Compete has been on the free research reports roll for the past few months. Publish some juicy social networking or Google comparisons and they get everyone watching. Well there is nothing wrong about that but only only problem is that their stats are way off the mark. Every webmaster would agree with the fact that Compete’s stats are 20-100 times off the actual. Now I don’t want to go case by case basis since it is all so obvious. Even Compete would agree with this fact.

My point here is that Compete is dishing out these comparison reports day-in-day-out for which an ordinary or occasional user has no real way to corroborate the facts. Consider the case when user hit the Google Checkout vs Paypal report published yesterday by All that visitor would learn from the report is – “oh well, Google is loosing”. But are those numbers really accurate as Compete wants it’s visitors to believe. What about all the transactions coming through the Checkout API, where the complete checkout happens on the merchant site itself. I know there are not many merchants who have gone for the advanced integration, but did Compete even include that in it’s cals?

In my opinion Compete should be putting up a disclaimer at the bottom of each report it publishes stating that data in fact might not be as accurate as it looks. Or at least give expected variance from the actual.  I think they owe this to the visitors and to the companies they profile. If publishing reports gets Compete employees a paycheck, building apps gets someone a paycheck. Trying to influence public opinion based on no so accurate stats can ’cause someone to loose his/her job somewhere in the world. 


11 Responses to “Can you trust stats?”

  1. Stat Head Says:

    I don’t know Vivek, a lot of people have verified that Compete stats are pretty accurate, when compared to their own stats. Granted their sites are pretty big with over 500k UVs a month.

    I remember Compete’s Technorati vs Google blog post, where their numbers were surprisingly pretty damn close to what Sifry reported on his own blog. Same is the case with Markus (owner of Plentyoffish)..


    It is really impossible, and perhaps not even advisable, to tie the competitive intelligence data with the clickstream data from your own website web analytics tool. The data collection is very different and the purpose is very different.

    In the context of competitive intelligence you are comparing two entities (you and your competitor) using the same tool / process / application and that normalizes the %u201Cbias%u201D. All other things being equal, you can still find actionable insights and make fundamental changes to your strategy.

    Disclaimer: I use Compete all the time and love the service, and follow their blog.

  2. Vivek Puri Says:

    Stat Head,
    Your assumption above is that Compete is accurate, and hence could be used for comparison purposes. I think that is flawed basis to start with.

  3. Stephen DiMarco Says:


    I work at Compete. After reading your post, I compared our data for your site to Quantcast (you’ve quantified your site so we can use that as a solid benchmark). shows that had 1,217 US visitors in March. Quantcast shows 926 visitors. That’s a difference of 1.3x. So I can’t agree with what you said about being 20-100x off.

    Its not just me, your own data just don’t agree.

    Read the last line of your post above. Now, I can agree with you on that point. Its really smart.

    Side note: you’re pretty outspoken about not understanding our Google Checkout blog. Why didn’t you just post a comment on the Compete blog? We would have responded to it quickly and had a great conversation. Its a bummer you chose to go solo and shoot us in the back. Our experience with our own blog (ranked 3438 today on Technorati) is that good data and open conversations will help you get into the top 5,000.

    Second side note: We love Snapshot, the Compete Toolbar, our blog, all of it. Its been such a huge part of our success that we’re going to launch even more metrics soon: subdomains, search, referrals, demographic and behavioral compositions. Its going to rock the industry!

    – Stephen

  4. Vivek Puri Says:

    It would be naive of you to think that only 1,217 or 926 visitors are coming to the site. Now I can very well imagine the kind analytic abilities you guys posses.

    As for Quantcast stats, in case you don’t know, Quantcast gives out a javascript code for site stats tracking. As of now i don’t have that code installed. Either way, if you had been wee bit smart, you could have guessed the traffic on my site.

    As for the rank of your blog on Technorati, it in no way ensures that what you are delivering is the perfect information. Also don’t ask this correlation question from me. Ask it from yourself – Do you really deserve to be in the place you are?

    As for my commenting on your site – If you are going to publish bull each day, don’t expect people to come and comment on your site to have a “open converation”. Even now you don’t realize that in the Paypal vs Google case, Paypal also has advanced API support, which means lot of transactions happen on the merchant site. In light of this fact, your stats are even more skewed. I don’t think people have the energy or time to spend on correcting someones mistakes and that also for someone at Compete who are so adamant about being right.

    Last but not the least, I am actually amazed at your statement “Its going to rock the industry”.

  5. TJ Mahony Says:


    I too work at Compete and I manage the group that runs the blog.

    I appreciate that you read and engage with the information we provide. I understand and agree with the issues you presented, but I hope we can agree that what can be measured by onsite behavior is a good proxy for overall adoption of any type of web service. If that is not the case, it would imply you believe the web is overweighted with API integration and RSS – which I would contend is not the case. Yes, we the early adopters leverage these services, but my mother and the average American do not.

    We have invested a considerable amount of time and capital in developing the largest and most diverse domestic panel in the market. The largest web properties in the world leverage our data to understand what is happening on the web. The disprepancies created by emerging backend integration services do not compromise our data nor insight.

    I doubt we can convert you into a fan, but I do feel its fair that you respect the value of the analysis we provide to help educate the market on how consumers are using the internet.

    My email is provided on this post if you would like to discuss our data and methodology. Our goal is to provide transparency and insight to a channel that is redefining how we are managing our lives.

    Despite the tone of the article I appreciate you including us in the conversation.


  6. Vivek Puri Says:


    There was nothing really wrong with the tone in the article. If you read near the end, i am just suggesting ways for Compete to improve on.

  7. TJ Mahony Says:


    Agreed. These are important conversations. We always need to be challenged to find the next level.

    Ultimately I hope you’ll find you can “trust” our stats. It’s our business, we take it seriously and we have stripped down the paid firewalls to let everyone see how consumer behavior is evolving.

    My line is always open. Please reach out if you would like to look into future issues.


  8. Passer By Says:

    I recommend everyone read Avinash Kaushik’s blog (, especially if you’re not familiar with the world of competitive online intelligence. It is a very different animal, that most people don’t get (which is understandable if you’ve only looked at site logs or alexa all along) and too easily equate with their own site logs.

    @Vivek, regarding Stat Head’s comment, I agree. Even if you *assume* Compete data is incorrect, you can still use it for comparative analysis. You’re comparing two sites, using the same tool, with same “bias”, methodology, etc. The data doesn’t need to be 100% accurate, unless say you’re using Compete vs. a Google Analytics to measure *just* yourself — which you shouldn’t be doing in the first place.


  9. Vivek Puri Says:

    Passer By,

    In that case what is wrong with Alexa?

  10. StartupSquad » Blog Archive » Can you trust stats? » Beitrag » Club Seeker Says:

    […] Original post by Cambrian House, Home of Crowdsourcing […]

  11. áåñïëàòíî Says:



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: