Different analytics results

  • I’m wondering if someone can give me some general advice please.

    I used Google Analytics for many years on several different websites. But eventually I decided it wasn’t for me – I didn’t like GA4, I couldn’t exclude my own visits (even after advice from a Google staff person) and, well, I am wary of Google these days.

    So after a lot of research I installed Matomo Analytics, compared the results with the GA results, and both seemed to be similar (I didn’t expect total agreement). So I removed the Google tracking code.

    Now, a year later, I am reasonably happy with Matomo, but it has doubled the number of fields in my database, and I just wonder if I need something that is so large. So after a lot more research, I have installed Burst Statistics, which is much simpler.

    But comparing stats between the two, just for a week, I find that Burst consistently give me results that are about 20% lower than Matomo. Of course, absolute accuracy isn’t all that important, I only really need to see trends, but it leaves me wondering.

    Are analytics programs so variable and inaccurate that they don’t tell us much really? Has anyone any references on checking accuracy – maybe someone has done accuracy tests?

    Thanks.

    The page I need help with: [log in to see the link]

Viewing 5 replies - 1 through 5 (of 5 total)
  • Moderator t-p

    (@t-p)

    I recommend asking at https://www.ads-software.com/support/plugin/matomo/#new-post so its developers and support community can help you with this.

    Thread Starter unklee

    (@unklee)

    Hi, tganks for trplying. ButI didn’t go to the Matomo forum because I don’t want to get a Matomo bias, or a Burst or Google bias for that matter. I was hoping someone had a reference where the question has been looked at in detail by someone neutral. I will have a go at doing a bit of an objective test myself and I’ll report back.

    Thread Starter unklee

    (@unklee)

    I’ve done a short test of four analytics packages on two websites, and the results are interesting. The four packages were Google G4, Matomo, Burst and Koko. I’ll post (1) my conclusions, (2), the actual numerical results, and (3) why I think there is variation. I hope someone else is interested.

    (1) Conclusions.
    Google: Everybody knows it, but I find it messy, difficult and complicated. I don’t like it. I think it is probably the second most accurate.
    Matomo: I like Matomo and I think it’s the most accurate. But it puts a mass of tables in the database.
    Burst: I really like Burst as a product, it has good support and is both comprehensive and easy to use, but I think currently it may be the least accurate, generally giving lower numbers than the other.
    Koko: Dead simple and minimal. I like it, but it seems to be only the third most accurate, generally giving higher results than the others.

    Thread Starter unklee

    (@unklee)

    (2) Numerical results

    I tested the four packages on two websites over varying periods from only 10 days up to 100 days. (That’s just the way it turned out). I compared results in 2 ways:
    (i) Ratio of site visits & pageviews of each pair of products to test if different packages gave consistently higher or lower results.
    (ii) How each product compared to the average of all four – I assumed the average was the “best” estimate. I calculated both the average deviation and the Coefficient of Variation (Cv) to test how consistent the results were (after all, consistency is the most important attribute). I think this is the best test.

    (i) Google and Matomo gave very similar results. Burst was generally 5%-15% lower numbers than Google and Matomo. Koko was generally 0-10% higher.

    (ii) Comparing daily results to daily average revealed the following:

    ? Overall, estimates for pageviews varied much more than estimates of site visits.
    ? Google was close to average on pageviews but not on visits. But its results varied both up and down from the average, so weren’t always consistent.
    ? Burst results were consistently lower so probably make better comparisons.
    ? Koko recorded visits close to average, but was always consistently higher on pageviews so alo not bad for comparisons.
    ? Matomo was probably the “best” overall – i.e. not too far from average and not too variable.

    I think there was enough variation in these results that a larger sample is needed. I’m not sure how long I will keep the comparison going, but it would be good for someone else to try this.

    Thread Starter unklee

    (@unklee)

    (3) Conclusions

    Analytics results aren’t as accurate as you’d hope for many reasons:

    (a) Some events are defined differently by different products.
    (b) Different products deal differently with bounces.
    (c) If cookies are used, some web visitors don’t allow them.
    (d) Different products deal differently with bots and none can possible eliminate them all. I’m thinking this is the biggest reason for variation.
    (e) Not all products properly eliminate self visits – I found G4 failed in this for me, despite help from a Google staff member.
    (f) Probably other reasons too.

    My very tentative and inexpert conclusions are that:

    1. I think I won’t keep using G4.
    2. I think Matomo may be the best product for a large site which needs detailed analytics.
    3. I think Burst may be the best all-round product for smaller sites, except that it under-records at present. But once it changes how it addresses bounces (see https://www.ads-software.com/support/topic/how-accurate-are-analytics-stats/), it should be more accurate.
    4. Koko is a very simple and nice product for small sites but tends to over-record pageviews.

    I hope all that isuseful to someone, and provokes someone else to do comparisons of actual data.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Different analytics results’ is closed to new replies.

Tags