Is something wrong with Project Usage Statistics?

Usually this fixes itself in a few hours. Please re-open if it's not fixed by Monday.

Not fixed yet.
http://drupal.org/project/usage/menu_import
zeros...

Util got rebooted 3 days ago.

The mongod that stats uses on util has to be manually started. I just started it up myself and started a processing run. I am marking this fixed again. Again, it may take a few hours for stuff to start showing up, but I'm confident I have resolved the underlying problem.

Still, I can't find a single proj which stats is non-zero on 2011/10/09, that is any specific project stats shows 2011/10/09 and zeros for it but http://drupal.org/project/usage/ list doesn't show the "October 9th, 2011" column at all (as one would have expected)

Before this latest Sunday (2011/10/09), all stats updates happened on a Sunday (and no other day) and since the last update on Sunday (2011/09/16) was already missed, I wonder if it would result in zero hole in stats further on?

Update:
Looks like I precipitated... after more checking I managed to find projects with non-zero stats on 2011/10/09
though the point is that zeros in stats, as being wrong, should not be shown at all?

I just cleared the project statistics cache.

@KeyCAPTCHA Scam:
"It's a feature."
If it shows up as obviously broken, people will report when things break, and word will get to me that I need to fix it.

Also, technically stats updates happen every day (There is a LOT of log data to process behind the scenes), but the numbers shown to the public are set to leave out the "current" week to reduce confusion somewhat, because it is constantly changing as sites call in throughout the week.

The usage overview uses a different method of determining the correct "newest" week to show than the per project stats page due to the crazy way the usage overview query has to be set up. (IIRC)

Also, everything is heavily cached for up to 24 hours.

Looks like this is happening again - all stats are coming back 0.

Also the sort by "most installed" for themes is some weeks out of date, perhaps that is cached also and needs to be cleared?

Would be good to see the themes page update, its quite out of date now, from what I can tell this hasn't updated since maybe Sep 25th.

I guess the status change was accidental. I know bdgragon was working on this today, I think it was going to take 24 hours before it really got fixed.

Seems like it goes wrong again today:
all digits for January 29, 2012 are either zeros or none.
Not a single non-zero digit for 29/01/2012

PS.
Previously this issue was
"Automatically closed -- issue fixed for 2 weeks with no activity.
October 31, 2011 - 18:30 : System Message"

Why wasn't it auto-closed after November 8, 2011's reopening?
What was the activity since, I wonder?

@Scammed by KeyCAPTCHA Only issues that have the "fixed" status are automatically closed; the rest of the issues is not automatically closed, and for what you say, it should not be closed because it is not fixed.

I guess my words will not discover America but it happens again.
Please clear project statistics cache to fix it now and consider finding a permanent solution of this happenning-from-time-to-time problem.

Stats still down.

From DamZ in #drupal-infrastructure ... I think this is related...

"basic`: nnewton: sdboyer: the mongo PHP extension is available but not loaded on util … as a consequence the statistics processing fails … where is the best place to fix that?"

Josh

And they are now back up. Thank you gentlemen!

It is back again
Usually it is updated on Sunday. Now it is Wednesday (May 29) and stats show 0s for May 19

+1
https://drupal.org/project/usage/menu_import

Same for May 26, and (as far as I can tell) for all projects (including Drupal itself).

Also noticing this for projects. Although it was happening before this, might it have something to do with the whole HTTPS implementation that was recently introduced on d.o?

Is this in the correct issue queue?

I believe this problems belong in the: https://drupal.org/project/infrastructure issue queue.

(However, I've tried to post it there, but it was almost immediately closed as a duplicate of this.)

It doesn't much matter which queue it's in, though you're right it is an infra issue.

Strange, the installations for Linked Field increased but when go to usage statistics is still zero :/

The statistics appear to be back again for all my projects, as well as for the core. (I think it normal for the numbers in the table to be zero for a short while after the number on the project page has been updated.)

However, I've noticed that there is a pronounced dip in the numbers for May 19 for all my projects. And for the majority of my (as well as for the core) the numbers for May 26 is lower than for May 12. While this of course may just be a coincidence. I'd be intererested to hear if there is another explanation. For instance: Has there been a change in the algorithm used?

There was no change in algorithm. It is possible that we lost some data due to rebuilding the servers.

So let's wait and see what will happen to the installations ;)

This issue seems to have occurred again but I won't reopen this until it can be confirmed by others and that it's actually a real issue and not some latent thing that will eventually right itself I guess.

Looks like it is back again
Today, Tuesday (20 August 2013), the stats show 0s for the 3d day since Sunday.

It happens every week. The total goes to 0 between Sunday and Tuesday. It's been like this for years.

Actually the "zero" statistic does not show up every week, I've seen on Sundays and Mondays fresh usage stats that had changed from the previous week that were not at zero. The zero stat thing seems to be happening more and more lately.

My concern here is, are we loosing data? At any rate, it's now Wednesday and both of my projects at least still show zero so I am pretty sure this is a "bug and not a feature" as it were...

It seems like there are two systems working out of sync. It tries to render data for a new week before that data has been made available.

It would normally have sorted itself out by this far into the week.

I'm not familiar with how this part of drupal.org works, but I suspect there is a cron job that takes longer as more projects are created.

Usage of the Persona module is still at 0: https://drupal.org/project/usage/persona

The util_sync_varnishlogs Jenkins job was failing because it was trying to get logs from www5, which responds to pings but is not actually in rotation. http://drupalcode.org/project/infrastructure.git/commit/e2b5ae0 will allow it continue if rsyncs fail. It may take up to a day for everything to get synced back up, parsed, and the cache cleared.

"jbrown on August 22, 2013 at 3:47am

It happens every week. The total goes to 0 between Sunday and Tuesday. It's been like this for years."

It, zero stats until Tuesday, doesn't happen EVERY week. Rather very rarely, that is once in a few months.
But with stats always already filled on Tuesday.

"Until Monday" happens more frequently, but not every week.

Today is Saturday with stats still being zeros (6 days past due date - Sunday) for any project I peeped into.

We now have two weeks without project usage statistics.

On August 23, 2013, drumm wrote:

It may take up to a day for everything to get synced back up, parsed, and the cache cleared.

It has now gone two days, and all projects are still at zero. Is anyone working on this?

All of my projects don't have the usage statistics in last 2 weeks. All zero on August 11, 2013 and August 18, 2013

The log syncing did complete successfully, but a character encoding problem came up in parsing.

Bdragon committed http://drupalcode.org/project/project.git/commit/40743e1 and I deployed it. A "Skipping corrupted site!" message has already been logged, so it is looking good.

The last successful parsing took two hours; this one may be longer since there are a few days to catch up on. Then there is a relatively long-lived cache after that, so still up to 24 hours to see all the results. Since this is aggregating and parsing logs, it is resilient, I expect the missing weeks to be filled in.

Wow, new usage statistics <3

I had new usage statistics, thank you drumm for your hard work

Up

Cross referencing #2128619: Project usage/download/install statistics not being generated in a timely manner, it's unclear if it's this old issue resurfacing or rather something to do with all the issues that occurred after the D7 upgrade. At any rate, both my projects have shown zero for the past few days.

You are caching the project pages for a very long time, is that right? On the Linked Field project i can see that it has 1285 installations but on the Usage statistics I can see that it has 1335 installations. The latest number of installations is from the 11/10/2013 that are 10 days to now.

Same is true for commit statistics in the sidebar block at the top right. Last commit date is also wrong on my project page for Webform Protected Downloads. So it looks like a caching issue as described by yannickoo.

Here is the problem:

05:03:00 Fatal error: Call to undefined function db_affected_rows() in /var/www/drupal.org/htdocs/sites/all/modules/project/usage/project_usage.drush.inc on line 104

Let's leave this issue from 2011 alone and use #2129811: Reported install usage statistics count on the project page is not being updated instead.

that, plus #2176757: Hide the last date row in project stats page when it has no values.

Reopened the issue and updated the description.

Please don't open this issue, it's been closed for a reason.

@Marc Carver, can you explain the reason for it being closed, since the problem persists, and is not tracked by any other open issue?

Please note that the alternate issue mentioned by drumm in #49 (#2129811: Reported install usage statistics count on the project page is not being updated) when closing is a different issue than the recurring issue that is the topic of this thread.

Because it's from 2011 and no, it is not really different from the issue mentioned by @drumm in #49. That issue is part of the parent #2128619: Project usage/download/install statistics not being generated in a timely manner, which we're all very well aware of. There is no need to keep this issue alive, especially since it was one of the maintainers who determined they wanted this issue closed.

@Marc Carver, this particular issue has been recurring with the same pattern since 2011. The pattern is that the usage stats go to zero on Sunday and isn't restored until several days later, when it is updated to reflect accumulated project usage as of last week. This pattern was first reported in 2011 and did not change with the D7 upgrade in Oct. 2013.

On the other hand, the issue referenced by @drumm (#2129811: Reported install usage statistics count on the project page is not being updated, now fixed) was not seen before the D7 upgrade. It also manifests itself differently. The project page accumulated usage statistics count did not go to zero, but remained stuck at whatever value it had on Oct. 20th. 2013.

As this issue describes a recurring problem that has never been permanently resolved, I believe it should be kept alive. But I am not going to insist.

I agree with @gisle. This here is a different issue.

I filed #2176757: Hide the last date row in project stats page when it has no values. as a temporary (?) workaround till we figure out what's the cause of this issue here, but the proposal in that other issue seems more like sweeping the dust under the carpet rather than actually solving the underlying problem.

The usage statistics show zeros for the two last Sundays for all projects
(that I've probed):

March 30, 2014 0 0 0 0
March 23, 2014 0 0 0 0

i.e. stats fail to be correctly updated for more than 2 weeks, again.
Usually it is updated in 1 week, at the end of Sunday or at the next 2 nearest following days (Monday, Tuesday).
Now is Wednesday

#2176153: Reported installs on project page not updating / not correct is already open for this. Please always look for other open infrastructure issues first.