Reported install usage statistics count on the project page is not being updated

Looks like it isn't updated at all.

Setting #2128619: Project usage/download/install statistics not being generated in a timely manner as "parent" issue as that currently seems most logical.

Agreed. I guess I was thinking more on a conceptual level than a technical one, since the tech specifics are still not clear. #2128619: Project usage/download/install statistics not being generated in a timely manner is has become something of a "master" description of all the outdated update stat issues, and until it's clear which sub-issues are dependent on others (at a technical level) I was just trying to tell the right story for user's who land on #2128619. In this case I just wanted to get the parent/child/related box to align with the issue description (and it seemed that this issue was spawned from that one).

In the case of these d.o. (site-specific upgrade) issue queues I'm honestly not sure what the process is. The proper analysis for some of this stuff seems to require special system/code access that I guess only d.o. site admin have. I figured that the best way I could contribute would be to just try and enhance some of the summary info for now.

I don't think the previous description of 'not being updated in a timely manner' can hold when we are seeing projects where the reported installs and downloads are not updated at all (as stated in #3 on Nov 21st). The timely manner relates more to the statistics 'graph page' which has seen varying degrees of lag in updates and is dealt with under issue #2128619.

There is also the added irk that the 'last updated' date in this section of the project page seems to be 'updating' whilst the reported statistics have been static for over a month now.

Given the ongoing problem, and the relationship of this issue to https://drupal.org/node/2128619 which is marked as 'major', I've amended its priority to bring them in line with each other.

I agree Rik, the importance of these statistics to newly released modules can't be underestimated. I have, in the past, seen comments where people don't want to use a particular module because it's usage statistics are low, and I have had to point out that it's a new module. Delaying stats on new modules can be damaging to their perceived popularity/quality. It is also very disheartening for the developer and will make people think twice about putting the time and effort into developing further modules.

I would prefer to have this section of the project page hidden until it is fixed, thereby forcing people to look at the graph (which I suspect remains buggy). Presumably this could be done easily enough?

What would be really useful would be some feedback as to what caused these issues, what are the likely timescales/obstacles to fixing them. I also think that issues which remain unresolved for longer and/or which effect a very wide user-base should receive higher priority.

Any feedback from the Drupal.org upgrade/webmaster team would be welcome.

Some summarising across issues: https://drupal.org/comment/8255729#comment-8255729

Downloads should be solved with #2133385: project_generate_download() query needs porting to D7 structure. That code is committed and running now. It took 15 minutes on D6; D7 may be slower or faster. If downloads don't update soon, that issue, or the parent issue should be updated.

The key problem is

Error: Call to undefined function db_affected_rows() in /var/www/drupal.org/htdocs/sites/all/modules/project/usage/project_usage.drush.inc, line 104

At the bottom of my project pages I added:

<strong>NOTE</strong>:
#2129811: Reported install usage statistics count on the project page is not being updated

It turns into a link to this issue with the title 'Reported installs count on project page not being updated'. Furthermore on hover the issue status is shown, 'Status: Active' at time of writing.

This way the issue gets known by site builders that currently might be unaware of it.

Thanks.

Re: #11
@drumm are you saying that this is fixed given the removal of 'Download Statistics' from the issue title? I'm not seeing any fix to the downloads aspect of this issue. Also, I don't get what you mean by referring to D6 when this issue relates to the current version of Drupal.org i.e. D7, can you explain?

Also, the parent issue relates to Drupal.org D7 upgrade QA but you've changed this to Drupal.org customizations which is confusing given that the problem arose from the upgrade issue, and is not a new feature.

davidmac: Consider D7QA a 'triage' area for upgrade problems ... once the root cause is discovered, we move to the issue to whatever project actually hosts the affected code.

And the title change was to indicate that there are two problems here ... download statistics and reported install counts. They have two different causes/solutions, and the download statistics solution was delivered in the linked issue.

Something like this should fix #12 (untested!)

@jthorson Many thanks for the explanation, however the issue is also for those that are experiencing the problem, so IMO, care should be taken when stating that it is fixed.

It may be fixed in the code but is not fixed for those who are reporting the issue in the first instance, because the problem persists. Removing aspects of the description makes it difficult to find for others, and leads to duplicate/overlapping issues being raised as evidenced by the parent and child issues here.

I'm trying to keep tabs on all this, partly to ensure the info in the parent issue (#2128619: Project usage/download/install statistics not being generated in a timely manner), which is starting to feel like a "meta" issue, is comprehensive. What I've been able to glean from all of this is roughly the following:

1. The issue related to download stats not being uploaded on project pages was addressed in #2133385: project_generate_download() query needs porting to D7 structure, and this is the only issue related to project statistics tracking that was addressed there.
2. This issue (download stats) has been addressed in the appropriate code repository, but the related commit has not necessarily been pushed to the specific instance of d.o. (meaning d.o. users do not yet see the download status problem as "fixed").
3. The issue related to reported install stats not being uploaded on project pages is pending, but people would like to continue to use this issue queue to discuss that.
4. The other problems noted in the parent (#2128619) may need other targeted issues opened for them.

How did I do? Am I way off?

This issue is *usage* stats, not download. Download stats are #2133385: project_generate_download() query needs porting to D7 structure.

The UI says "install", which is a synonym for usage here.

Sorry, I got the wrong project.

I went ahead and committed #16. Leaving open since there could easily be other problems uncovered when this is deployed and runs.

rjacobs - otherwise, your summary is okay.

@drumm @rjacobs

Thanks for clarifying the latest state of affairs.

As you know, the download field has updated over night, but will need to be monitored to see if it's in line with trends.

As far as I'm aware there has been no loss of data throughout this hiccup, is that correct?

No data should be lost. The scripts are designed to parse all the logs available.

Corrected typo in issue title.

I took care of the last notice on line 490 of sites/all/modules/project/usage/project_usage.drush.inc. https://drupal.org/drupal-4.7-cvs thought it was a release of the #4878: Support file uploads via blogapi project, but that is an issue. I corrected it to be a Drupal core release.

We still have some

Notice: Undefined offset: 1 in _project_usage_process_varnish_file() (line 264 of /var/www/drupal.org/htdocs/sites/all/modules/project/usage/project_usage.drush.inc).

But that went way down. Either that is fixed by fixing the bad data for line 490, or there were just fewer lines in the parsed logfile triggering this notice.

The big problem is that the query killer is killing this while storing results.

Let's wait a day for the next run to see what notices remain. Once the notices are gone, I think we can close this, and leave the parent issue. The blocker is the query killer, which exists because of #2136119: Text search on issue queues is slow and sometimes WSODs.

What if we were to get rid of the database transaction and the db_delete line, and use db_merge instead of db_insert? I know this would result in 1000 times more individual database transactions (since db_merge() doesn't support multi-values); but at the same time, the fact that our insert is running longer than the killer script timeout shows that the status quo isn't exactly efficient either.

Thanks to #2178437: Add a DB user for cron, this has gotten slightly further. The next problem is

PHP Fatal error: Call to a member function rowCount() on a non-object in /var/www/drupal.org/htdocs/sites/all/modules/project/usage/project_usage.drush.inc on line 104

I believe I have fixed this, http://drupalcode.org/project/project.git/commit/6219465.

Seems this is working: https://drupal.org/project/usage/drupal

I also am seeing this as fixed.

Thanks