Screenscrape AWSTATs to have running tallies of Drupal downloads available publicly

An example of the .htaccess (or httpd.conf) would be something like:

<Location "/awstats">
AuthType Basic
AuthName "awstats super sekrit"
AuthUserFile /usr/local/apache2.2/user.passwd
Require valid-user
Allow from 111.11.11.1 [this is the IP of the box that runs the scraper]
Allow from localhost [sometimes, you JUST need this line; depends on network config]
Order allow,deny [allow those who match; deny everyone else]
Satisfy any [this says to satisfy EITHER the HTTP auth OR the Allow/Deny rules]
</Location>

Also: "amazon: sure. i don't have any examples of http auth with drupal_http_request, but ... there is scraping code in bot_project.module (that scrapes project.module from d.o)... line 121 through 143ish here.

Drupal5 drupal_http_request cannot to http auth. In Drupal6 it can http://drupal.org/node/182410

Also, note that there is a lot of good discussion on this topic http://lists.drupal.org/pipermail/development/2006-November/020841.html

As far as I know, awstats doesn't include all modules. It stops keeping track of files at some point (the last time I made a report on this it was around 150 downloads a month). Especially now that we've got more projects I'm not sure that awstats can do what we need.

AWSTATs is currently behind an HTTP Auth.

An example of the .htaccess (or httpd.conf) would be something like...

I appreciate the suggested apache config, but as usual things are more complicated here because we have a more complex setup; awstats runs on a separate box, that pulls logs from the webnodes, but it pushes the reports to an accessible web server, which is behind Squid so I would have to use X-Forward-For environmental variables instead of Allow/Deny, etc. We can't pull from Apache anyhow (keep reading), so it doesn't matter.

Also, I don't think running this `scraper' from within Drupal (presumably via drupal cron job) is a good idea, it's more robust for me pull via rsync in a system cron job. Basically, I can drop these statistics into the Drupal web NFS space, we should be able to have a module parse the data already on disk. This is similar to how we have a system cron job updating /var/www/api.drupal.org/src/ which is parsed by the site in /var/www/api.drupal.org/htdocs.

As far as I know, awstats doesn't include all modules.

Depends. It's true that the reports do not show all the modules, however the compiled awstats data files that are used to build the reports do have this data. It's more efficient to parse these data files than the html reports anyhow.

Sample is the month of October (auth required). The section you'd want is the last one, between BEGIN_SIDER and END_SIDER (and you can use the POS_SIDER byte offset in your script to seek directly to this section).

Amazon has now access to awstats.