Quota API

A few comments, just from looking at the code:

* "quota" is fairly imprecise for a term... i would have thought should implement a more flexible solution for quota management. I'm not sure this should block this patch for now, but it'd be interesting to expand this functionality to support various types of quota and indeed implement a quota API.
* furthermore: I wonder how this would bridge with ecommerce solutions? how does the user change his own quota? can he "buy" more?
* once the quota is reached, shouldn't we just remove the 'add site' buttons everywhere (and therefore hook this into the permission system)?

Code otherwise looks sound, I'll test the patch now.

One thing is missing: you need to add a hook_update_N() to hosting_client.install so that our database schemas are updated. Otherwise i see this:

# user warning: Unknown column 'quota' in 'field list' query: SELECT name as client_name, organization, email, quota FROM hosting_client WHERE vid = 7249 in /var/hostmaster/drupal-6.12/profiles/hostmaster/modules/hosting/client/hosting_client.module on line 358.

Just a quick review for the sake of it...

+++ hosting/client/hosting_client.module	1 Aug 2009 02:42:53 -0000
@@ -209,6 +218,13 @@ function hosting_client_validate(&$node)
+  if (!is_numeric($node->quota)) {
+    form_set_error('quota', t("Quota figure invalid."));
+  }

could also check for negative values

+++ hosting/site/hosting_site.module	1 Aug 2009 02:42:53 -0000
@@ -374,6 +374,14 @@ function hosting_site_validate($node, &$
+    if ($current_quota == $set_quota && $current_quota != "0") {

this should be >= $set_quota to work around potential race conditions (say two sites are created at a same time)

+++ hosting/site/hosting_site.module	1 Aug 2009 02:42:53 -0000
@@ -374,6 +374,14 @@ function hosting_site_validate($node, &$
+      form_set_error('client', t('This client has reached his or her maximum quota of sites'));

"this client" could be confusing for users that create their own sites... Maybe just say something more neutral like "reached maximum site quota"?

This review is powered by Dreditor. (THIS is the real reason I pulled an extra review here, too cool project... :)

I've refactored this into a new module "hosting_quota" which provides a fairly simple schema for managing quotas for any resource.

The actual quota's are stored in a new db table which has fields for "client", "resource" and "value". Its assumes the code managing the resource understands the context of the value used by that resource and the validation of the value is done at the application level.

For example quota's for aegir sites or email accounts would probably just be stored as an integer value, while quotas for file storage or bandwidth might be MB or GB.

The main use of this feature is to provide a way to limit the usage of resources by a client so that it could integrate with an ecommerce systems. But it also provides a was to see which clients/sites are using all the resources.

It has a a hook to allow other module to add resource to add a limit/quota on - email is a good example.

In the current patch there is everything required to limit the number of sites a client can have at once and when that limit is reached no new sites can be created for that client. So it should provide at least the features in the patch in this issue.

I have also added two other resources which I think would be useful to manage:

1. storage (ie: disk space)
2. bandwidth

Neither of these actually work - well you can set limits but it wont mean anything as at the moment the module can't determine the current usage of these resources.

Bandwidth might be a little bit of work to calculate, but storage is relatively simple to calculate by adding the size of the database tables and the size of the files directory for a site (I would probably disregard the actual space used by drupal code since it's being shared across multiple clients and with the current cost of storage the amount used by code is negligible).

The drush "status" command might be a useful place to add the disk space usage information for a site and I'm sure if had a look I could put together a patch to add that information there.

But my question is how can I get this information back to the frontend? I'm thinking that it might be something that needs to be done via a scheduled task (when that gets implemented) and stored in the frontend, that way you you don't end up constantly querying the database and file system for the size of everything? It would mean it is a little behind the actual usage, but I probably wouldn't make these quota's hard limits anyway, it would just want to be able to see what clients/sites are actually using. The only action that I might want to take is to notify the client they might need to upgrade to a higher plan or if necessary migrate a site to a different server if its effecting other sites.

I have added an screenshot so you can get an idea of what this shows when viewing the client page.

@univate: thank you for your contribution. Unfortunatly, I have issues with the way this patch implements a more generic quota system. My problem is that it's not hookable. It assumes that admins are going to manually edit the users' quotas. For example, I hardly see how we could hookup Ubercart in there so that people can buy "packages" (n sites, or X GB of bandwidth). I would therefore like the patch you provided to be a /default/ quota system that would hook into a more generic API that would be part of the core.

We should have hooks to get/set quotas on arbitrary resources, and have modules implement those hooks, simply. We just need a layer within your patch and everything will be okay.

Another issue is that you use the form validation hooks to enforce quotas, that's not exactly the right way. There's a hook that does that already:

  $results = module_invoke_all('allow_domain', $url, $nid);

Maybe that's really all we need and your quota system is fine and another quota system should be developped independently for Ubercart.. but I can't help but think there's got to be a more common API there..

I took a look at hook_allow_domain(), but it prints out the message that the domain is already in use, which is not the correct error, as well as it doesn't completely make sense to limit based on the allow domain, not sure if that hook would be used for anything other then site creations? you are asking "Is this client allowed to have another site?" so I think doing the site quota check on the site node validate form is exactly where it makes sense to happen.

I think a quota system in aegir should be as simple as possible and leave a lot of heavy lifting to other modules like ubercart. What i think aegir just needs is a way to:
1) set limits for any resource (sites, bandwidth, emails etc...)
2) allow administrators to view and alter these limits in aegir
3) show the current usage
4) trigger something to happen when limit is reached (although for most resources these triggers are probably best left to other modules or customized methods like the rules module)

In my current patch there is a api layer of sorts to manage quotas for resources - I'm sure it could be improved and happy to take feedback and willing to spend some time making it work right.

As an example if you wanted to add quotas for email accounts under a client, this would be done by:

/**
 * HOOK: To define the resource(s) that has a quota/limit.
 *   - title: Human visible name of the resource.
 *   - description: text shown under the textfield for entering this info
 *   - usage callback: function to call to get the current usages (# of email accounts created for this client)
 */
function myemailmodule_client_quota() {
  $quota['email'] = array(
    'title' => t('Email Accounts'),
    'description' => t('The number of email accounts this client can have provisioned under them at any one time. Set to 0 for unlimited.'),
    'usage callback' => 'myemailmodule_email_usage',
  );
  return $quota;
}

function myemailmodule_email_usage($client) {
   return $number_of_current_email_accounts;
}

/** 
 * Another useful function for other modules would be function to add/update a quota for a specific client/resource:
 */
hosting_quota_save_quota($client, $resource, $value);
/**
 * There is no need for a load quota function as the quotas are loaded into the client object so you just need to call node_load($client)
 */

The way to integrate this with ubercart or another ecommerce system would be to create another module that integrates the resources that have a 'quota' with products/attributes and when a sale occurs update the quota for that client & resource. Ubercart is going to have another major problem and that is that sales occur against a user, while in aegir you probably want them to appear against a client.

This new patch just cleans up a few small issues.

Alright, we had a bunch of discussions with the ubercart people around billing and quotas, and I wrote a small spec in #630336: billing support that i will need to transfer here.

Basically, the implementation proposed here is pretty much right on, although it will need to get a few adjustments. I propose a set of functions and hooks that would constitute the Quota API:

* hook_hosting_quota() declares the resources available in a module
* hosting_quota_get() would return the current quota usage based on the collected stats, based on callbacks defined in hook_hosting_quota() (hosting_quota_get_client_quota() and hosting_quota_get_quota_resource() in the current code)
* hook_hosting_quota_check() compares the stats against the current limit (return false if the limit is reached)
* hosting_quota_set_limit() would set the limit in the database (hosting_quota_save_quota() in the current patch)
* hook_hosting_quota_set_limit() is fired when the current limit for a given resource is set using the function above (could fire up a task to set the quota on a disk, for example)
* hosting_quota_set_usage() would set the current usage in the database for resources that live outside of aegir (like bandwidth) (not in the current patch, requires db changes) this would be used by the stats queue

I like the approach made in the patches of using callbacks instead, because it allows declaring multiple quotas per module. Also, to get stats from the filesystem or elsewhere, we'll need to bring back that ol' stats queue that will send feedback to the frontend regularly.

So I'm keeping this at needs work. At the very minimum, it needs to have comments and follow a cleaner API. Grouping the API functions together would clarify the code. Also, the bandwidth and disk quotas should be removed from the patch if they don't do anything. Finally, the site quotas should be part of the hosting_site.module.

sub

Hi folks,

I am going to be doing a big push on developing ubercart and quota modules for koumbit's aegir deployment over the next few weeks. I will be starting with the basic ubercart deployment we need for our initial service offering, but then I expect to work heavily with the code submitted by @univate. In the meantime I am assigning myself this issue.

Whew. After a few days of sustained development and lots of talk with @mvc and @anarcat, here is my working prototype of the hosting_quota module, as a seperate module in Koumbit's git repo :

http://git.koumbit.net/?p=aegir_kt/quota.git;a=shortlog

Once the current work the active branch of hosting is finished, I will refactor this into a patch to the main hosting module.

API Documentation

Functions

These are all defined in hosting_quota.module

hosting_quota_get($resource)

Get info about a particular resource.

hosting_quota_get_usage($client, $resource, $start = NULL, $end = NULL)

Get usage info about a resource for a given client, by calling that resources usage hook.

hosting_quota_resource_render($resource, $usage)

Render a given resource, by calling its rendering hook.

hosting_quota_set_limit($client, $resource, $value)

Sets the limit for a given clients quota.

hosting_quota_check($client, $resource, $start = NULL, $end = NULL)

Convenient function to return a boolean, true if the client has not reached their limit, false if they have.

hosting_quota_get_all_info($client, $start = NULL, $end = NULL)

Retrieve all of the info for a given clients quotas, within a declared period (defaults to the last month).

Hooks

Hook definitions are documented in hosting_quota.hooks.php. A partial usage example was created in hosting_site.quota.inc

hook_hosting_quota_resource()

Returns an array containing info about each resource provided by a module.

hook_hosting_quota_get_usage($client, $resource, $start, $end)

This hook should return an integer that can be compared to the set limit.

hook_hosting_quota_resource_render($resource, $usage)

This hook should return a human readable version of the usage for this resource

looks good, doesnt interfere with anything.

i say commit it to HEAD and leave it disabled.
that way we can build on it and extend in the future.

we might need this for ip pool management.

also, not just clients get quotas.

Here's a first patch, but I reworked it a bit after talking to @anarcat into the patch in #21

Here is a reworked patch that makes the necessary changes to hosting_site

Committed as http://git.aegirproject.org/?p=hostmaster.git;a=commitdiff;h=0c308ce1747...

Cool stuff...

One things I noticed:

+++ b/modules/hosting/site/hosting_site.quota.inc
@@ -0,0 +1,32 @@
+function hosting_site_hosting_quota_resource_render ($resource, $usage) {
+  return $usage . ' sites';
+}

Should probably have a t() around it.

function hosting_site_hosting_quota_resource_render ($resource, $usage) {
  return t('@usage sites', '@usage' => $usage);
}

Also the idea in #19 of making this more generic so other things can have quota would be interesting, e.g. being able to define limits to the number of sites on a server or a platform.

Powered by Dreditor.

Thanks, I will try and correct those poorly translated strings

For the attaching quotas to other objects, I was hoping we could discuss it around drupalcon copenhagen. I am interested in it, but would like to push forward with the billing support work based on this quota stuff for now.

We've committed a patch for that translation, thanks.

Can we open seperate issues now for any other bugs that come up as well as the quotas supported for object X feature?

this is fixed. the patch is already committed.