billing support

19:14:55 <@anarcat> mikejoconnor: basically, we'll need UC support to be generic enough so that people can be billed on different packages, both in terms of bandwidth, disk space, number of sites, which profiles/platforms available, etfc
19:15:15 <@anarcat> mikejoconnor: basically, UC would become one implementation of our quota system (which has yet to be written)
19:15:35 <@anarcat> http://drupal.org/node/533330
19:15:39 <mikejoconnor> anarcat: so basically we need to associate recurring, with quota's
19:15:45 <@Vertice> yeah
19:15:50 <@Vertice> also. aegir has the concept of clients
19:15:56 <@anarcat> which are nodes
19:16:04 <@anarcat> to which users are connected, through an ACL system
19:17:03 <mikejoconnor> anarcat: how long before you have a working quota system?
19:17:29 <@anarcat> mikejoconnor: the issue above has a pretty good basis for discussion, but not much more... short answer: i don't know
19:17:47 <@anarcat> mikejoconnor: kind of chicken and egg here: i'm not sure how far we can go without knowing how UC works
19:27:26 <mikejoconnor> anarcat: I think it might be semi-simple.  Here's how I think it would work out of the box.  1. we have a quota module, which keeps track of stats per site node.  2. we have a client(already supported?)
19:27:30 <mikejoconnor> is a client a node?
19:27:37 <@anarcat> i have a "Drupal" package with attributes like "Type (aquia, managing news, open atrium, ubercart)", "Bandwidth (500G, 1TB, etc)", "Disk space (...)", "Number of allowed sites (N)" that would affect pricing
19:27:41 <@anarcat> mikejoconnor: yes, a client is a node
19:28:09 <mikejoconnor> anarcat: would you be selling client types, or would you be selling sites?
19:28:17 <@anarcat> we would be selling site packages
19:28:24 <@anarcat> like "you are allowed to have N sites"
19:28:38 <@anarcat> and "you have 500GB bandwidth in your monthly quota for all your sites"
19:28:44 <@anarcat> we could also have quotas like "N domains"
19:28:52 <@anarcat> or "N email addrs"
19:28:54 <mikejoconnor> anarcat: so what we need to do is add support to uc_recurring, to do grouping by customer
19:29:05 <@anarcat> ah
19:29:06 <mikejoconnor> and add prorated support
19:29:16 <@anarcat> i don't actually know what that means :)
19:29:20 <@anarcat> but i kinda guess
19:29:24 <@anarcat> as for quotas
19:30:00 <@anarcat> we need to have a separate "statistics" module that would collect stats from various sources (disk quotas, logfiles, aegir database counters, etc) and then the quota module would compare that with
19:30:14 <@anarcat> either hardcoded quotas or quotas "bought" by the user through ubercart (if that is installed)
19:30:15 <mikejoconnor> currently, as far as I know, recurring works per order. We don't want to charge a customer 42 separate transactions if they have 42 sites
19:30:23 <@anarcat> that's the basis of my quota spec, the way i see it
19:30:32 <psynaptic> we need to be able to sell single sites, just thought I'd mention that
19:30:46 <@anarcat> psynaptic: that's easy: it's a "one site package"
19:30:53 <psynaptic> cool
19:31:08 <@anarcat> mikejoconnor: basically, if a user has 42 sites, he needs the "50 sites" package
19:31:11 <@anarcat> that costs X$
19:31:19 <@anarcat> the issue is how to switch packages
19:32:09 <mikejoconnor> so the idea is you would sell a site, and it would add it to the users subscription list, and bill them a prorated amount when they add the new site.  At the end of the billing cycle, it looks at all the sites they have ordered, and does one order, with a line item for the cost of reach site they ordered, and does one transaction
19:32:13 <@Vertice> the immaturity of uc_recurring is what stopped us before
19:32:40 <@Vertice> mikejoconnor: that's basically how it needs to work
19:32:42 <mikejoconnor> Vertice: we moved uc_recurring out of core so it could mature a bit faster
19:32:46 <@Vertice> yeah
19:32:49 <@Vertice> good stuff
19:32:49 <@Vertice> =)
19:33:00 <@anarcat> what is the "users subscription list"?
19:33:09 <@anarcat> but yeah, i think it makes sense
19:33:27 <mikejoconnor> anarcat: something that doesn't exist right now, but should
19:33:29 <psynaptic> well, this is the bit that needs creating to make it all work
19:33:49 <@anarcat> sell one to N site, then add billing if site usage (M) goes beyond N then at the end of the billing cycle, consolidate on M sites
19:34:17 <@anarcat> i'm not sure i understand the users subscription list concept
19:34:22 <@anarcat> would it sit in UC or aegir?
19:34:26 <mikejoconnor> anarcat: in ubercart
19:34:29 <@anarcat> i see
19:34:42 <@anarcat> 19:33:49 <@anarcat> sell one to N site, then add billing if site usage (M) goes beyond N then at the end of the billing cycle, consolidate on M sites
19:34:52 <@anarcat> site can also be bandwidth, disk space, emails, etc
19:35:01 <mikejoconnor> basically a subscription list would integrate into uc_recurring, basically grouping all of your monthly subscriptions into one transaction
19:35:15 <@anarcat> oh i see
19:35:18 <mikejoconnor> anarcat: so then the uc/aegir integration would be something like this.
19:36:10 <mikejoconnor> we would integrate with the quota system to add a price field for each item we monitor.
19:37:08 <mikejoconnor> and every month we would iterate over the subscriptions, gathering the quota info to determine the total cost of each site.  Group it on one order, and do one transaction
19:38:09 <mikejoconnor> anarcat: this is actually a really good use case for uc_recuring
19:38:55 <mikejoconnor> anarcat: it would be much easier in ubercore
19:40:17 <mikejoconnor> or at least with the theoritical concept of what a product is in ubercore
19:40:45 <mikejoconnor> univate: any thoughts?
19:41:05 <mikejoconnor> anarcat: univate is the current maintainer of uc_recurring
19:41:51 <univate> trying to read back to understand questions....
19:42:48 <@anarcat> ubercore?
19:43:15 <@anarcat> so that makes sense
19:43:42 <@anarcat> it's flexible enough to allow both for "quota-based" systems (where you pay for N sites and can be limited to it) or "usage-based" systems
19:43:58 <mikejoconnor> anarcat: we are basically looking back at everything we have learned since we started ubercart, and looking to create a core api for ubercart
19:44:18 <@anarcat> basically, every P period (month, by default) you would look at the quota Q for the resource R
19:44:34 <univate> there is already support in uc_recurring to setup a recurring payment and then at each recurring payment there are hooks that other modules can use to adjust the order that gets charge (ie: it could sum up resources and 
                   charge that or bill based on product select somewhere)
19:44:35 <@anarcat> resource usage is U
19:44:45 <@anarcat> if U < Q then bill price P for Q
19:44:58 <@anarcat> if U > Q then bill price P plus overquota O
19:45:06 <@anarcat> if Q is 0, then it's completely usage-based
19:45:21 <@anarcat> mikejoconnor: a rewrite?
19:45:49 <univate> anarcat, that can be done via the uc_recurring_renewal_pending() hook
19:46:24 <mikejoconnor> anarcat: more or less, but more a change in process/approach.  Currently ubercart tries to do the job of a install profile, module, theme, as well as provide api's
19:46:27 <univate> you need to be using a gateway where uc_recurring as full control over what amount is charge - auth.net CIM works this way
19:46:56 <@anarcat> univate: and not paypal (or maybe payflow would work...)
19:46:58 <mikejoconnor> anarcat: on top of that, we try to make it support everything out of the box.  The result is a lot of code, that is very difficult to maintain.
19:47:05 <@anarcat> mikejoconnor: indeed
19:47:51 <univate> anarcat, you need to be using a gateway where you can change the amount on each recurring billing
19:48:09 <@anarcat> univate: yeah, basically, you need to be able to charge arbitrary amounts
19:48:25 <mikejoconnor> anarcat: so the idea is to change the approach(small core, good api's), change the process(clear goals, with a clear path/roadmap) to encourage community participation, and change the future of ubercart(possible 
                        it's better as an install profile)
19:48:34 <@anarcat> auth.net doesn't require you to keep the credit card number on your site for that, am i correct?
19:48:50 <mikejoconnor> anarcat: correct.  you store a reference to the account info
19:48:56 <mikejoconnor> anarcat: but not the info itself
19:49:12 <mikejoconnor> anarcat: that reference can only be used from you, so if someone breaks in, they don't have anything useful
19:49:27 <univate> exactly, uc_recurring just billings that profile what every the amount is
19:50:18 <mikejoconnor> univate: so if I place 3 separate orders, for 3 separate subscriptions, will uc_recurring run 3 transactions every month, or can it group them to one?
19:51:09 <@anarcat> univate: okay, nevermind, i understand
19:51:21 <@anarcat> (uc_rec just bills the user whatever everything amounts to)
19:51:43 <univate> uc_recurring will run a transaction for each recurring fee
19:52:17 <@anarcat> can we consolidate those transactions?
19:52:42 <@anarcat> i mean
19:52:46 <@anarcat> can we consolidate the recurring fees?
19:52:56 <@anarcat> anyways, i think we have an understanding
19:53:03 <@anarcat> i'm still unsure as to where to go from here
19:53:05 <univate> You could have some logic on top of uc_recurring that detects an existing order for that customer and instead of creating a new order it just updates the recurring fee
19:54:09 <mikejoconnor> univate: what happens if you place 5 recurring fee items on one order?
19:54:18 <@anarcat> you guys mind if i file all this in this issue? http://drupal.org/node/630336
19:54:21 <Druplicon> http://drupal.org/node/630336 => billing support => Hosting, User interface, normal, active, 0 comments, 1 IRC mention
19:54:34 <mikejoconnor> anarcat: have @ it
19:54:35 <univate> at the moment it creates 5 recurring fees
19:54:53 <@anarcat> mikejoconnor: what?
19:54:55 <mikejoconnor> univate: would you be open to modifying that?
19:55:11 <univate> I was taking to someone recently about creating a uc_recurring_order module
19:55:12 <mikejoconnor> anarcat: have at it, filing all this in the issue
19:56:07 <univate> which you wouldn't add recurring fees to products but instead have a recurring options at checkout that would create a recurring order
19:56:07 <univate> which you wouldn't add recurring fees to products but instead have a recurring options at checkout that would create a recurring order
19:56:18 <mikejoconnor> anarcat: sorry.  have at it is equivalent to yes, please do
19:56:48 <mikejoconnor> univate: what about this.
19:57:37 <mikejoconnor> univate: instead of having recurring orders, or recurring fees as individual items, you have a recurring account.
19:58:21 <mikejoconnor> univate: when ever you place an order with a one or more recurring items on it, or when ever you create a "recurring order", it adds an additional item to your account
19:58:46 <mikejoconnor> univate: and starts the monthly process of billing your account total
19:58:48 <univate> no exactly, at the moment the reason uc_rec doesn't try and combine items into the one recurring fee is items can have different billing schedules
19:59:12 <mikejoconnor> univate: yep
19:59:49 <univate> instead the idea was add items to the cart and then when you checkout you can have the option to make this a recurring order, so purchase this same order every month, 3months etc....
20:00:46 <mikejoconnor> univate: however it seems like it might be better to approach the issue as a billing scheduler.
20:01:49 <mikejoconnor> univate: so uc_recurring could hold information for recurring fees.  each recurring fee could have the option of being billed individually, or it could be grouped into a monthly billing cycle
20:02:39 <mikejoconnor> univate: if you decide to place a recurring order, it would simply add a recurring fee to the table, with a "bill separately" indicator.  You could specify an amount, and frequency
20:03:32 <mikejoconnor> univate: doing it that way would allow a site admin to do all sorts of cool things
20:03:52 <mikejoconnor> univate: I should probably sketch out what's in my head.
20:04:33 <mikejoconnor> univate: btw, the recurring order would be really useful for one of my current clients.
20:04:54 <univate> it seems like you are adding an extra level of complexity that isn't needed, since you are not really grouping recurring fees into the one order you are needing to group products into the one recurring fee
20:05:06 <univate> products is what you are purchasing...
20:06:15 <mikejoconnor> depends on how you look at it.  For this usecase, yes, it's a bit more abstracted than it would need to be, however I'm thinking that it would be more useful in the end than writing an individual implementation of  uc_recurring for simple usecase
20:08:00 <mikejoconnor> usecases that is.
20:08:33 <mikejoconnor> i.e. yes, more engineering up front, but it would also be a bit easier to extend.  however I am not nearly as experienced with the uc_recurring module as you are
20:09:12 <mikejoconnor> univate: I should probably familiarize myself with it a bit more, and then make suggestions
20:10:50 <univate> to me what would make sense from a billing application is that you add items to the clients account as they incur them and then trigger a bill to happen on a certain date which would sum up those items and charge them  the total
20:11:13 <mikejoconnor> univate: exactly
20:11:41 <mikejoconnor> univate: so you can schedule billing events, and you can add charges to an account
20:13:16 <mikejoconnor> univate: it also seems like a billing event should be able to support a specific charge, incase someone wants to stick to a separate monthly transactions per order schedule, rather than a monthly grouping

So this chat was a good occasion for me to reflect on our requirements and on our side of the implementation. Basically, what we need on our side is:

* a quota system - maybe #533330: Quota API is a good basis
* a statistics system - this will be necessary to collect information that is outside of aegir (bandwidth, disk usage and so on). this would work by collecting information from the outside and file it in the Aegir database. then that information (and information that is already in Aegir like the number of sites or which platforms/profiles are available) would be available through hooks.
* a billing system that would regularly check the quotas (Ubercart, for now)

It would look something like this:

                               Aegir                                                              
                 -----------------------------------                                              
                /                                   \                      +--------------+       
                +-----+                       +-----+                      | Ubercart     |       
                |stats+-----------------------+quota+----------------------+ uc_recurring |       
                /-+---\                       +-----+                      | ...          |       
               /  |    \                         |                         +--------------+       
           hook_hosting_stats()              hook|hosting_quota()                                 
             /    |      \                   hook|hosting_quota_set()                             
            /     |       \                  hook|hosting_quota_check()                           
+----------/ +----+----+ +-\-------------+       |                                                
|disk space| |bandwidth| |number of sites|      ... same as stats                                 
+---+------+ +---+-----+ +-------+-------+                                                        
    |            |               |                                                                
hook_hosting_stats_update()   (direct)                                                            
    |            |               |                                                                
+---+------+ +---+----+  +-------+-----+                                                          
|filesystem| |logfiles|  |   aegir db  |                                                          
+----------+ +--------+  +-------------+                                                          

Some explanations on the hooks declared above:

* hook_hosting_stats() declares the resources available in a module
* hook_hosting_stats_update() is fired regularly to update the current usage (probably by running a task or something, on some modules, like hosting_site, it's a null operation because it's already in the DB)
* hook_hosting_quota() would return the current quota usage based on the collected stats
* hook_hosting_quota_set() is fired when the current limit for a given resource is set (could fire up a task to set the quota on a disk, for example)
* hook_hosting_quota_check() compares the stats against the current limit

I'm unsure, at this point, if it's relevant to have a separate "stats" module from the quota system. I don't see the gain in keeping this separate. But I drew this nice graph so I wanted to put it forward. We can probably all fold this into the quota system that would manage resources.

So this would fold into:

* hook_hosting_quota() declares the resources available in a module
* hook_hosting_quota_update() is fired regularly to update the current usage (probably by running a task or something, on some modules, like hosting_site, it's a null operation because it's already in the DB)
* hook_hosting_quota_get() would return the current quota usage based on the collected stats
* hook_hosting_quota_set_limit() is fired when the current limit for a given resource is set (could fire up a task to set the quota on a disk, for example)
* hook_hosting_quota_check() compares the stats against the current limit (return false if the limit is reached)

The algorithm for checking the quotas would be:

* quotas are checked every P period (e.g. a month)
* R is the resource, with a Q quota and a U usage this month
* A is the amount paid for the monthly Q quota, O is the overquota charged when the quota is exceeded
* the the amount charged every month is: A + (U-Q)*O

If A == 0, then the scheme is usage-based: that is, people only pay for what they use.

If A > 0, then the scheme is quota-based: that is, people pay a regular fee and (optionnally, if O > 0) an extra when they overrun their quota.

I believe that system is flexible enough to accomodate all needs and is extensible enough to accomodate new ones.

Let's move the quota system discussions back into #533330: Quota API and keep this here for Ubercart interoperability.

I don't think we should leave this as needs review. People that want to try the module should try it out and submit issues in the other queue. This has been clobbering this queue for too long, sorry. :)

I'm sure it's got the publicity it needed by now. ;)