Hi frameworkers,

I have been doing quite a bit of work on gateway functionality and have have just done txtlocal and clickatell module updates, complete with incoming message receivers.

I would now like to be able to handle delivery receipts from these gateways.

I propose adding the below function to the core framework to allow for handling delivery receipts. It includes a simple set of statuses that may be mapped to the gateway status codes, so that they can be handled on common ground.

Of course, a delivery receipt may not be useful unless it can be related it to a previously sent message. I have enabled use of the 'reference' tag function my gateways, but referencing and relating messages to receipts must currently be handled outside the SMS Framework. This maybe an idea for an optional module that would store meta-data of sent messages, apply a reference code and tick them off as the receipts arrive. I might consider that, but what do you think?

Thanks all,

/**
 * Callback for incoming message receipts. Allows gateways modules to pass
 * message receipts in a standard format for processing, and provides a cut-down
 * set of status codes for common code handling.
 * 
 * Allowed status codes. The receipt receiver function must map the gateway
 * status to one of these codes. The gateway code may also be provided in the
 * $options array.
 * <ul>
 *   <li>0 - Delivered</li>
 *   <li>1 - Invalid message</li>
 *   <li>2 - Delivery timed out</li>
 *   <li>3 - Gateway or routing error</li>
 *   <li>4 - Other error</li>
 * </ul>
 * 
 * @param string $number
 *   The sender's mobile number.
 * 
 * @param string $status
 *   An SMS Framework receipt status code. See comment above.
 * 
 * @param array $options
 *   Extended options passed by the receipt receiver.
 */
function sms_receipt($number, $status, $options = array()) {
  // Execute three phases
  module_invoke_all('sms_receipt', 'pre process', $number, $status, $options);
  module_invoke_all('sms_receipt', 'process', $number, $status, $options);
  module_invoke_all('sms_receipt', 'post process', $number, $status, $options);
}

Comments

aspope’s picture

So, this would define a common set of status codes. The integers in the below code would be better for identifying ranges:

/**
 * Callback for incoming message receipts. Allows gateways modules to pass
 * message receipts in a standard format for processing, and provides a cut-down
 * set of status codes for common code handling.
 *
 * Allowed status codes. The receipt receiver function must map the gateway
 * status to one of these codes. The idea of numbering is so that you can match
 * all statuses above or below a certain integer.
 * <ul>
 *   <li>1 - Delivered</li>
 *   <li>2 - Queued</li>
 *   <li>11 - Invalid message</li>
 *   <li>12 - Delivery timed out</li>
 *   <li>13 - Gateway or routing error</li>
 *   <li>20 - Unknown error</li>
 * </ul>
 * The gateway code and string may also be provided in the $options array as
 * gateway_status and gateway_status_text.
 *
 * @param string $number
 *   The sender's mobile number.
 *
 * @param string $status
 *   An SMS Framework receipt status code. See comment above.
 *
 * @param array $options
 *   Extended options passed by the receipt receiver.
 */
function sms_receipt($number, $status, $options = array()) {
  // Execute three phases
  module_invoke_all('sms_receipt', 'pre process', $number, $status, $options);
  module_invoke_all('sms_receipt', 'process', $number, $status, $options);
  module_invoke_all('sms_receipt', 'post process', $number, $status, $options);
}

Here is an example of a gateway module implementing this:

/**
 * Receive a message send receipt from Clickatell
 *
 * Will generate an $options array with the following variables:
 * <ul>
 *   <li>reference - A message reference code, if set on message send.</li>
 *   <li>gateway_status - A Clickatell message status code.</li>
 *   <li>gateway_status_text - A text string associated with the gateway_status.</li>
 *   <li>receiver - The destination MSISDN number. Clickatell param: 'to'</li>
 *   <li>api_id - Your Clicaktell API ID. Clickatell param: 'api_id'</li>
 *   <li>charge - A credit amount that is charged. Decimal value. Clickatell param: 'charge'</li>
 *   <li>cliMsgId - Same as reference. Clickatell param: 'cliMsgId'</li>
 *   <li>moMsgId - The message ID code that refers to a message originally sent with the 'expectreply' or 'mo' parameter. Clickatell param: 'moMsgId'</li>
 * </ul>
 */
function sms_clickatell_receive_receipt() {
  $number = $_REQUEST['from'];
  $gateway_status = $_REQUEST['status'];

  $gateway_status_codes = sms_clickatell_message_status_codes();
  $gateway_status_text = $gateway_status_codes[$gateway_status];

  // SMS Framework status code (format gateway status to framework status code)
  switch ($gateway_status) {
    case '003':
    case '004':
      $status = 1;
      break;
    case '002':
    case '011':
      $status = 2;
      break;
    case '005':
      $status = 11;
      break;
    case '010':
      $status = 12;
      break;
    case '001':
    case '006':
    case '007':
    case '009':
    case '012':
      $status = 13;
      break;
    default:
      $status = 20;
      break;
  }

  // Message reference tag
  $reference = (array_key_exists('cliMsgId',$_REQUEST)) ? $_REQUEST['cliMsgId'] : NULL;

  // Gateway-specific variables
  $options = array();
  $options['reference'] = $reference;
  $options['gateway_status'] = $gateway_status;
  $options['gateway_status_text'] = $gateway_status_text;
  if (array_key_exists('to', $_REQUEST) && !empty($_REQUEST['to'])) {
    $options['receiver'] = $_REQUEST['to'];
  }
  if (array_key_exists('api_id', $_REQUEST) && !empty($_REQUEST['api_id'])) {
    $options['api_id'] = $_REQUEST['api_id'];
  }
  if (array_key_exists('moMsgId', $_REQUEST) && !empty($_REQUEST['moMsgId'])) {
    $options['msg_id'] = $_REQUEST['moMsgId'];
  }
  if (array_key_exists('charge', $_REQUEST) && !empty($_REQUEST['charge'])) {
    $options['charge'] = $_REQUEST['charge'];
  }
  if (array_key_exists('cliMsgId', $_REQUEST) && !empty($_REQUEST['cliMsgId'])) {
    $options['cliMsgId'] = $_REQUEST['cliMsgId'];
  }
  
  // Invoke the SMS Framework receipt handler
  sms_receipt($number, $status, $options);
}

I have tested this, and it works pretty well.

Anyway, let me know what you think. Thanks!
~ap

burningdog’s picture

Status: Active » Needs work

@aspope, this is a great idea! I need to be able to track the status of individual sms's sent via Clickatell, and so adding function sms_receipt() makes perfect sense. I like that it has invokes 'pre process', 'process' and 'post process' hooks, just like sms_incoming()

I think it's a better idea to use constants for the values of the returned gateway status integers. The e-commerce module has to face the same kind of thing when making payments; payment gateways return certain values, which the contrib gateway modules must

Take a look at how it's done there: http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/ecommerce/e...

So I'd suggest something like this (along with appropriate explanatory comments):
define('SMS_RECEIPT_STATUS_DELIVERED', 1);
define('SMS_RECEIPT_STATUS_QUEUED', 2);
define('SMS_RECEIPT_STATUS_INVALID_MESSAGE', 11);
define('SMS_RECEIPT_STATUS_DELIVERY_TIMEOUT', 11);
define('SMS_RECEIPT_STATUS_ROUTING_ERROR', 13);
define('SMS_RECEIPT_STATUS_ERROR', 20);

Regarding the matching up of delivery receipts with previously sent messages: e-commerce handles the same sort of problem (i.e. how to match up a particular receipt returned from a gateway with a particular transaction) by having an "allocation" column in the ec_receipt table, which tells e-commerce which transaction that receipt belongs to.

So, like you said, this means the creation of a meta-data module which stores the initially-returned status id from the gateway for the sent sms, which will be matched up later when the final receipt status is returned.

Hmmm...does this mean that sms_receipt() is only useful if we also have this meta-data module enabled?

Another thought occurs to me - what if we can directly update the watchdog table upon the return of a receipt status, so that when the user clicks upon a particular watchdog entry for the sms, they can view the initially sent status ("Accepted for delivery" or whatever) and then the delivery status ("Delivered"). Of course, a single watchdog entry only has one time, so if these two status's occurred an hour apart we wouldn't know, but it's a thought.

What do you think?

aspope’s picture

Hi Roger, thanks for the feedback - I like your thinking! I'll address your message in two parts. Part one:

After reviewing the link you sent, I agree that constants are the way to go. The SMS Framework doesn't currently define any constant sets, so we would probably need clearance from the maintainers before this would get in - they might suggest that sms_receipt() should be in a separate module.

Anyway, in the spirit of good doc here are my initial definitions: (it seems that this post is removing my indents...!)

/**
 * Message delivered
 * 
 * The SMS message has been delivered successfully.
 */
define('SMS_RECEIPT_STATUS_DELIVERED', 1);

/**
 * Message queued for delivery
 *
 * The message is in a delivery queue - it may have been delayed by gateway issues
 * or the target MSISDN may be unable to receive SMS (device off, no signal, etc).
 * If the message stays in the queue for too long then it might time out.
 */
define('SMS_RECEIPT_STATUS_QUEUED', 2);

/**
 * Invalid or malformed message
 *
 * The gateway rejected the message because a part of the message is invalid -
 * this could be the headers or the body text.
 */
define('SMS_RECEIPT_STATUS_INVALID_MESSAGE', 11);

/**
 * Message delivery timeout
 *
 * The message could not be delivered within the time limit - the gateway has had
 * the message in a delivery queue and has been retrying periodically. It may have
 * been delayed by gateway issues or the target MSISDN may be unable to receive SMS
 * (device off, no signal, etc).
 */
define('SMS_RECEIPT_STATUS_DELIVERY_TIMEOUT', 12);

/**
 * Message routing error
 *
 * The message could not be delivered because the MSISDN is invalid or the gateway
 * cannot send to the required mobile network.
 */
define('SMS_RECEIPT_STATUS_ROUTING_ERROR', 13);

/**
 * Other error
 *
 * There was an undefined error when trying to send the message. Feel free to define
 * more of these error constants/codes if you need to handle other specific errors,
 * and please feed back to the SMS Framework group.
 */
define('SMS_RECEIPT_STATUS_ERROR', 20);

Let me know if you (or anyone else) thinks of any other codes that we should add as default.
~ap

aspope’s picture

Hi again Roger. This is part deux of my reply (about message-receipt matching):

I had the same thought about the usefulness of sms_receipt() without the ability of matching messages to receipts. My conclusion is that you usually need to be able to link the receipts to messages - otherwise sms_receipt() just gives you blind statistics! So I think that I will work on the message-receipt matching module, which will be completely optional and will have to implement a DB table.

About using watchdog, good idea, but I think that it would be considered bad form to update existing records of a logging table, and also the single timestamp issue that you wrote about would negate the usefulness of this. However, we could write the message reference in receipt records sent to watchdog, which the user could then search to view the lifecycle of the message (sent, queued, delivered). This would require a patch to sms_send_log() to write a message ID.

I will get to work on that module, but let me know if you have any other thoughts on this. Thanks Roger!
~ap

aspope’s picture

Hi again, I have been at work on the message delivery tracking module, when I realised that the receipts hasn't really been accepted in the SMS Framework yet.

Will, it would be great to have some input from you about integrating receipts into sms.module. I also have need for an extra hook in sms_send(), just before the return, so that I can capture the message send status from the gateway response.

Roger, is this module something that you would like to use? If so then I will continue.

Thanks guys,
~ap

burningdog’s picture

Status: Needs work » Postponed

@aspope: I'm not that familiar with the internals of sms.module - I think best to ask the maintainer about that.

>Roger, is this module something that you would like to use?
Not right now - I did a project some months back involving Clickatell, but the lifespan of the project was only 3 weeks. This kind of functionality is probably something needed at some stage, especially for reporting purposes - but I personally am not in need of it currently. Fine if I mark this as "postponed"?

So if it's useful for you to keep working on it, go for it! Alternatively, wait until someone else who has a dire need for message receipts from Clickatell discovers this thread and asks politely for the feature to be implemented :)

Sorry for the late reply, btw - I changed my email notification settings and haven't been checking the site for updates to things I'm tracking for a while now.

aspope’s picture

Ok @RogerSaner, I don't have any particular desire to keep working on it. But if there was a need I would have happily done so.

Marked as postponed :-)

PS. I haven't figured out how to get email updates for these issues at all, so apologies for the late reply.

burningdog’s picture

@aspope: no problem, maybe someone will pick up this issue sometime in the future.

As for email updates, I don't get them either - I periodically go to My account -> Tracker (I've bookmarked it) to see if there's anything new. Your link would be http://drupal.org/user/431955/track

aspope’s picture

@RogerSaner, FYI I just found that in the Issues for SMS Framework page there is a Subscribe link, where you can subscribe to All or My issues. It works for me - I am now getting immediate updates by email.

walterheck’s picture

Alternatively, wait until someone else who has a dire need for message receipts from Clickatell discovers this thread and asks politely for the feature to be implemented :)

That someone just came along, and I'm asking politely. I'm in dire need of the functionality for a site/service I'm launching per October 1st. I'm more then happy to fund the development to get it implemented super-fast (and of course all code goes back into this module).

@aspope: are you who I should be looking at, or can you recommend me someone that can help me on short notice? I just had a freelancer walk out on me with 0 lines of code in over a month (luckily he realised he f%^ked up and didn't charge me, but i still lost a lot of time :( )

Great work with this module guys!

aspope’s picture

Hi @walterheck,

Take a look at the 6.x-1.x-dev release. You will find that receipts are implemented in the modules there, and sms.module provides a hook that you can implement for processing receipts.

I was hoping to have a message tracker module ready by now (that would process receipts for you) but I haven't had time to iron out the kinks just yet.

Let me know if you need any info about how it works. I haven't written any wiki docs, but the code documentation should be good enough to help you implement this feature.

walterheck’s picture

Issue tags: +sms, +clickatell, +callback

@aspope, thanks for the quick response.

I've been looking into the code a bit, looks like most of what I need is already there.

All that I need to have added is the keeping of a balance for each user. That means that when a callback is made, the module looks up the user that has that specific mobile number (mobile numbers for drupal users are available when the sms_user submodule is enabled) and decrease it's balance by the number of credits used returned in the callback.

Additionally, the metadata that is returned by the callback function needs to be stored in a table so we can do detailed reporting in a later stage.

As we have a separate application sending the original message, I don't need the mapping of callback to original message for now.

Does this make sense? Also, can I sponsor someone to implement this for me? I'm now looking at elance, but i'd much rather go with someone who already knows this code..

Thanks for the great work you guys do with this module!

walterheck’s picture

Well, the elance way failed miserably :(

I happen to have done a bit of module development recently, so I'm now more comfortable with the idea of doing this myself. I think what I will implement is a new submodule that creates a logging-table and stores the data coming in through hook_sms_receipt(). Then I'll also add a field to the user profile that shows the current balance.

Let's see if I can implement this in a weekend..

univate’s picture

Version: 6.x-1.0 » 6.x-2.x-dev
dpi’s picture

Status: Postponed » Closed (outdated)