Finite state machine (FSM) for issue status

Yeah, there are ways that restricting the allowable transitions would help a lot. The existing permission-based thing is fairly wonky, and, AFAIK, not really used, since it's either too clumsy and/or not powerful enough. However, this brings up a few questions for me:

A) It starts to sound an awful lot like what I understand the workflow module is trying to be. Perhaps it's smarter to try to integrate with that, instead of reimplementing it ourselves. I've never actually looked at that module, so I have no idea if it's even possible to tie into it in a meaningful way for issue status values like this. There's also a workflow ng module, but that appears to have been abandoned at the 5.x version...

B) I haven't thought about it at all, but what would the admin UI look like for defining the transitions (and permissions) for the state machine? Perhaps screenshots of other tools that implement something like this would be helpful to get an idea. A giant table of checkboxes for every possible transition seems unworkable. Plus, it's not clear how you associate permissions with each checkbox. Yikes...

C) Seems like such a system would completely replace the existing permission system. How would the upgrade path work for sites that currently use the permission system? Not a show-stopper, but something to consider.

D) While this would be cool for many cases, it's definitely not needed all the time. Can we design/implement this in such a way that it's all done in an external add-on module? I.e. make whatever changes we need to make to project_issue to provide hooks and API calls that support something like this, but then put the actual FSM, admin UI, etc, etc, in project_issue_status_fsm.module (or whatever we call it) so folks that want it just turn it on, and folks that don't avoid the extra code and complexity...

Anyway, sounds cool. ;) I certainly have no plans to implement something like this any time soon, but good luck if you decide to do it yourself -- I'd certainly be willing to help you figure out how to do it, answer design/implementation questions, etc.

Cheers,
-Derek

WTF is FSM?

Thanks for your thoughts, Derek. These are all excellent questions.

I did not know about the Workflow module. I will see if it has any ideas suitable for borrowing.

The UI is a beast. I envision something I call a bidirectional expandable tree view: First the admin picks whether they want to think in terms of inputs states or output states. (These are the same states, he's just deciding which end he wants to work from.) The admin can switch between input-->output and output-->input trees at will. That's the bidirectional bit, purely for convenience. Suppose just for example he chose the output-->input view because he is thinking "how can users get here?" rather than "where can users go from here?" The module would present the list of (output) states. Each state is a node in a tree. Clicking on that node expands a subtree, in which each node is an output state. That output state has a list of roles (very similar to Drupal's existing admin/users/permissions) which a checkbox for each role to be granted the permission. The ability to expand/collapse the tree is what makes this workable. The data storage is (at most) NxNx(M-1) where N is the number of states and M is the total number of roles (including the admin, which can do anything, so we don;t need to store admin permissions). The attached figure is crude but conceptual. Note that everyone is allowed to loop any state back to/from the same state, so this could be visually suppressed to reduce confusion.

Indeed, this system would either be turned off entirely or completely replace the existing permissions for issues. But never both. So an external module would be the preferred implementation. Obviously a lot of people don't need this enough, or it would already exist. :) The default would be to use the existing system for teams who do not need this complexity.

It will be some time before I can even dream of implementing this myself. This would be pretty ambitious for a first module! The good news is, thatgives us plenty of time to hear what users want. Maybe there's a better way.

Good chatting with you. Thanks for sharing your ideas.

Thanks,
Carlos

@aclight: Sorry for the CS jargon. ;) FSM == "Finite State Machine":

http://en.wikipedia.org/wiki/Finite_state_machine

@cmundi: Generally, I'm opposed to things that act like permissions which aren't actually permissions. For example: #110242: make access to filters first class permissions. ;) So, it makes me nervous that the UI you propose lets you select stuff per-role in a place that has nothing to do with the core role/permission UI. Furthermore, it seems like a GIGANTIC pain in the butt if you add more roles to your site and then have to go back through this monster expand/collapse UI to add the right state transitions for each new role. Maybe there's no way around that problem. Also, I don't see how your proposed UI would degrade at all without JavaScript. Yikes. ;)

Anyway, we're probably jumping a bit too deeply into the cosmetic details for this stage in the game. But, as people chew on this proposed feature, I'd like them to start now dreaming up a reasonable UI, since, from my perspective, that's going to be one of the worst parts of this. But, we still need to figure out how to implement this in a "foreign" module -- what API changes are necessary to project_issue to support an external system like this?

@dww: Thanks. The first Google hit for "FSM" is http://www.venganza.org/, and I didn't think that made much sense.

@dww: I share your interest in keeping "site user permissions" separate from "issue management permissions". But given that there are multiple roles common to most software teams big enough to need a tracker, some notion of "permissions" is central to the motivation for the FSM rules. Without permissions, anyone can do anything, by going through multiple fsm transitions until they get where they want. In which case, the fsm is debatably useless and I would not suggest it.

In my experience, issue management roles are spelled out up front, and the tool has to conform, not the other way around. Just as in drupal at large, users may be added or removed, but roles are pretty much fixed upfront. No change there. But even if rolls are added or removed, so what? The same way I have to update/verify user permissions in drupal every time I add a module... and every time I add/remove a group in drupal... I need to set its permissions. I see this the same way.

And I will offer a counterpoint. I think it might be more of a nightmare to have different "classes of groups" than to simply use the drupal groups. Separate classes of groups (one for drupal@large and one for issue management) invites inefficient use -- to put it mildly. I am biased by experience on this point but could be persuaded.

I agree with your assertion that the view and the controller should come after the model. The controller is almost trivial. The view is problematic as you point out. I've tried to outline the model and do not intend for my roughview to be taken too literally. Although that is how some systems look, albeit implemented in native code.

Your point about javascript went over my head. I'm interested. Please elaborate.

Carlos

@aclight: Sorry about the obscure lingo. Wikipedia is better for some things than others. I was a mathematician in a past life. :)

@cmundi re: javascript: I think the point that dww was making was that it would be possible to use javascript to make this monster of a matrix easier to use, for example by expanding/collapsing groups, or switching the rows and columns of the matrix, etc. However, in Drupal, we typically try to make functionality not dependent on whether or not the user has javascript enabled. A lot of us completely disable javascript in our browsers, or use plugins to selectively do so (eg. NoScript for Firefox). We take pride in the fact that at least all functionality in Drupal core can be accomplished with javascript disabled, and typically it is not that much more difficult than when it is enabled. So for this monster matrix, the question would be how to code it so that it will work, and preferebly be somewhat user friendly, if javascript is disabled.

One thing to keep in mind here--if this monster FSM matrix of permissions/roles/statuses is on a separate page from the regular Drupal permissions page (which it most likely will have to be), there's another potential usability problem. Roles can be restricted on to what extent they can view, modify, and delete project and issue nodes, and those permissions would (should?) supersede any "permissions" to move an issue from one status to another. But if permissions on view/edit/delete of issue nodes was somewhat restrictive (ie. not all roles could view or edit issues, or if commenting was restricted to certain roles) it might make the matrix even more confusing, since an admin could allow certain transitions for a role but a user with that role might not be able to, for example, view the issue in the first place.

If an additional node access control module were used on the site that makes this problem even worse.

@aclight: Thanks for your explanation re: JavaScript. I am 100% on board. I'm not a fan of JavaScript and I admire the Drupal community for the make-it-optional approach. That's one of the things that is attracting me to Drupal for personal projects actually.

You raise a very good concern about the interaction of node permissions, comment permissions, issue permissions and and FSM permissions. This will require careful design. I think you are on the right path, suggesting that node permissions should supersede issue permissions. One possibility (not perfect but might work) would be to make the issue permissions expressly conditional on have modify permission for comments. If a user can modify a comment [or issue] (because they have permission to edit their own comments/issues or because they have global comment/issue editing permission) then they can change the issue state. But if they cannot modify the issue, then their FSM permissions are moot. The fsm module (if activated) could detect this and display a suitable informative message. This is actually a nice result of using the comment system for issue status. So you've convinced me that I was wrong in my previous concerns about using comments for issues!

Carlos

Ok. So I'm over at api.drupal.org reading up so that maybe I can come up with strategy that goes with the flow of drupal. Whew! Lots to learn! It may be a while before I have anything and I expect to make some false starts.

The more I think about this... Flying Spaghetti Monster might be a pretty good name for this thing after all. :)

Carlos

@dww: I installed the workflow and workflow access modules. The former defines the states and allowed transitions. The latter grants role-based permissiosn to the transitions. True, conceptually, it is almost exactly what we have been discussing. Workflows can be defined and associated with any content type. Pretty cool. But it has three problems:

1. There is no identity transition.
Transitions from a state to the same state seem to be excluded.
This is potentially problematic when updating an issue without changing state.

2. Workflow states are stored in a table managed by the workflow module.
I don't see an existing way to "link" the workflow states to project_issue states.
The authors might not have anticipated interactions between modules.

3. The UI is extremely unwieldy.
See attached screenshot from my site.
My example has only seven states and three roles. This UI scales very poorly, as we anticipated.

The first two problems seem pretty easy to solve. The third one still seems hard to me. The only other idea I have for the UI problem -- and it's not a great idea -- is to give each input (or output) state its own page. That gives N pages, each of which is an Nx(M-1) matrix. This would have the advantage of allowing us to present a sequence of relatively compact "tables" (one per state) in which each cell is a simple checkbox. Namely each table is a 2D slice of the 3D matrix.

Reading on the workflow group, I see quite a few posts about shortcomings which even I can already see how to fix pretty easily. For example, people want an API for "given an input state, what are the currently allowed output states for a given user?" It's pretty obvious how this would be useful in project_issue. And the answer is a query which is at worst O(N).

Workflow also has some other nice features like actions for email notifications of transitions.

Anyway, I'm going to look over the code. Even if I can't reuse it, I'll learn a lot.

Carlos

The first two problems seem pretty easy to solve. The third one still seems hard to me. The only other idea I have for the UI problem -- and it's not a great idea -- is to give each input (or output) state its own page. That gives N pages, each of which is an Nx(M-1) matrix. This would have the advantage of allowing us to present a sequence of relatively compact "tables" (one per state) in which each cell is a simple checkbox. Namely each table is a 2D slice of the 3D matrix.

You wouldn't necessairly need to do each slice of the matrix on its own page would you? Couldn't you just have multiple tables on one page? At some point the page would get to be pretty long, but I think that would still be better than having them be separate pages. Alternately you might at least use some javascript if you had the tables on separate pages so that a full page reload wasn't needed to change which table is being viewed.

Also, have you looked into whether or not you can easily theme this unwieldy table provided by the workflow [access] module? This might be an issue where changing the interface at the theme level is really too late, but it's possible that you could do what you want without requiring any changes to the module itself.

Couldn't you just have multiple tables on one page?

That's a good idea. Especially if each table is tagged for quick paging.

Also, have you looked into whether or not you can easily theme this unwieldy table provided by the workflow [access] module? This might be an issue where changing the interface at the theme level is really too late, but it's possible that you could do what you want without requiring any changes to the module itself.

Another good idea. I'm not sophisticated enough yet to answer that question, so it is a good thing for me to think about...

Thanks,
Carlos

Quick updates...

Family summer stuff has taken me away from the keyboard lately (yeah!) but here are a couple notes.

I've been thinking about the earlier discussion about using CCK to do more of this. Might be more feasible than I thought.

Also, I had another thought about the UI. Breaking from drupal convention in the admin/users/permissions... another possibility is displaying a separate matrix for each role. Select a role from a list and be presented with a dynamic view of the matrix for that role. Anyway, I hope to have time to work on this.

Carlos