Fire isNode on mouseUp and keyUp.

You don't mention which editor(s) you've tested with so I'll write a short summary about those currently supporting cross-editor plugins.

In the TinyMCE and CKEditor implementations, isNode is called whenever the editor detects a selection change. The events we use to call isNode are onNodeChange in TinyMCE and OnSelectionChange in CKEditor. These might not be called on each key event for performance reasons, as mentioned in ckeditor/_source/plugins/selection/plugin.js:

41 // Firing the "OnSelectionChange" event on every key press started to
42 // be too slow. This function guarantees that there will be at least
43 // 200ms delay between selection checks.

Since these events are already triggered on every change the editors are aware of, "pinging" them more frequently than that might not be such a good idea. Each state update could potentially trigger a cascade of events and other updates, which could make it look like the editor (or even everything on the page) freezes as scripts and events can only run one at a time.

The FCKeditor implementation calls isNode whenever the editor wants to know a button's state (the GetState event), which in the end is pretty much the same result as in the previous editors. AFAIK, this callback is the only place in which we can actually change the button state (using the return value). So, unless the editor explicitly requests to know if the state has changed, we can't change it in another event.

Also, when clicking inside that span (assuming the isNode implementation does something like return $(node).is('#plugin-result')), it'll never return true in CKEditor because node will not reference the span nor the p tag. But that's another issue which comes from TinyMCE returning the parent element of a selection if it doesn't exactly match one element, while CKEditor doesn't. Even if we called isNode on each key/mouse event, it wouldn't make a difference there.
In TinyMCE on the other hand, it clearly updates the button state when clicking outside the span tag and still inside the p tag, if the isNode implementation works like the example above.

Do you have a plugin where this is needed?
So far, the plugins I've seen all insert an image (either as a placeholder for complex markup or as the end result, like a smiley) and these only need to change their state when just image element is selected. Some plugin authors have expressed a desire to be able to manipulate the selection, and I can see why, but we'll need to rewrite quite a bit for that to work well though so that's on ice for now.

I only had time to take a quick look at your code, but my first impression is that it would probably have been better implemented as separate native editor plugins rather than a single cross-editor plugin.
The plugin is already interfacing directly with each editor's API and adds its own code to detect and load the editor-specific "sub-modules" depending on which editors are loaded. You'd get all that for free, including the editors' already cross-editor compatible selection handling and event/callback systems, if using native plugins instead.

What I'm trying to say is that it's more work to implement editor-specific plugins via Wysiwyg's cross-editor plugin API than building native plugins simply because it isn't designed to do that. Wysiwyg's API is, as you've already noticed, not very mature when dealing with selections and complex interactions with the editor or the DOM inside and these tasks are better suited for native plugins.

mmm, that's actually something we need to fix. The Drupal community shouldn't waste time with native editor plugins.

@sun, I agree about that, but until our API can support it, building cross-editor plugins that use the native editor APIs could lead to even more wasted time. :(

Back to the topic and my first response: Even if using the native editor APIs, the editor itself will still not trigger events or more often than it does now. It fires with about the same frequency (and in response to the same changes) in most editors, the main difference being which node(s) their API returns when queried.

This is why I've said before that we might need our own handling of selections in Wysiwyg, so that we can always send exactly the same data to plugins in response to editor events, and let plugins modify the selection using a single cross-editor/browser API.

This plugin seems to be doing part of that already, since it's often ignoring the node sent to the callback method and fetches the selection on its own, and that's really something they should not need to worry about.

EDIT: I apologize in advance if the text below seems like a repetitive and stubborn rant. It was not meant to be that but I often end up writing things like after a bad day at work. I do think the points I'm trying to make get through, but I'd like to stress that I'm not trying to be a stubborn no-sayer on purpose and that I understand your design is a compromise of the compromise Wysiwyg currently is. ;)

The only reason that I have any native plugin code in the module is that Wysiwyg API doesn't offer those hooks, however my code itself proves that Wysiwyg could provide them.

Not sure exactly which hooks you talk about. Your code is quite hard to follow when reading it the first time since it seems to add another abstraction layer to plugins. The snippet you mentioned above in regard to spans not triggering isNode (which they do, see below) makes no sense to me at the moment. I could probably figure it out by tracing the code back and forth for a bit longer, but what it actually does and why seems a bit irrelevant when the point of the original post was just that it seems like isNode isn't called often enough.
My response to that is simply that it's already called as often as a change is detected, either in the DOM or the selection. If it's not called, it's because the editor has not detected a change.

I hope you consider adding more plugin methods/API hooks, because while they haven't been needed to this point, it will be hard to provide more advanced cross-editor plugins without more support from you guys.

Yes, we do and I understand the need for this. We've planned to "forward" many more events from the editors to cross-editor plugins in the future. There's a lot of research that needs to be done in order to get this right though, and we also need to make sure what we have works as intended before adding even more to that pile. Note however that when, and how often, events/callbacks are triggered is entirely up to the editor, Wysiwyg is just a transport layer.

About your second point: Like I've said above, I believe isNode is called as often as needed, simply because it's called on every change in the editor, including the one where clicking outside the span in your original post. To test this, I simply set a breakpoint in the Teaser Break plugin's isNode method, inserted a paragraph with a nested span, clicked inside the span, the breakpoint is hit. Then I clicked outside the span but still inside the paragraph, and the breakpoint is hit again.

This is even easier to confirm by looking at the "path/location bar" at the bottom of TinyMCE or CKEditor. It updates to show the currently selected node via the same event that triggers isNode. It does lag behind slightly when using the arrow buttons to move fast between element boundaries, but that's just the intentional 200ms delay kicking in so the editor and browser won't be overloaded.

I think one big problem here is that the "node" passed to isNode is only guaranteed to be an actual node in all editors when exactly one node is fully selected in the editor. A collapsed (just the caret) or partial selection inside a paragraph or span will still trigger isNode, but the "node" passed to it is null. That is why you need to fetch the selection manually, get the ancestor node of the selection and compare that to the criteria you have for determining the button state to return.

If Wysiwyg was to react to the same editor events it does now but instead of just passing a [possible] node reference - retrieved from the editor - to a plugin method, it could instead pass a cross-browser/editor compatible reference to the current selection, you'd have much less work to do when checking if the selection is interesting to you.

Note that even if we were to call isNode on every single event out there, we couldn't guarantee that editors actually update the state of the toolbar buttons (which is what that method was intended for). FCKeditor is a great example of this, as I mentioned above it only "listens" to state changes in the specific callback from which isNode is currently being called.

If what you need is to run code precisely when something changes in the editor, it might even be better to register your own event handlers for the events you're interested in either via the editor's API or jQuery, depending on the event. isNode is at the mercy of the editor as it is the editor's way of asking the cross-editor plugin for information, not the other way around. (At least not how it works now.)