Hi,
this one is going to be a bigger request.

The problem:
Consider you have an input format "Filtered HTML", where blank lines are turned into new paragraphs.
Text written with this input format looks nice in the edit form: Instead of ugly html tags, you see blank lines.

Now you add a wysiwyg editor on top of this format. Say, tinymce.
You write a text with the editor. Invisibly, line breaks go into the textarea with explicit paragraph html tags.
You switch to "disable rich text", and see the ugly paragraphs.
You write more, using the blank line for new paragraphs.
You click "enable rich text", and the paragraphs are gone.

At least, on display, both the explicit paragraphs and the blank-line ones are displayed correctly.
However, in some other input formats, the wysiwyg might be totally incompatible with some of the filters.

Proposed solution:
A warning: The proposal I'm going to make involves a lot of work.

The difficult part:
For every (server-side) input filter, someone may provide a client-side counterpart. This has to work in two directions: From html to user-input text, and from user-input text to html. So, one direction would turn blank lines into paragraphs, and the other direction would turn paragraphs into blank lines (difficult!)

In theory, this only works if the input filter is a bijective function. This is a tough criterion - probably not the case for many of the existing filters.
In practice, if we do this with a non-bijective filter, we might just accept the damage.

Providing this logic would also allow for some nice preview features..
But, it can't be denied this is going to be a lot of work to do for each single filter.

Instead of providing this client-side conversion logic, one can also just say that the filter is incompatible with any wysiwyg. This can be used to deny wysiwyg for some formats.

And, there can be those filters that neither have any client-side couterpart, nor are they explicitly incompatible with wysiwyg. For these, things work as they used to.

Putting it together:
When using the wysiwyg editor, the result does not go into the textarea directly, but into an (invisible) buffer. The buffer can be a hidden textarea, or it can be just a js variable.
Every time that buffer changes (or only on form submit, and when clicking "disable rich text"), the buffer text is converted to user-input, and then written to the textarea.
On the other hand, when the page is loaded, or someone clicks "enable rich text", the text is converted to html, which then goes into the buffer for the wysiwyg editor.

Dealing with non-bijective filters:
It can happen that some of the filter logic is not bijective. At least, this can be detected: If we execute A -> B, then execute the B -> A conversion, then the result should be the same. If not, we can show a warning, or save the unfiltered text, whatever.

-------

As said, this is going to be more work.
But, just providing the infrastructure, this can be a realistic task.

Comments

donquixote’s picture

donquixote commented

Thinking about it, some of this could be solved in an independent module. The api that allows other modules to provide the client-side logic. Then modules like Live and Wysiwyg can use the stack of client-side filters registered with this api.
(currently, Live is using ajax instead)

twod’s picture

twod
Primary language Swedish
Location Sweden
commented

Hi, thanks for sharing your opinion on the matter, and recognizing it would indeed be a lot of work to pull this off.

There are already several plugins (either cross-editor plugins or native plugins for specific editors) capable of handling serverside filters. Most of them are provided by the module providing the filter itself, which is what we try to encourage since it keeps the serverside and clientside code in sync.

For the specific case of the Linebreak filter, there's the Wysiwyg Linebreak module. It implements Wysiwyg's cross-editor plugin API, which makes clientside conversion between linebreaks and HTML paragraphs/breaks possible with TinyMCE, CKEditor and FCKeditor. (More editors will follow when we have time to build support for it.)

Many editors already treat the original textarea or a variable somewhere as a buffer and only convert contents to the output format once disabled, much like you describe. Adding another buffer layer on top of (or rather, below) that would complicate the situation to a point where it might become unreliable for plugins trying to do last-minute changes to it before form submission.

Wysiwyg's API currently provides two plugin methods one can implement, to act upon content while an editor is being attached to a textarea, and after being detached/disabled. This is what the bundled Teaser Break plugin uses to replace the <!--break--> comment - which would otherwise be invisible, or even lost, in WYSIWYG mode - with an image.

Granted, there are a couple of problems with the API which we have not yet been able to sort out, but most simpler bijective filter functions should already be possible to reproduce using it. We're however at the mercy of the editor most of the time. If the result of the function performed by the filter is regular markup, but it's possible to easily detect it in whichever "escaped" format it's stored in while in the database (the "marker" or "filtertag"), it should be relatively easy to go from "marker" to markup. The other way might be more difficult since the user is allowed to interact with the markup once it's in the editor's content area. It might not be possible to prevent things like users breaking the connection between an image and its caption (the very problem Wysiwyg image upload, Insert, Inline, and perhaps Media are facing). I hope we'll be able to give plugins more control over that situation by relaying more selection information to them, so they can prevent user interference and have an easier time finding and parsing markup when turning it back to the "marker/filtertag". There's still the matter of different editors not applying identical rules to structuring the final markup tho, but I have a feeling we can't really get away from that without extreme measures.

donquixote’s picture

donquixote commented

Most of them are provided by the module providing the filter itself, which is what we try to encourage

If I were to create a new API for this, it would allow different modules to provide / suggest client-side filter implementations, with an option to put weights on each suggested implementation. The engine could then choose the one with the biggest weight, or try which of them is the most "bijective". Or provide a configuration page, where the admin can decide about which filter to prefer.
I think this is better than letting them fight with module weight.

Adding another buffer layer on top of (or rather, below) that would complicate the situation

Adding a buffer did sound like the reasonable solution to me. But it might not be necessary, if instead we just to a stack of conversions that is already there.

plugins trying to do last-minute changes to it before form submission

This sounds evil.. any examples?

There are already several plugins

But probably not tied to specific input filters, via a unified API...

twod’s picture

twod
Primary language Swedish
Location Sweden
commented

I think this is better than letting them fight with module weight.

Hopefully, the admin is smart enough to only install/enable one plugin designed to work with a text filter. Anything else is pretty much pointless, isn't it? I mean, if you have one text filter to transform [[img:my_picture_id_42:title=Funny cat:caption=My cat doing something funny:w=100:h100]] to <div class="img_with_caption"><img src="http://www.example.com/files/funny_cat" width="100" height="100" title="Funny cat" /><span class="caption">My cat doing something funny</span></div>, only one module needs to provide an editor plugin to show a placeholder picture in the editor, and that is usually the same module providing the filter and tools to upload/manage the images.
I have yet to see two modules providing plugins for dealing with the same filter in an editor.

If there were, only one of them would be usable at a time anyway. After the first filter has processed the content during rendering, the second one would either find no unprocessed content left to do anything with, or mangle what the first one did. Does it make sense to install two that similar modules or enable two that similar plugins on the client too then?

Adding a buffer did sound like the reasonable solution to me. But it might not be necessary, if instead we just to a stack of conversions that is already there.

The basic procedure when attaching an editor is a single call to someEditor.init({field:'fieldID',settings:"a bunch of settings"}). The editor then fetches all files it needs, picks up any existing content from the original textarea, hides/transforms it, fires off an event or callback to let plugins (and Wysiwyg) do pre-/post-processing on the content, and finally it goes into WYSIWYG mode. There's not much more to it.

The detach process is pretty much the reverse when disabling the editor or detaching the form. I'm not sure what we'd need an extra buffer for since [most] editors let us do our thing during these sequences anyway. For the editors without corresponding attach/detach events/callbacks, we try to override/wrap some of their routines to put our own hooks in, with as little interference as possible to avoid conflicts with future plugins or editor versions.

>plugins trying to do last-minute changes to it before form submission
This sounds evil.. any examples?

The first example that comes to mind is the whole AJAX system in Drupal (and the various AJAX-related contrib modules). When a form gets submitted in the background - an "event" most editors don't get a chance to act on as they don't know about it - Wysiwyg must trigger the editor's routines to generate the final markup and sync it to the original textarea or it'll be lost when the AJAX request completes. Drupal's (or rather, jQuery.form's) AJAX form submitter must take the contents of all fields in a form and serialize them for the POST data, something which must not happen before Wysiwyg has had a chance to force the editor sync first (This is why the "Add another field"-buttons can cause data loss in D6).

Other modules might want to hook into this sequence of events either before the request is made, or when processing the response and adding event handlers to new elements. Modules which do not integrate with Wysiwyg or the editors might attach their own event handlers to get notified of AJAX requests, which we won't have any control over. So we need to stick as close as possible to the original event sequences established by Drupal Core so we can make the editors as transparent as possible compared to just the original textarea.

But probably not tied to specific input filters, via a unified API...

I think all the modules I mentioned in my previous post provide a text filter for serverside processing and an editor plugin for clientside WYSIWYG placeholders, with the exception of the Linebreak plugin and our own Teaser Break plugin which work with "filtertags" and content generated by Core filters. All the modules use Core's format/filter system to register and manage the serverside part, and Wysiwyg's hooks to notify the editors of the clientside plugins. The users installs/enables the filters and plugins they need, depending on their use-case.

I think you can even have all of the aforementioned filters and plugins active at once, since they have slight variations in which filtertags they use. I don't know of a situation where that would be prefereable, but it should certainly be possible. If two modules use the exact same filtertags to mark where their rendered content goes, their filters would of course be incompatible. If they do happen to be compatible, place the same type of data at the exact same part of their filtertags, and happen to produce the exact same output based on the same tag, you most likely only need one of them.

How do you suggest the new API should be structured?
You mention that modules should be able to suggest a weighted list of filter implementations. Would these be implemented as clientside editor plugins? Would they be selected based on which filters are enabled for a specific format? Should the suggestions be fetched when configuring an editor profile? How would modules know which implementations to suggest? (Are they given a the current content or some other data to base its decision on?)

The way it works now is pretty simple. Wysiwyg calls hook_wysiwyg_plugin() and hook_wysiwyg_include_directory() (see wysiwyg.api.php) to find out which editor plugins are available. It then presents these under "Buttons and Plugins" when the user configures an editor profile. Based on which plugins were selected, Wysiwyg inserts stylesheets and scripts to load the editor and its plugins and builds a list of settings to pass to them. What the plugins actually do is of little concern to Wysiwyg itself, it just moves content between them without looking at it.
When do you suggest this implementation-suggestion process takes place?

donquixote’s picture

donquixote commented

Hopefully, the admin is smart enough to only install/enable one plugin designed to work with a text filter.

The API would choose from the available plugins for one filter, and never _execute_ more than one in a row.
Whether any alternative plugins are "installed", "enabled", "loaded", and what that would mean, we shall see.
It might be interesting to use alternative plugins as fallback if one does not work, but still only one is executed with an effect.

Typical situation I think of:
Module X is years old, and provides an input format. The module author is asked to support the new input filters api, but is quite unresponsive, even after being provided with a patch.
The same could easily happen with input formats provided in core.
Now the author of the new API thingy (which could be me, or you guys, or someone else) wants to support as many formats as soon as possible, so instead of waiting for these maintainers, he decides to include this support in the API module itself.
One year later, the maintainer of module X finally decides that this filter should rather be included in her own module, where she plans to add a few improvements. At this time, the author of the API thing has become as unresponsive as she was one year ago. Or even if not, some users might not upgrade the API module, for whatever reason.
So, she adds the plugin code, and now there are two implementations next to each other.

With the solution I imagine, the author of the API module would set a weight of -10 to his own implementation, allowing others (preferably the maintainer of X) to provide a new implementation with weight 0.

donquixote’s picture

donquixote commented

When a form gets submitted in the background - an "event" most editors don't get a chance to act on as they don't know about it - Wysiwyg must trigger the editor's routines to generate the final markup and sync it to the original textarea or it'll be lost when the AJAX request completes.

Ah, very true indeed!

The basic procedure when attaching an editor is a single call to someEditor.init(...)

Ok, so then we don't actually need a buffer.
Instead, we could do whatever conversion before the editor is initialized, and then do the reverse when the editor is uninitialized or before form submission.
Or if possible, use the editor's own hooks (as you mentioned), or hijack some internal functions.
Probably this is how existing plugins already do it.

donquixote’s picture

donquixote commented

How do you suggest the new API should be structured?

I thought about something on server side and something on client side.
Server side should mostly deal with configuration settings, and give a chance to include additional js files.
The client side stuff would register the actual logic.

So, it could be like this:

<?php
/**
 * Implements hook_jsfilters()
 */
function mymodule_jsfilters($api) {
  // We have setup specific to the 'myfilter' filter (which is defined elsewhere)
  $api->withFilter('myfilter')
    // Tell the API that any textarea with 'myfilter' needs the myfilter.js.
    // $api will assume that the js path is relative to the module dir.
    ->js('js/mymodule.myfilter.js')
    // Every instance of the filter can have its settings altered, before they are passed as a setting to the client side.
    ->settingsCallback('_mymodule_myfilter_jsfilter_settings')
  ;
}

/**
 * Callback to manipulate filter settings
 * (this could be done with a by-ref parameter instead of return value - can be discussed.
 */
function _mymodule_myfilter_jsfilter_settings(array $settings) {
  $settings_client_side = $settings
  .. // manipulate
  return $settings_client_side
}
?>

Now the real logic happens on client side.
We create conversion functions for the two directions

// mymodule.myfilter.js

$(function(){

  var ConversionPlugin = function(settings) {
    ...
    this.filter = function(marker) {
      var markup = .. // modify the marker html
      return markup;
    };

    this.unfilter = function(markup) {
      var marker = .. // modify the markup html
      return marker;
    };
  };

  var first_run = true;
  // TODO: We only want a way to run this _after_ Drupal.jsfilters is made available as an object.
  // Doesn't have to be a behavior.
  Drupal.behaviors.mymodule = function() {
    // TODO: Better way to run only once..
    first_run = false;
    // Register our plugin, with a weight of -10.
    Drupal.jsfilters.registerConversionPlugin('myfilter', function(settings) {
      // Every instance of the filter can have new settings, and thus requires a new instance of the ConversionPlugin.
      return new ConversionPlugin(settings);
    }, -10);
  };
});

That php and js might still look a bit ugly.. forgive me.

donquixote’s picture

donquixote commented

The way it works now is pretty simple.

Maybe I will think the same, when I take a closer look on it :)

When do you suggest this implementation-suggestion process takes place?

As mentioned above, every module can register its own plugins on client side, with a weight setting. Then the API can choose based on the weight, and execute that one.
This is just a proposal, it could be done differently.

twod’s picture

twod
Primary language Swedish
Location Sweden
commented

The API would choose from the available plugins for one filter, and never _execute_ more than one in a row.

As a site builder, I want to be very sure about exactly which modules/plugins run on my site, and I do not enable any I do not need. I think my main question is why I would give the code an opportunity to choose between two similar execution paths on its own, leaving me to guess which plugin ran at which point in time? What if only some instances of identical content samples get corrupted because one of the plugins misbehave in some browsers?

It might be interesting to use alternative plugins as fallback if one does not work, but still only one is executed with an effect.

How would we determine if a plugin works or not? How do we know what it's supposed to do with the content at any given time, so we know if it had an effect or not?
If it throws an exception, we can be pretty sure it doesn't work, but then it should be disabled and fixed. If we disguise that by running another plugin instead we're just hiding a problem and making the user uncertain of which plugin really did the work. It gets really confusing if some plugins doesn't throw exceptions every time, but only when certain content is processed in combination with a certain browser.

... So, she adds the plugin code, and now there are two implementations next to each other.

I think I see what you're saying here, but the use of the terms "format" and "filter" seem slightly confused. If you mean a "format" is a serverside input filter processing content in a check_markup() call, and a "filter" is the type of WYSIWYG editor plugin we've talked about. A "format" is really no more than a named collection of enabled serverside filters which does nothing on its own.

The problem you're describing here is that there would be one filter module doing some processing to the content while it's rendered, but two modules providing editor plugins to "simulate" what the filter does, in WYSIWYG mode. And the question is, which module should have it's plugin simulate in WYSIWYG module what the user will see in the rendered content.

I'd say only the person responsible for updating the modules - and/or the person responsible for determining what types of content are allowed - on a site could decide that. If it's desirable to use the [newer] WYSIWYG plugin, disable the one from the module providing the serverside filter - you won't need it at all. If the filter module's own plugin is good enough, don't install the newer module at all. (Or at least don't enable its editor plugin and it won't put extra load on your site.)
Updating either module (assuming both are left installed, but only one of their plugins is enabled) won't change anything unless you decide the plugin you're using now has been surpassed by the other module's plugin.

Or if possible, use the editor's own hooks (as you mentioned), or hijack some internal functions.
Probably this is how existing plugins already do it.

The cross-editor plugins don't know anything about how we interact with each editor, it just talks to Wysiwyg. Wysiwyg's editor-implementations do wrap some editor functions (or use the editor's API the way a native plugin would) to be able to invoke the cross-editor plugin's methods at the correct point in time, but cross editor plugins won't notice.
Native plugins usually talk exclusively to the editor's API, as they're designed to work outside Drupal as well. Some plugins written by, for example, filter module authors might talk to Wysiwyg or Drupal via their APIs, but they'll obviously not work on say a Wordpress site.

Server side should mostly deal with configuration settings, and give a chance to include additional js files.
The client side stuff would register the actual logic.

Indeed, that is what's it like now. The syntax is a bit different, but the structure is there.

Replace hook_jsfilter() with hook_wysiwyg_include_directory() and hook_INCLUDE_plugin() (or hook_wysiwyg_plugin() if you've got the logic contained as a native editor plugin) and you've got the means to pass plugin meta-data to Wysiwyg.
For settings, a module providing a plugin can use Core's hook_form_alter() to inject its form elements and logic where needed, or start from hook_menu() to get their own settings pages.

The metadata points to a JS file with the equivalents of your filter() and unfilter()

Drupal.wysiwyg.plugins.myPlugin = {
  // Wysiwyg passes the [HTML] content from the original textarea and expects to get it back with placeholders instead of filtertags.
  attach: function(content, settings, instanceId) {
     return content.replace(marker, markup);
  },
  // Wysiwyg is about tp pass the [HTML] content back to the original textarea and expects to get it back with filtertags instead of placeholders.
  detach: function(content, settings, instanceId) {
    return content.replace(markup, marker);
  },

  // Cross-editor plugins are currently required to provide a toolbar button.
  // This method is invoked when the button is clicked.
  // I'm going to work on making the button optional at some point...
  invoke: function(data, settings, instanceId) {
    alert("We don't really need a button for this filter plugin.");
  },

  // Wysiwyg passes the selected DOM node and asks you to return TRUE when your toolbar button should be highlighted.
  isNode: function (node) {
    return false;
  }
}

How the equivalent "filter plugin" would look as a native plugin of course depends on the editor it's written for. The cross-editor API is currently loosely based on TinyMCE's plugin API, so in that case the flow would be similar.

donquixote’s picture

donquixote commented

I'd say only the person responsible for updating the modules - and/or the person responsible for determining what types of content are allowed - on a site could decide that.

Makes sense.

If it throws an exception, we can be pretty sure it doesn't work, but then it should be disabled and fixed. If we disguise that by running another plugin instead we're just hiding a problem and making the user uncertain of which plugin really did the work.

Caught me. Stupid wild thinking.

--------------

The remaining question is, why would i suggest a new API for this, if we already have the possibility of plugins?

As far as I understand, current plugins still need to figure out by themselves,
- which format the wysiwyg-enabled field has.
- which filters are enabled for this format.
- the sequence of those filters, within that format.
- the per-filter settings of those filters, within that format.

Then different plugins have to make sure they run in the correct sequence.

And, those plugins need to hook directly into wysiwyg, so they are not directly reusable for something else, such as a preview module.

With the suggested new API, the API would take care of all those things.
A plugin then only needs to care about one filter with settings, no matter how this filter is part of any format, associated with a wysiwyg textarea, etc.

donquixote’s picture

donquixote commented

Drupal.wysiwyg.plugins.myPlugin

Hm, would this "myPlugin" correspond with one filter?
And, does wysiwyg call plugins based on the filters being configured?
Or is the plugin just called whatsoever, and then needs to figure out by itself if the wysiwyg textarea's format has the respective filters active?

twod’s picture

twod
Primary language Swedish
Location Sweden
commented
  1. -
    - which format the wysiwyg-enabled field has.
    - which filters are enabled for this format.

    Irrelevant to all plugins I've seen so far. Each [cross-editor] plugin operates independently of any other. A counter-example would be TinyMCE's "Advanced Image/Link" plugins, which extend the UI of the "Image" and "Link" plugins, respectively. They do however utilize dependency info and hooks/events to achieve relatively loose coupling. If "Advanced Image" is activated without the "Image" plugin, nothing happens. If a cross-editor plugin would rely on another plugin being active, it could do so by, for example, inspecting the editor profile[s] it's been activated on to know if the relevant plugin is active. If the right conditions are not met it could either remove itself from the list of plugins to enable or fall back to doing nothing or showing a warning message.

  2. - the sequence of those filters, within that format.

    I don't think the editor APIs even allow their own native plugins to affect their execution order, at least not directly. Some editors do allow for specifying dependency information in the plugin meta-data, which does affect loading order. But all the APIs I can name off the top of my head pass content around to plugins in either an undefined order or the order in which they were loaded, for content pre-/post-processing etc. If we end up with plugins which need to be guaranteed to be run before another plugin, we could add dependency info to our meta-data hooks as well, at the cost of slightly more time spent loading/processing plugins.

  3. - the per-filter settings of those filters, within that format.

    All settings information is currently available in the global Drupal.settings.wysiwyg.configs.editorname.formatname or Drupal.settings.wysiwyg.plugins variables. AJAX operations could theoretically update the info held there too.

  4. And, those plugins need to hook directly into wysiwyg, so they are not directly reusable for something else, such as a preview module.

    Wouldn't cross-editor plugins need to talk to Wysiwyg's APIm no matter what it looks like, if they'd like to interact with an editor or even get loaded? Native plugins written for an editor's API knows nothing of anything Drupal. The only Drupal/Wysiwyg thing needed to "present" them to an editor is an implementation of hook_wysiwyg_plugin() for meta-data, which any module could implement.
    Wysiwyg runs no code during previews or rendering, so neither could plugins unless the module implementing it also implements an input filter, which is a completely separate API.

Hm, would this "myPlugin" correspond with one filter?

It corresponds to a plugin, being called when content is passed to an editor and retrieved from it. What it does is up to you. It could simulate the effects an input filter has on content when Drupal renders it, modify/swap/purge tags before database storage, randomly insert smilies, or launch a dialog when a toolbar button is clicked.

Or is the plugin just called whatsoever, and then needs to figure out by itself if the wysiwyg textarea's format has the respective filters active?

In the code I posted, the plugin is internally called "myPlugin", yes.
Does it matter to the editor plugin if a filter is active or not?
If the admin enabled the plugin, it's code runs, otherwise it's not loaded/initialized at all. Maybe you've found Module A which has a decent serverside filter turning names of famous people into Wikipedia links, but an awesome editor plugin with an intuitive interface to let the user pick the correct person in case of ambiguities. It works pretty well but the serverside filter has gotten old and still replaces your [[Famous:Dries_Buytaert]] filtertag with http://en.wikipedia.org/outdatedurl/Dries_Buytaert. This would likely be trivial to fix in Module A, but you've also stumbled upon Module B - a fork of Module A but with additional filter features. Not only does it correctly transform [[Famous:Dries_Buytaert]] to http://en.wikipedia.org/wiki/Dries_Buytaert, but it also show's Dries' picture when hovering the link! Similar to your own previous example, I now want to use the old editor plugin from Module A (which Module B assumes I do since it doesn't implement one on its own) with Module B's serverside filter.

Since the editor plugin and the serverside filter in Module A are independent of eachother, only loosely coupled by the syntax of the filtertag, that is now possible. But if Module A's plugin was not loaded unless Module A's filter was also active, it would never have been possible for Author B to write the module without duplicating the plugin code.