Filter contexts (and fix some small issues in filtering)

BTW I would like to open up my series.module, but unless this issue is remedied, it is not too convinient to install, so I am not goint to add it to contrib CVS.

Since my only commenter suggested that this is a good idea, here is a patch. To provide possibly the best flexibility, I have opted to pass on the full node object instead of just the nid itself. The patch also fixes some small issues. Concluded contents:

• Pass on node object to filters so they can act differently based on node properties (taxonomy classification, node id, etc).
• Convert all types of newlines to Linux ones, so Windows and Mac newlines should not be dealt with in filters
• Fix title.module compose tips (a long due TODO item)

IMHO this should be applied in HEAD and 4.4 too. The patch is against HEAD.

I want to show a lot of stuff in the node body, including links to articles in the same serial, files (images) attached to that node, links connected to that node, etc. These all will be cached with the filtercache (for all users including logged in users), but if I need to do this in a sidebar or something else, I need to do caching separately for each stuff (more code, worse performance).

Check this stuff out: http://weblabor.hu/cikkek/cssalapjai2. This in Hungarian, but you can get the idea. The image showed on the top left is the assigned 'icon' of the article. There are also attached images displayed in the node body (this is a CSS introduction, where there are some illustrations about the box modell). Also there is a list of the other articles in the series. Unless the filtering code knows what node it filters the content for, it cannot get the assigned node icon, the attached images, and cannot determine if the node is part of a series, and what are the other nodes in the same series. You can probably understand that showing the node icon or the images out of the node body would not help much in visualizing what the text is about.

Sites like DevShed, Zend and others have article navigation in the text body, just because this helps driving users' attention to related content. For images and other attached files, this is vital. A lot of sites (eg Sitepoint, newsforge, wired, etc) display ads in the page text (often connected to what you read), this can only be achived if the filter putting the banner ad code in knows about the taxonomy, series or other details about the text.

This is not only convinient for the users but also for the engine, since the filtercache will store the full cached content, so next time we don't need to fetch the node image, the list of nodes in the series, the details about the attached files, but they are all loaded as HTML from the filtercache.

I have no interest in filter profiles, since all our site users can be said to use the same toolset (and/or the special filters we use are restricted by user_access() checks), so I am not entirely sure that holding this improvement back is worth it waiting for some uncertain future development. The user group right merging patch was not turned down because of there are some big plans around user rights, and along these lines, I don't think that a patch introducing node information for filters should not be applied.

I would be happy to reengineer the patch to approach the problem from a different perspective (ie. use some context information variable, and embed the node object in it somehow), but I am more than convinced that the filter needs to know about the node, so it is probably pointless to try and convince me about the opposite :) BTW I think that a lone node object parameter is better then integrating it in some array or another object, since this way someone knowledgable can optimize it to use pass by reference and consume less memory.

Series modules and other modules belong as a filter, because they

• would like to display content in the node area
• they are suitable to be cached with the node itself

This means one cache request per node display and not n+1 cache request per node display (n being the number of filters eager to put stuff into the node area). This currently comes down to either one SQL query or three per node display on my site. The page context is not a solution for this problem, since it will not work for pages displaying multiple nodes (while the filter context would).

Here is an updated patch which takes into account that different contexts could be possible, and provides enough space for your dreaming for that time when you will have enough time to extend possible contexts. It also passes $node by reference. It might be a good idea to pass on what field of the data part the filter is expected to filter, so we don't need to pass on that data two times (first as the $text, and then in the context "data"). But then this should be handled as a special case in all filter modules, and all filter modules should be updated. So discuss first before I put time into providing some patch which will find root level rejection.

The solution offered by killes might work for attachments, but will not work for most of the other stuff. Let me provide some use cases:

• Displaying related banner ads
  • http://msn-cnet.com.com/Thirty+years+with+computers/2010-1001_3-5221124.html?part=msn-cnet&subj=ns_5221124&tag=tg_ns
  • http://builder.com.com/5100-6370-5181395.html?tag=tt

  To print out proper code to display banners, the filter needs contextual info about the text (taxonomy, title, keywords, etc). This way related banners can be displayed at any place where the author desires, or if he does not specify a place, then a default can be pointed at and used.

• Illustration to text
  • http://www.w3.org/TR/CSS21/box.html#box-dimensions
  • http://www.alistapart.com/articles/onionskin/

  This means that you need to upload files (sometime), and you can decide on where you put the file by using filter tags (as you do with any markup - let it be HTML, bbcode, textile or other stuff). Either you reference the images, flash illustrations, etc. by name, or you need to know what node is the text connected to, so you can put in the right attachments.

• Files attached to nodes
  • http://weblabor.hu/hirek/20030513/mysqltalalkozo

  You write about some source code, and you attach the needed files for the user to try it out at home. You can put a nice box containing the list of the attached files automatically, providing meta information about them. Of course the filter needs to know where to put the box, so you have a tag for it, and the filter needs to know what attachments to list, so it needs to know the node id. The example above lists slides (the news item is about a session held by some MySQL people).

• Displaying more information about the author
  • http://www.sitepoint.com/article/defined-functions-coldfusion

  The filter needs to know about the node author to display that kind of information in the node body.

• Displaying links to related content
  • http://www.xml.com/pub/a/2004/05/26/www2004.html (related reading box at the middle right)
  • http://wired.com/news/medtech/0,1286,63655,00.html?tw=wn_1techhead (see also list)
  • http://news.yahoo.com/news?tmpl=story&u=/ap/20040602/ap_on_re_mi_ea/iraq_us_military_3

  The filter needs to know some classification information about the text displayed (ie. taxonomy, node keywords, etc) to select some related content (from the online shop, from another part of the site, or from the same taxonomy). Since it is a filter, it allows authors to place this block where it fits with code examples, and images, so the content is well balanced, and there is no ad overload at any place. Even if the author does not specify the place, the filter can put it somewhere, if it wants to.

I have tried to select prominent sites as examples, so they will not be dismissed :). You will probably be able to realise that most of the example URLs I have listed above fit more than one use case (wired displays banners and related info; sitepoint displays banners and author info while having illustrations too on some pages). It is out of question that all of the above do inject content right into the node body output (with Drupal lingo), so to achive this, we need some method of modifying the body/teaser output, before it gets printed by the theme - if it is a justifiable aim to support these kind of features.

Now the filter system is supposed to get the unprepared node body/teaser as an input, and generate HTML ready for presentation by the theme. Some of the use cases presented above evidently need placement instructed by the author (illustrations at least). I beleive that not only illustrations need that. Imagine that I write an article which starts with long unbreakable URLs (true for this node :). If a program automatically ads author information on the top, or a banner, then there will be a big blank space by the side of the banner / author information, since I have content which does not fit that space. This is equally true if I add a wide chunk of code (pre). So to make the output look better for those authors who care, it is good to provide some means to specify a place for injected content. Those who will not specify could get that injection at some predefined place.

So as I have said, the filter system is supposed to prepare the content, so this does fit there currently. Plus to fill all the stuff above, we need to do a lot of database queries (get more author info, get list of related content, get list of attached files, etc). This is however all (somewhat remote) properties of the node being displayed, so it is logical that we cache it with the filter output as well (via filtercache). This means that whether we have these special features on our site or not, it is just interesting on the first node display, when the filter output is cached (that is on preview), and all subsequent requests will pull the output from cache, meaning that adding all these extras don't not mean any performance decrease problems for most of the requests. You might say that banners should not be wired into the node filtered output. If you look into the current banner implementation, it uses JS to be compatible with page caching, so the same method could be used for filtercached node displays as well.

How would you provide the above features without passing on context information to filters and still not having any compromises in performance?

Chris, filters are used to provide alternate markup (bbcode, textile, etc) options, which does not fit the defition you outlined in the first paragraph. These filters do provide authors and editors with tools to express their intentions.

Chris, the resulting HTML is stored in filtercache and/or the page cache. We should be able to present the bbcode to the future editors of the node (as well as placeholders of where illustration images and other stuff are), so the original text typed to the textarea should stay in the DB.

An alternative would be to rewrite the filter instructions needing context information to include that context info in the view nodeapi hook (invoked before the filter gets executed), so [see-also-list] would become [see-also-list:nid=8000] for example. Then the filter can get the relevant context info right from the filter instruction.

Drawbacks. First, you need to pass on that context info in all filter instructions (might not be that much really). Second, you need to rewrite both the body and teaser, since you don't know what is going to be filtered. This is not a serious issue for those using the filtercache, including me.

It really does not matter that you put in the required placeholders into default places in the view hook or in the filter itself. See my use cases if you don't understand this paragraph.

Please note that the view hook does a kind of filtering and transformation in this case! Which means it becomes a prefilter filter.

I have expected Steven or someone else to jump in and shout that the alternative method I have described is not right. Here are the reasons:

• We introduce another filtering layer out of the filter system, so it is conceptually duplicating what we already have.
• We cannot add HTML to the node directly in the view hook, since the primarily executed html escape or html strip filter built into Drupal will escape / strip out our additions. The text is treated as "untrusted" at the start of the filter process, so we are unable to add HTML. We need to add non-HTML formatted placeholders. This means that we need to implement both the view hook and the filter hook. This makes it a two pass process to add something more intelligent to the text output, since only the filter would be capable of putting the right HTML in (since it is executed after the escaping/stripping).
• So you might say we can introduce a new hook which is invoked after the filter process. That is still problematic, since the filter process is supposed to output stuff ready for presentation. It does add line breaks to the text if it is needed, etc. I am actually working on a filter which would add proper paragraph tags instead of line breaks to let semantics rule. Also my usual argument comes in here: it would not be filtercached.

All in all, the filter system is supposed to present display ready HTML out of some untrusted text portion. Either we plug into the process before the filter system run or after it, we don't fit into the picture, since the filter process finalizes the HTML for presentation. We are doing something which is the task of a filter (just try to push in some more info for the filter to use with abusing the node body and teaser text itself).

It might also be an important point for Steven and others that if we use the nodeapi hooks to add placeholders into the text (or the HTML itself if a postfilter nodeapi hook would be added), then that does not solve the problem of passing on comment or user context info, since the nodeapi hooks are not called on comment or user property presentation.

More opinions and comments are welcome!

After getting accomodated to the view hook moved after the filtering process, it is now possible to do what I wanted to do outside the filter system, and it actually makes the series code more simple (no need to invalidate caches). It is a little bit slower on node display pages, but I can live with it. The reduced code size and complexity however is better for the whole site.

Now that the view hook is after the filtering, I am convinced that I went on the wrong road before. I will shortly update the filtercache module (4.4) to remove this hack and the series module to make use of the 'view' hook (with a patch for 4.4).