Duplicated summary

I'm not able to reproduce this. Were those nodes created with a separate teaser/summary before Wysiwyg module was enabled?

I confirm that the teaser gets duplicated inside the editor when not using the

and 'Show summary in full view' is not checked.

I ran into the same problem with some of my nodes. Not sure if this is something new in 6.14, but I didn't notice it before that. It only seems to be an issue if the contents of the teaser don't match the contents of the node. In my case I deleted the teaser from the DB and everything went back to normal.

I'm having the same problem. Drupal 6.14/Wysiwyg 6x-2.0. It happens regardless of editor; I initially thought it might be a bug in TinyMCE so I tried FCKeditor, and YUI Editor it happens with these also.

Steps to reproduce:

Create a new Page node
Enter some text (at least two separate lines)
Save
Edit the node.
Preview

The preview looks fine but in the editor panel the content is duplicated and if a user saves from the Preview page they get duplicated text.

Yep. We get the same thing. Just updated the core to 6.14, latest WYSIWYG and TinyMCE 3.2.7.

No amount of form_alter is preventing the teaser from being stuffed back into the content area, which is why it looks like content is duplicated. The more you save the node, the more times the teaser gets put at the top of the body for the editing page.

How are you supposed to stop this from happening?

Edit:

A former coworker who has gone on to UNC fixed this for us awhile ago, I just remembered.

He upgraded Superteaser module from Drupal 4 to Drupal 6. It wasn't turned on for our site. I just turned it on, and the problem vanished.

Here is the code, feel free to use it or read it whatever:

<?php
/* $Id: superteaser.module,v 1.2 2006/06/10 15:14:42 alexreisner Exp $ */

/**
 * @file
 * This module increases control of the length and readibility of teasers.
 * Truncating algorithms are pitted against each other and the best result
 * for a particular text is chosen.
 */

/**
 * Get a teaser for a node.
 *
 * Use this function instead of node_teaser() for more specific control
 * over length and to generate teasers on load rather than save (allows
 * auto-teaser to change as body changes).
 *
 * @param object $node
 * @param int $length
 *   Target length, in characters.
 * @param string $fuzzy
 *   'shorter' : if not exact, text will be shorter than $length
 *   'longer'  : if not exact, text will be longer than $length
 *   else      : best of 'shorter' and 'longer' will be used
 * @return string
 */
function superteaser($node, $length = 100, $fuzzy = NULL) {

//  if ($node->teaser) { return $node->teaser; } // if manually specified

  $body = $node->body;
  $format = $node->format;

  // Filter text before deciding how long it is.

  // We check for the presence of the PHP evaluator filter in the current
  // format. If the body contains PHP code, we do not split it up to prevent
  // parse errors.
  if (isset($format)) {
    $filters = filter_list_format($format);
    if (isset($filters['filter/1']) && strpos($body, '<?') !== false) {
      return $body;
    }
  }

  // If the body is shorter than the desired length, use the whole thing.
  if ($length >= strlen($body)) {
    return $body;
  }

  // Get chopping algorithms and sort so more desirable ones take precedence.
  $algorithms = _superteaser_get_algorithms();
  asort($algorithms);

  // Adjust teaser length based on amount of text which is HTML tags.
  $html = _superteaser_percent_chars_tags($body, 0, $length);
  $length += intval($length * $html);

  // Run each algorithm and give the result a score.
  $best_score = 1000;
  foreach ($algorithms as $func => $value) {
    $text = call_user_func('_'.$func, $body, $length, $fuzzy);
    if (strlen($text) == 0) { continue; }
    $score = abs($length - strlen($text)) - $value;
    if ($score <= $best_score && strlen($text) > 0) { // lowest score is best
      $best_score = $score;
      $best_text = $text;
    }
  }
  $best_text = superteaser_remove_cut_off_tags($best_text);
  return superteaser_close_tags($best_text);
}

/**
 * Modify the database when upgrading from 4.6 to 4.7.
 */
function superteaser_update_1() {
  $algorithms = variable_get('superteaser_algorithms', array());
  foreach ($algorithms as $name => $value) {
    variable_set(substr($name, 1), $value);
  }
  variable_del('superteaser_algorithms');
}

/**
 * Get the list of chopping algorithms and their stored values.
 * An algorithm's value is the number of characters you're willing to miss
 * the targe length by for the type of break provided.
 *
 * @return array
 */
function _superteaser_get_algorithms() {
  $algorithms = array(
    'superteaser_truncate_at_break'     =>  180,
    'superteaser_truncate_at_paragraph' =>  100,
    'superteaser_truncate_at_sentence'  =>   80,
    'superteaser_truncate_at_space'     =>   10,
    'superteaser_truncate_exact'        =>  -50,
  );
  foreach ($algorithms as $name => $default) {
    $algorithms[$name] = variable_get($name, $default);
  }
  return $algorithms;
}

/**
 * Get the stored "fuzzy" setting (longer/shorter/best).
 *
 * @return string
 */
function _superteaser_get_fuzzy() {
  return variable_get('superteaser_fuzzy', 'best');
}

/**
 * Get the stored explicit text break string.
 *
 * @return string
 */
function _superteaser_get_break() {
  return variable_get('superteaser_break', '<!--break-->');
}

/**
 * Get the percentage of characters which make up HTML tags.
 *
 * @param string $text
 * @param int $offset
 *   Where to start looking at the string. Like 2nd param of substr().
 * @param int $length
 *   How many chars to look at from $offset. Like 3rd param of substr().
 * @return float
 */
function _superteaser_percent_chars_tags($text, $offset = 0, $length = null) {
  $text = $length === null ?
    substr($text, $offset) : substr($text, $offset, $length);
  if (!strlen($text)) { return 0; }
  $htmls = 0;
  $open = 0;
  while (($open = strpos($text, '<', $open)) !== false) {
    $close = strpos($text, '>', $open);

    // Handle unclosed tag at end of text.
    if ($close === false) {
      $close = strlen($text) - 1;
    }

    $htmls += $close - $open + 1;
    $open = $close + 1;
  }
  return $htmls / strlen($text);
}

/**
 * Truncate text at the nearest explicit breakpoint.
 *
 * @param string $text
 * @param int $length
 * @param string $fuzzy
 *   'shorter' : if not exact, text will be shorter than $length
 *   'longer'  : if not exact, text will be longer than $length
 *   else      : best of 'shorter' and 'longer' will be chosen
 * @return string
 */
function _superteaser_truncate_at_break($text, $length, $fuzzy = NULL) {
  $breakpoints = array(_superteaser_get_break() => 0);
  return _superteaser_truncate_at_points($text, $length, $breakpoints, $fuzzy);
}

/**
 * Truncate text at a space.
 *
 * @param string $text
 * @param int $length
 * @param string $fuzzy
 *   'shorter' : if not exact, text will be shorter than $length
 *   'longer'  : if not exact, text will be longer than $length
 *   else      : best of 'shorter' and 'longer' will be chosen
 * @return string
 */
function _superteaser_truncate_at_space($text, $length, $fuzzy = NULL) {
  $breakpoints = array(' ' => 0);
  return _superteaser_truncate_at_points($text, $length, $breakpoints, $fuzzy);
}

/**
 * Truncate text to an exact length.
 *
 * @param string $text
 * @param int $length
 * @param string $fuzzy
 *   Just a placeholder parameter.
 * @return string
 */
function _superteaser_truncate_exact($text, $length, $fuzzy = NULL) {
  return truncate_utf8($text, $length);
}

/**
 * Truncate text at the end of a paragraph (or list).
 *
 * @param string $text
 * @param int $length
 * @param string $fuzzy
 *   'shorter' : if not exact, text will be shorter than $length
 *   'longer'  : if not exact, text will be longer than $length
 *   else      : best of 'shorter' and 'longer' will be chosen
 * @return string
 */
function _superteaser_truncate_at_paragraph($text, $length, $fuzzy = NULL) {
  $breakpoints = array(
    "\n"     => 0,
    "</p>"   => 4,
    "<p>"    => 0,
    "<br>"   => 4,
    "<br/>"  => 5,
    "<br />" => 6,
    "</ul>"  => 5,
    "</ol>"  => 5,
  );
  return _superteaser_truncate_at_points($text, $length, $breakpoints, $fuzzy);
}

/**
 * Truncate text at the end of a sentence (or list item).
 *
 * @param string $text
 * @param int $length
 * @param string $fuzzy
 *   'shorter' : if not exact, text will be shorter than $length
 *   'longer'  : if not exact, text will be longer than $length
 *   else      : best of 'shorter' and 'longer' will be chosen
 * @return string
 */
function _superteaser_truncate_at_sentence($text, $length, $fuzzy = NULL) {
  $breakpoints = array(
    '. '  => 1,
    '! '  => 1,
    '? '  => 1,
    '.<'  => 1,
    '!<'  => 1,
    '?<'  => 1,
    '." ' => 2,
    '!" ' => 2,
    '?" ' => 2,
    '<li>' => 0,
    '</li>' => 5,
  );
  return _superteaser_truncate_at_points($text, $length, $breakpoints, $fuzzy);
}

/**
 * Truncate text at any of various specified points.
 *
 * @param string $text
 * @param int $length
 * @param array $breakpoints
 *   Arrary of possible breakpoints in format:
 *   array(
 *     ". "  => 1,
 *     ".\n" => 1,
 *     ".\"" => 2,
 *   )
 *   Keys are breakpoints, values are number of chars to preserve.
 * @param string $fuzzy
 *   'shorter' : If not exact, text will be shorter than $length.
 *   'longer'  : If not exact, text will be longer than $length.
 *   else      : Best of 'shorter' and 'longer' will be chosen.
 *   Note that 'shorter' or the default may result in an empty teaser.
 * @param bool $case
 *   Case insensitive breakpoint match? Default false.
 * @return string
 */
function _superteaser_truncate_at_points($text, $length, $breakpoints, $fuzzy = NULL, $case = false) {
  if (!$case) {
    $orig_text = $text;
    $text = strtolower($text);
    foreach ($breakpoints as $key => $b) {
      $breakpoints[$key] = strtolower($b);
    }
  }
  if ($length > strlen($text)) {
    $length = strlen($text);
  }

  // Longer.
  if ($fuzzy == 'longer') {
    $pos = strlen($text);
    foreach ($breakpoints as $b => $l) {
      $new = strpos($text, $b, $length);
      if ($new >= $length && $new < $pos) {
        $pos = $new;
        $len = $l; // preserve breakpoint's good chars
      }
    }
    $text = substr($text, 0, $pos + $len);
  }

  // Shorter.
  else if ($fuzzy == 'shorter') {
    $text = strrev(substr($text, 0, $length));
    $pos = $length;
    foreach ($breakpoints as $b => $l) {
      $new = strpos($text, strrev($b));
      if ($new && $new < $pos) {
        $pos = $new;
        $len = strlen($b) - $l; // cut off breakpoint's bad chars
      }
    }
    $text = strrev(substr($text, $pos + $len));
  }

  // Best of 'shorter' and 'longer' (closest to desired length).
  else {
    $s = _superteaser_truncate_at_points($text, $length, $breakpoints, 'shorter');
    $l = _superteaser_truncate_at_points($text, $length, $breakpoints, 'longer');
    $text = $length - strlen($s) < strlen($l) - $length ? $s : $l;
  }

  if (!$case) {
    $text = substr($orig_text, 0, strlen($text));
  }
  return $text;
}

/**
 * Remove an unfinished HTML tags from the end of a teaser.
 *
 * @param string $text
 * @return string
 */
function superteaser_remove_cut_off_tags($text) {
  return preg_replace('|<[^>]*$|', '', $text);
}

/**
 * Close all open HTML tags in a teaser.
 * Note this will not fix text with too many closing tags.
 *
 * @param string $text
 * @param array $ok_open
 *   Array of tags which may be left open (optional).
 * @return string
 */
function superteaser_close_tags($text, $ok_open = null) {

  // Default list tags which may be left open.
  if ($ok_open === null) {
    $ok_open = array(
      'img',
      'li',
      'br',
    );
  }

  preg_match_all('|<\s*(\w+)( [^>]*[^/])?>|', $text, $open);
  preg_match_all('|<\s*/(\w[^> ]*)>|', $text, $close);
  $open = $open[1]; $close = $close[1];
  if (count($open) != count($close)) {
    foreach ($open as $i => $o) {
      if (($j = array_search($o, $close)) !== FALSE) {
        unset($open[$i], $close[$j]);
      }
      else if (array_search($o, $ok_open) !== FALSE) {
        unset($open[$i]);
      }
    }
    if (count($open)) {
      while ($o = array_pop($open)) {
        $text .= "<$o>";
      }
    }
  }
  return $text;
}

/**
 * Implementation of hook_help.
 */
function superteaser_help($section) {
  switch ($section) {
    case 'admin/settings/superteaser':
      return t('<p>In order to make teasers readable, they may not be the exact length you specify. Instead, text is truncated at a point chosen for optimal readability. 
Set the number of characters by which you are willing to miss the target length for each kind of break point below.</p><p>The !link is !len characters.</p>',
        array(
          '!link' => l(t('current teaser length'), 'admin/content/node-settings'),
          '!len' => variable_get('teaser_length', 600)
        )
      );
  }
}


/**
 * Implementation of hook_menu.
 */
function superteaser_menu() {
  $items = array();
    $items['admin/settings/superteaser'] = array(
      'title' => t('Superteaser'),
	'description' => 'Configure superteaser settings',
      'page callback' => 'drupal_get_form',
      'page arguments' => array('superteaser_settings'),
      'access arguments' => array('administer site configuration'),
	'type' => MENU_NORMAL_ITEM,
    );
  return $items;
}

/**
 * Implementation of hook_settings.
 */
function superteaser_settings() {
  $labels = array(
    'superteaser_truncate_at_break'     =>  t('Explicit break, if found'),
    'superteaser_truncate_at_paragraph' =>  t('End of paragraph'),
    'superteaser_truncate_at_sentence'  =>  t('End of sentence'),
    'superteaser_truncate_at_space'     =>  t('Between words'),
    'superteaser_truncate_exact'        =>  t('Mid-word'),
  );
  $algorithms = _superteaser_get_algorithms();
  arsort($algorithms);

  // Algorithm value fields.
  $form['superteaser_algorithms'] = array(
    '#type' => 'fieldset',
    '#title' => 'Break Point Values',
    '#collapsible' => true,
    '#collapsed' => false,
  );
  foreach ($algorithms as $name => $value) {
    $form['superteaser_algorithms'][$name] = array(
      '#type' => 'textfield',
      '#title' => $labels[$name],
      '#default_value' => $value,
      '#size' => 6,
      '#maxlength' => 5,
    );
  }

  // Break tag field.
  $break = _superteaser_get_break();
  $form['superteaser_break'] = array(
    '#type' => 'textfield',
    '#title' => t('Explicit break string'),
    '#default_value' => $break,
    '#size' => 12,
    '#maxlength' => 50,
  );

  // Fuzzy field.
  $fuzzy = _superteaser_get_fuzzy();
  $options = array(
    'longer'  => t('Longer than target length'),
    'shorter' => t('Shorter than target length'),
    'best'    => t('Whichever is closest to target length'),
  );
  $form['superteaser_fuzzy'] = array(
    '#type' => 'radios',
    '#title' => t('Teaser length preference, if not exact'),
    '#options' => $options,
    '#default_value' => $fuzzy,
  );

  return system_settings_form($form);
}

/**
 * Implementation of hook_nodeapi.
 */
function superteaser_nodeapi(&$node, $op, $teaser = NULL, $page = NULL) { 
  $node->teaser_include = 0;
global $user;
  /* superteaser is not compatible with Drupal 6's teaser voodoo */
  switch ($op) {
    case 'presave': // presave seems to be the new "submit" op for Drupal 6
        $node->teaser = superteaser($node, variable_get('teaser_length', 600), _superteaser_get_fuzzy());
		$node->teaser = strip_tags($node->teaser, '<a><strong>');
		$node->teaser = '<p>'.$node->teaser.'...</p>';
		break;
    case 'prepare':
	$node->teaser = $node->body;
	break;
  }
}


/**
 * Implementation of hook_form_alter().
 * superteaser is not compatible with Drupal 6's teaser voodoo
 */
function superteaser_form_alter(&$form, $form_state, $form_id) {
	unset($form['#post']['teaser_js']);
}

I think the final two functions are the only important parts to addressing this problem, but I am not entirely sure at the moment. Hope this helps someone.

This appears to happen when the `node_revisions` table has data in the `teaser` field, but none in the `body` field.

The following SQL code seems to have fixed it for me (your mileage may vary)
UPDATE `node_revisions` SET `body` = `teaser` WHERE `body` = ''

This looks like a pretty serious bug for 6.14. I hope it gets fixed ASAP as it could cause havoc.

i noticed this bug in Drupal 6.1 sometime back as well so it's not unique to Drupal 6.14.

the problem is in the node.module/node.pages.inc code

you can mess around there to resolve this directly, but I have patched it many times over now, so can't recall what I did :-(

i'll feedback if I do

I just noticed that today! I just use FCKeditor by itself (not through WYSIWYG) so I guess that's not specific to you and thus would possibly be a problem in the core. Note that when I use the <!--break--> marker then everything is fine.

One idea would be that before the editor concatenates the teaser with the body, it should first check whether it is necessary to do so by looking for the <!--break--> marker in the body. If not present, ignore the teaser.

Now, the other possibility is that some other 3rd party module saves the $node->teaser when the Core does it right. If you do not have that 3rd party module, it may very well work right...

What do you think?

Thank you.
Alexis

P.S. This only happens when the teaser is marked as part of the body and there is no <!--break--> marker in the node content.

Got it... It's the Core that does it all wrong.

The following test does not take in account the fact that the teaser may be modified to include closing tags as required (i.e. if a paragraph is cut, then they may add '...' and close the necessary tags such as </p>).

  // Check if we need to restore the teaser at the beginning of the body.
  $include = !isset($node->teaser) || ($node->teaser == substr($node->body, 0, strlen($node->teaser)));

Hmmm... The Teaser as defined in Core is just all broken!

Alexis

Also, if you have a WYSIWYG or put a empty line at the start of content, the teaser does not get generated.

Yeah it looks like an empty teaser is not correctly managed. I think it's just not expected to be empty (how useful would that be?)

Anyway... I have a fix for the Core Drupal 6.19 which should work on older versions too. It completely replaces the completely broken node_teaser() function with something that never fails (he! he!)

Plus, to avoid the double teaser in the edit box you want to apply the fix to the node_body_field() too. (that's in the same patch.)

The problem is that the teaser will be transformed because it is incorrect HTML and thus it won't match the body anymore. With a correct teaser, you can test it properly against the body and distinguish it as expected.

#221257: text_summary() should output valid HTML and Unicode text

Thank you.
Alexis

That Drupal core patch is one possible cause for this issue. If you applied it, then your {node_revisions}.teaser values are bogus now.

I'm sorry, but neither TwoD nor me can reproduce this problem on a clean Drupal site. So there must be some other module or code that hi-jacks your data. In any case, the issue is not caused by Wysiwyg.

Sun,

The problem is when the very first paragraph is "so large" that it gets cut at the wrong place. Then when you click Preview teaser or Save it messes up everything.

This being said, you are right that is not your problem. It's the Core that doesn't know how to handle the teaser properly.

See reference in #12 above.

Thank you.
Alexis

I'm pretty sure this is caused by the revisions module or at least it coincides with my installation of that module. I misunderstood the post above. Do you have to hack the core to fix this? is there an update that deals with this issue?

As mentioned in #12, there is a patch available here: #221257: text_summary() should output valid HTML and Unicode text

It probably won't be easy to have it added to core, but when using a system like FCKeditor and with the Core bug in cutting the teaser without taking the HTML in account, it will do all sorts of funky things to your content.

I don't use the revisions module much at all so I'm sure that the problem described is very similar to what I mentioned. Either way, I think that you are not responsible. However, if you can help push the patch in Core, it would be great. 8-) More votes will eventually force them to accept the fact that the current version can generate a big bad loss of data and thus the correction is required.

Thank you.
Alexis

I may have found a way to reproduce this, but I don't have time to further debug or outline the details yet.
And I'm not sure this can be completely fixed within Wysiwyg either...
Marked duplicates of eachother and this issue: #945910: Wysiwyg always insert <!--break--> before text in textarea after submit edited node, #862822: Breaks tag doubles, doesn't work. The symptoms are a bit different, but I'm pretty sure they are caused by the same bug.

EDIT: Referenced the wrong issue above.

TwoD,

Would you have the time to test, see whether it fixes your problem to patch your system with #221257: text_summary() should output valid HTML and Unicode text and be so kind as to report the results here?

Thank you,
Alexis

P.S. Btw, the patch is in #49

Yes I'll test that, but first I need to be 100% sure what happens without it, will fire up the debugger as soon as I get some spare time.

I think this issue is most likely to happen if Wysiwyg was enabled when there are existing nodes with auto-generated teasers, or was it with those having separate teasers? (I forgot the details since I last dug into this.)

TwoD,

Yes. One very long paragraph at the top of your body ("long" means more than the max. number of characters allowed in your teaser,) and let the automatic teaser system do the work... It cuts the paragraph without closing the HTML tags. When you come back, the missing tags are added by the WYSIWYG editors and break the comparison and that means you end up with the teaser + body instead of just body in your editor.

Thank you.
Alexis

Hope you don't mind me tentatively reactivating this, but I've just fallen foul of this using the recommended version of WYSIWYG on D6.

In case it's useful, my teaser length is set to "unlimited".

(ick, you can ignore the attachment - it's no longer relevant)

I'm seeing this too, in some very specific circumstances - only when Upload Paths is enabled, and it changes a link, then the subsequent edit has a duplicated teaser. The teaser looks just fine in the node_revisions table, though, and it happens even when the Upload Paths rewrite isn't in the teaser itself.

Reinette,

I've been using this patch on my end:

http://drupal.org/node/221257#comment-3735536

It's a pretty heavy patch on core to fix the teaser problem for all WYSIWYG editors (I use CKEditor, mainly).

That thread is really long, but full of interesting stuff...

Thank you.
Alexis Wilke

For those who - like me - don't like the Teaser function at all and can't find a decent solution for the duplicated summary problem in D6, here is how I got a rid of teasers (from the user's perspective) all together. You have to put these two function into a custom module.

function YOUR_MODULE_NAME_nodeapi(&$node, $op, $a3 = NULL, $a4 = NULL) {
    if ($op == "prepare") {
		$node->teaser = '';
	}
}
   
function YOUR_MODULE_NAME_form_alter(&$form, $form_state, $form_id) {
    switch ($form_id) {    
      case $form['#node']->type .'_node_form': {
      // hide show summary in full view 
	      $form['body_field']['teaser_include'] = array(
             '#type' => 'value',
             '#value' => TRUE	,
          );
          break;
      }     
    }
}

I'm going to close this issue as a duplicate of #221257: text_summary() should output valid HTML and Unicode text because we can't fix it in Wysiwyg alone, and the motivation and time I have to spend on the 6.x branch is limited.

I may reopen it later, when I've gotten around to fixing other issues. Getting D6 summaries fixed in Core is pretty much a requirement for a useful patch here.
If the Core patch gets committed, this issue may even disappear entirely from Wysiwyg's perspective. (Note, that only applies to new content created/edited after the patch, any existing duplicated summaries are likely messed up beyond safe automated repair anyway.)

As it is now, you have to be very careful about where the teaser splitter gets inserted. If it gets inserted into a tag, Drupal will try to correct the faulty HTML resulting from that split in the field, but not in the summary. Drupal does not recognize that the summary is actually the first part of the field contents (because it has changed slightly) when the node is edited, so it prepends the summary to the field value and inserts the teaser break comment between them.