How to stop boost crawler to hit external links?

Thanks in advance.

Comments

drib’s picture

drib commented
Title: CCK redirection field: do not cache node if it's using one. » Stop boost crawler to hit external links

_____________________
Descoperiti carti bune!

TimelessDomain’s picture

TimelessDomain commented
Component: Caching logic » Miscellaneous

boost does not cache external links. are you worried about boost crawler crawling outside your website & onto external websites? i don't think that it has this functionality, so no problem here. someone else please confirm this before closing issue

mikeytown2’s picture

mikeytown2 commented

Boost doesn't read the html's content looking for links. Uses the url_alias table.

drib’s picture

drib commented

Thanks for your answers. I know that boost uses url alias table, but I'm using a CCK redirection field that points to an external link (I noticed later that this was the reason for hitting that external links).

Even when I'm using this code (found it on http://drupal.org/node/805246):

(empty($node->content['field_redirection'])) ? (return TRUE) : (return FALSE);

on "PHP code returns TRUE", with "Crawl All URL's in the url_alias table" checked, seems the Boost crawler hits the external link through that CCK redirection field.

To rephrase, I want Boost not crawl the nodes with a CCK redirection field when "Crawl All URL's in the url_alias table" is checked. Is this possible?

Thank you again for your answers.

mikeytown2’s picture

mikeytown2 commented
Title: Stop boost crawler to hit external links » CCK redirection field: do not cache node if it's using one.

Try this code

$arg = $arg();
if (!empty($arg[0]) && $arg[0] == 'node' && !empty($arg[1]) && is_numeric($arg[1]) && empty($arg[2])) {
  $node = node_load($arg[1]) {
  if (!empty($node->content['field_redirection'])) {
    return FALSE;
  }
}
return TRUE;
mikeytown2’s picture

mikeytown2 commented

_cck_redirection_get_redirect_fields looks interesting. I would need to join the node type to the url alias table at a reasonable speed in order to do this.

drib’s picture

drib commented
Title: Stop boost crawler to hit external links » CCK redirection field: do not cache node if it's using one.

I tested the code and so far it looks fine. I'll test it more and I'll come back with results.

Thank you very much for your work and for answers.

_____________________
Descoperiti carti bune!

drib’s picture

drib commented

It works fine until I used again "Crawl All URL's in the url_alias table".

As I noticed, the code stops "Boost" caching the pages that contain CCK redirection field (I didn't find any cache files for the nodes that contain CCK redirection field), but I think that the Boost crawler checks that nodes and, of course, through CCK redirection field hits the external links. I think the cause is using "Crawl All URL's in the url_alias table" and the solution is to prevent for crawling that specific nodes.

Is there any solution to use for crawling other list of nodes than url_alias table ?

Thanks again for all your answers.

mikeytown2’s picture

mikeytown2 commented

If the node has been cached before, it will be crawled again. You can also enable a setting that puts the node into the boost_cache table on node creation