We need an api function to be able to extend the robots.txt dynamically. For example XML Sitemaps (formerly known as Google Sitemaps) needs to extend the file. With a line

Sitemap: http://www.example.com/sitemap.xml

or

Sitemap: http://www.example.com/gsitemap
CommentFileSizeAuthor
#7 hook_robotstxt.patch822 byteshass
#3 robotstxt_hook.patch798 byteshass

Comments

david strauss’s picture

That's a good idea. How do you suggest the API should work? This module should probably move toward a rule-based system instead of just a bunch of text.

hass’s picture

i only thought about a hook yet, but if you have a better idea... why not. hooks maybe to slow!? i don't know.

what is a rule based system you are talking about?

hass’s picture

Status: Active » Needs review
StatusFileSize
new798 bytes

Here is a patch for this feature. Let me know what you think about and if you'd like to commit it, please.

A module like gsitemap is then able to implement this in the following way (as an example):

/**
 * Extend the robots.txt file created by robotstxt module
 */
function gsitemap_robotstxt() {
  return = array('Sitemap' => url('gsitemap', NULL, NULL, TRUE));
}
hass’s picture

small mistake. this is the correct code example:

/**
* Extend the robots.txt file created by robotstxt module
*/
function gsitemap_robotstxt() {
  return array('Sitemap' => url('gsitemap', NULL, NULL, TRUE));
}
david strauss’s picture

Status: Needs review » Reviewed & tested by the community

I'm going to make a few tweaks to the patch for performance, but I'll just mark it as "ready to commit."

hass’s picture

i found one problem, but i'm not sure if this is why i'm not so familiar with some array functions... it looks like that it is not possibly to add arrays with the same Keys together... if the Key is 'Disallow' all array members with Disallow are merged into the last inserted array member... isn't it possible to have more then one key with the same name added to an array?

then there should be something to be changed...

hass’s picture

StatusFileSize
new822 bytes

Reworked the patch with some changes.

Are you able to *official* release the module - for "update_status" module, please? :-)

hass’s picture

Code example for other modules like XML Sitemap:

/**
* Extend the robots.txt file created by robotstxt module
*/
function gsitemap_robotstxt() {
  return array("Sitemap: ". url('sitemap.xml', NULL, NULL, TRUE));
}
hass’s picture

Version: 5.x-1.x-dev » 5.x-1.0
Assigned: Unassigned »
Status: Reviewed & tested by the community » Fixed
Anonymous’s picture

Status: Fixed » Closed (fixed)