If the user follows the instructions given in the project page, he will delete, or rename the robots.txt file that is present in the root directory of his web site; that is required for the module to work, otherwise the callback it defines will never be called from Drupal (an existing file, or directory takes the precedence over a callback defined in the Drupal menu system).
Being this the case, it's useless for _robotstxt_get_file_contents() to look for a file named robots.txt in the site root directory because it will not be found.

Comments

AlexisWilke’s picture

AlexisWilke commented

This is definitively a problem.

I see a couple of issues here:

1. When I upgrade from Drupal 6.x to 6.(x + 1), I will put the new robots.txt file in there. That's annoying... The module should generate a warning/error report if the file exists.

2. The module should offer 2 setups: (1) a way to append to the existing default robots.txt file; (2) a way to overwrite the entire file.

Case 2.(1) would help greatly whenever there is an update of the Core.

hass’s picture

hass commented
Status: Active » Closed (works as designed)

Guy's - we have a hook_requirement() for beginners! We try to read the most recent robotx.txt - that's all. At very first - in the core root and than if not found in the modules folder. By this logic we make sure to import the changed D 6.8 file first and not the since a few weeks less up-to-date robots.txt from the module. That's all the logic behind.

But nevertheless - I know very well what I've changed in the core robots.txt and therefore I guaranty you there are no changes that needs a new robotxt.txt in the modules directory. It was only a cosmetic cleanup task that I've made on the core robots.txt.

The requirement hook hardly complains with REQUIREMENT_WARNING in admin section if there is a file in Drupal's root. Additionally for people without clean URL we show REQUIREMENT_ERROR that the module does not work at all if clean URL are disabled. What should we do more???

If this errors and warnings are not annoying enough give me a hint... but I decided to use REQUIREMENT_WARNING for the existing file as the site delivers the default robots.txt and therefore nothing is really broken from site view... only the modules does not show the configured robots.txt - easy to fix - delete the file and the warning goes away. This is the way how many Drupal modules are working...

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented

Still the function is looking for a file it cannot find. The code that tries to open robots.txt cannot be removed, as that file will never found.

I am not saying that the warning about the file being present is wrong, but that is useless to try to open that file, when it has been renamed, or moved, or deleted.

Just to make a summary: the module warns if it finds the robots.txt file in the Drupal root directory (that is correct, as with that file present the module menu callback will never be called), but then a module function looks for that file which will never been found, if the administrator of the web site using RobotsTxt follows the instructions given him (from the module documentation, or by the warning the module shows).

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented

To make clear what I am talking of, the following code is present in _robotstxt_get_file_contents():

  if (file_exists('./robots.txt')) {
    $file_contents = check_plain(file_get_contents('./robots.txt'));
  }
hass’s picture

hass commented
Status: Closed (works as designed) » Active

This is wrong - the file MAY exists if a new user is NOT aware about the way how robotstxt is working. So the check is ok. If there IS a file we import it - otherwise we do nothing.

hass’s picture

hass commented
Status: Active » Closed (works as designed)