Allow admin to specify the default.robots.txt file

thanks Dave, works great. didn't check the upgrade, only fresh install.

robotstxt_get_content() is not an API function and is missing _ prefix. Aside I remember - I've said Dave some time ago - I do not like to read the disk for the robots.txt file after the installation has completed. I do not like to commit this as it overcomplicates this simple module without a benefit except ~100 new support requests.

I may not fully understand the need for this as every site could also have it's own custom sites/example.com/modules/robotstxt/robots.txt and otherwise there is a way to override the var via settings.php...

Need to find some time to look into...

No thanks.

I need this change too, it's a hell to have to install and enable and configure this module when you're running 20+ sites on a single drupal installation. Would be good to know if this can be implemented in the module, so I don't need to run a forked version

You are still able to have a default file without the patch

Perarnet have complained about something missing, but he have not understood how the module install works. He said he need to enable the module and configure on all sites by hand afterwards. This is wrong! He can add a pre-configuration in the module folder, enable the module and the config becomes his pre-configuration. His issue is thereby non-existent.

Dave - you are asking for a change than may go in if there are reviews, but there seems no interrest or we would have some reviews and comments here about test results.

Dave,

I see that you are much more persistent than I. I just dropped the ball on my end. Hass does not seem to understand maintenance of a multi-site install. And 20 websites isn't very many...

I noticed today that most of my users forbid access to user/<uid> but Google will hit those folders because it is not in the default robots.txt (reason is you may allow anonymous users to access those folders, obviously.) Note: Google knows because of Google Analytics which should probably be turned off on those folders. But it does not change the need for a correct robotstxt file.

That means many more entries in the db log that do not need to be there and false advertisement to search engines.

This being said, I reviewed your code changes. Doing so I noticed we could add a hook in our own modules. Cool!

Of course, I'd have one more request... a way to remove entries from the default robots.txt file... (he! he!) not too important though.

One thing though:

+
+  $file = variable_get('robotstxt_file', drupal_get_path('module', 'robotstxt') . '/robots.txt');
+  if ($file && variable_get('robotstxt_file_modified', 0) == filemtime($file)) {
+    $content[] = variable_get('robotstxt_file_contents', '');
+  }
+  else {
+    $content[] = $file_contents = file_get_contents($file);
+    variable_set('robotstxt_file_contents', $file_contents);
+    variable_set('robotstxt_file_modified', filemtime($file));
+  }
+

The $file parameter is used in the else part even when empty. Either change it to an elseif () or do this (better optimized):

+
+  $file = variable_get('robotstxt_file', drupal_get_path('module', 'robotstxt') . '/robots.txt');
+  if ($file) {
+    if (variable_get('robotstxt_file_modified', 0) == filemtime($file)) {
+      $content[] = variable_get('robotstxt_file_contents', '');
+    }
+    else {
+      $content[] = $file_contents = file_get_contents($file);
+      variable_set('robotstxt_file_contents', $file_contents);
+      variable_set('robotstxt_file_modified', filemtime($file));
+    }
+  }
+

The really cool thing with this is you can upload a robots.txt file to your files folder and point this module to that file. This means any user can do it without having to change anything in the modules folder (still need to chop the Core robots.txt file though.)

Thank you.
Alexis Wilke

P.S. One more thing, by the way, the new way prevents you from seeing the content of the txt file from the admin screen. It could be nice to add one more widget with the content the way it looks at this point. So on save you can check and see that it looks as you intended.