Is is possible to provide some quick configuration settings?
Instead of only showing a textfield in which the administrator needs to enter the robots.txt contents, you could show a series of quick settings like

  • Use Drupal default
  • Deny access to all pages
  • ...
  • Custom

As the last option, you could add 'Custom' after which the administrator needs to enter contents in the textfield.

This should allow an administrator who wants to deny access to all pages to just select the option, instead of having to lookup the exact robots.txt syntax.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

hass’s picture

Status: Active » Closed (works as designed)

Seems to be overkill for this module, logic and usability wise.

xatoo’s picture

Version: 6.x-1.2 » 7.x-1.x-dev
Status: Closed (works as designed) » Active

Actually I agree with Iva that such a setting woud be useful.
For example: Developers might want to prevent access to all pages for their test/acceptance versions but restore the robots.txt content as soon as the site goes live. Currently there is no easy way to do this. (Other than to implement a custom module that calls "variable_set('robotstxt', file_get_contents());")

Would you reconsider this issue if I provide a patch that allows for easy restoration via the UI and Drush?
I think I can provide a way to make it very clear usability wise how this would work.

hass’s picture

Version: 7.x-1.x-dev » 6.x-1.2
Status: Active » Closed (works as designed)

There is a hook that we are using with xmlsitemap, too. Check it out, please.

xatoo’s picture

I know about that. However, that does not provide a quick way to set or restore a complete robots.txt file. I will still provide a patch and hope that you will reconsider.

xatoo’s picture

Version: 6.x-1.2 » 7.x-1.x-dev
Status: Closed (works as designed) » Needs review
FileSize
8.81 KB

- Adds a hook that modules can implement pointing to their own directory with robots.txt files
- Adds a way to quickly restore content from such a preset file
- Adds a way to get and set the robotstxt content via drush

hass’s picture

Status: Needs review » Needs work
+++ b/robotstxt.admin.incundefined
@@ -11,6 +11,29 @@
+  $options = array('' => t('-- Choose --'));

Please review core. We normally use a wording like "- Please select -" or something like this.

+++ b/robotstxt.admin.incundefined
@@ -18,7 +41,30 @@ function robotstxt_admin_settings() {
+    '#prefix' => '<div id="robotstxt_content">',
+    '#suffix' => '</div>',

What is this for?

+++ b/robotstxt.admin.incundefined
@@ -18,7 +41,30 @@ function robotstxt_admin_settings() {
+  $form['#submit'][] = 'robotstxt_admin_settings_form_submit';
+

As I know, this is not required. hook_form_submit is always executed.

+++ b/robotstxt.admin.incundefined
@@ -18,7 +41,30 @@ function robotstxt_admin_settings() {
+  if (!empty($form_state['values']['file']) &&
+    ($file_content = robotstxt_get_from_file($form_state['values']['file']))) {

Single line without linebreaks, please.

+++ b/robotstxt.admin.incundefined
@@ -18,7 +41,30 @@ function robotstxt_admin_settings() {
+    $form['robotstxt']['#description'] = t('Note: The form still has to be saved in order to store the current content.');

Don't we typically use a drupal_set_message for such notes?

+++ b/robotstxt.api.phpundefined
@@ -17,3 +17,15 @@ function hook_robotstxt() {
diff --git a/robotstxt.drush.inc b/robotstxt.drush.inc

diff --git a/robotstxt.drush.inc b/robotstxt.drush.inc
index 284d962..7c40aee 100644

index 284d962..7c40aee 100644
--- a/robotstxt.drush.inc

--- a/robotstxt.drush.inc
+++ b/robotstxt.drush.incundefined

+++ b/robotstxt.drush.incundefined
+++ b/robotstxt.drush.incundefined
@@ -10,9 +10,9 @@

@@ -10,9 +10,9 @@
  */

A diff of a files that do not exists in the project? How is this possible?

+++ b/robotstxt.moduleundefined
@@ -82,7 +91,7 @@ function _robotstxt_get_content() {
-      drupal_get_path('module', 'robotstxt') . '/robots.txt',
+      //drupal_get_path('module', 'robotstxt') . '/robots.txt', // TODO

?

hass’s picture

Issue summary: View changes
Status: Needs work » Closed (won't fix)

Closing for lack of progress.