I just installed Boost and then went to the status report where I was greeted with the following warning message:

"Disallow: /boost_stats.php" entry missing in robots.txt
Your robots.txt file does not contain "Disallow: /boost_stats.php". This is needed in order to prevent the output of this from accidentally getting indexed by search engines.

Curious, I checked my robots.txt (which is actually dev_robots.txt using Rewrite for a multi-site installation) and it contains the following entry:

# This example indicates that no robots should visit this site further:
# go away
User-agent: *
Disallow: /

In my understanding, then no pages on the site should be indexed so I shouldn't need a specific entry for the boost_stats.php. My suggestion is to not display the warning if we have the specific recommend entry or the Disallow: / entry.

Comments

Georgii’s picture

Georgii commented

From the search engines perspective you are right, "Disallow: /" is enough to disallow indexing of the entire web-site.

However,if you look into the boost code

if ($robots && !stristr($robots, 'Disallow: ' . $base_path . 'boost_stats.php')) {
        $requirements['boost_robots'] = array(
          'title'       => $t('Boost'),
          'description' => $t('Your root robots.txt file does not contain "Disallow: !stats". This is needed in order to prevent the output of this from accidentally getting indexed by search engines.', array('!stats' => $base_path . 'boost_stats.php')),
          'severity'    => REQUIREMENT_WARNING,
          'value'       => $t('"Disallow: !stats" entry missing in robots.txt', array('!stats' => $base_path . 'boost_stats.php')),
        );
      }

you will see that it searches for the exact 'Disallow: ' . $base_path . 'boost_stats.php' line inside.
Have you tried adding this line to your robots.txt?