I'm using custom robots.txt files for every single domein by using the Domain Robots txt module.
The xml sitemap module automaticly adds a sitemap line to the robots.txt file. This is good for a single domain installation.

For a multi domain installation I have individual robot.txt files for each domain. I don't want this extra automatic sitmap lin in the custom robots.txt files. Wich looks like this:

How can I exclude this line created by the xml sitmap module?

CommentFileSizeAuthor
#8 xmlsitemap-configurable-robotstxt-integration-1927654-8.patch3.13 KBfeng-shui
#5 xmlsitemap-configurable-robotstxt-integration-1927654-5.patch2.42 KBfeng-shui
#2 xmlsitemap-sitemap_line_configurable_robotstxt-1927654-2-D7.patch1.71 KBattekilpela

Comments

jawi’s picture

jawi commented
Issue summary: View changes

spellling adjustment

mgifford’s picture

mgifford
he/him
Primary language English
commented
Issue summary: View changes

Is "custom robots.txt" a contributed module or your own code?

attekilpela’s picture

attekilpela commented
StatusFileSize
new xmlsitemap-sitemap_line_configurable_robotstxt-1927654-2-D7.patch1.71 KB

I created a patch, which adds the functionality to choose whether the sitemap is added automatically to robots.txt.

attekilpela’s picture

attekilpela commented
Status: Active » Needs review
feng-shui’s picture

feng-shui commented
Title: Remove Sitemap line from robots.txt » Make robots.txt module integration configuratble
Version: 7.x-2.x-dev » 8.x-1.x-dev
Category: Support request » Feature request

We've had a patch in D8 for some time to do the same thing, bumping this to 8.x and adding an 8.x patch.

feng-shui’s picture

feng-shui commented
StatusFileSize
new xmlsitemap-configurable-robotstxt-integration-1927654-5.patch2.42 KB

And the patch...

feng-shui’s picture

feng-shui commented
Title: Make robots.txt module integration configuratble » Make robots.txt module integration configurable

Status: Needs review » Needs work

The last submitted patch, 5: xmlsitemap-configurable-robotstxt-integration-1927654-5.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

feng-shui’s picture

feng-shui commented
StatusFileSize
new xmlsitemap-configurable-robotstxt-integration-1927654-8.patch3.13 KB

Fix some test failures.

feng-shui’s picture

feng-shui commented
Status: Needs work » Needs review

I'm not sure if the patch will need to include an update hook to get cache clear to run with the addition to the schema file? Happy to take some direction on that, and if anything else is needed to set the default value of the setting to true.