Problem/Motivation

robots.txt syntax checking link broken

After submitting site url at

# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html

The requested URL http://www.dcs.ed.ac.uk/cgi/sxw/parserobots.pl is not valid.

Proposed resolution

patch with working robots.txt syntax checking link submitted

I have tried this robots.txt syntax checker, it is a good one:

http://www.frobee.com/robots-txt-check

Remaining tasks

review and test patch

User interface changes

syntax checking link changed in robots.txt file

API changes

none

CommentFileSizeAuthor
#1 robots.txt_syntax_checking_link_broken-change_link_in_comment-2276469-1-8.patch323 bytesdecibel.places

Comments

decibel.places’s picture

decibel.places commented
StatusFileSize
new robots.txt_syntax_checking_link_broken-change_link_in_comment-2276469-1-8.patch323 bytes

here is a patch

decibel.places’s picture

decibel.places commented
Issue summary: View changes
Status: Active » Needs review

edited to use Issue Summary Template

decibel.places’s picture

decibel.places commented
Issue summary: View changes

clarify problem

jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented
Status: Needs review » Reviewed & tested by the community
Issue tags: +Needs backport to D7, +Needs backport to D6

I have confirmed the error reported here, and agree that the replacement seems to work. Seems like a good idea!

The issue was mistakenly reported in D6/7 with separate issues, so patches for 6/7 exist on:
#2276457: robots.txt syntax checking link broken
#2276447: robots.txt syntax checking link broken
although probably the 8.x patch will apply for 7 and 6 anyway.

Anyway, this patch seems RTBC to me; will leave here for a few days before committing in case someone disagrees.

decibel.places’s picture

decibel.places commented

my first core patch :D

thanks for clarifying the backport process

jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented

Great! Hope you'll come back and do more! :)

webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented
Version: 8.x-dev » 7.x-dev
Status: Reviewed & tested by the community » Patch (to be ported)

Great work, decibel places! :)

Committed and pushed to 8.x. Back to 7.x.

  • Commit 69e9596 on 8.x by webchick: 
    Issue #2276469 by decibel.places: Fixed robots.txt syntax checking link...

  • Commit e2b5ce9 on 7.x: 
    Issue #2276469 by decibel.places: Fixed robots.txt syntax checking link...
tstoeckler’s picture

tstoeckler
he/him
Primary language German
Location Essen, Germany
commented
Version: 7.x-dev » 6.x-dev
Issue tags: -Needs backport to D7

This actually got comitted to 7.x by "webchick" (check the commit link above... :-)), but could theoretically be backported to D6.

  • Commit 80cf244 on 6.x by webchick: 
    Issue #2276469 by decibel.places: Fixed robots.txt syntax checking link...
webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented
Version: 6.x-dev » 8.x-dev
Status: Patch (to be ported) » Fixed
Issue tags: -Needs backport to D6

Sorry, I actually meant to commit to all three branches AND follow up that I'd done that, except that the internet crapped out right in the middle. :P

Should be done now. :)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

decibel.places’s picture

decibel.places commented

oh my http://www.frobee.com/robots-txt-check is coming up "404 Not Found"

three options for a stable solution:

  1. provide a search link for "robots.txt syntax checker" https://www.google.com/search?q=robots.txt%20syntax%20checker
  2. build our own robots.txt syntax checker hosted by drupal.org
  3. don't provide any link for robots.txt syntax checker
decibel.places’s picture

decibel.places commented
Status: Closed (fixed) » Needs work
jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented
Status: Needs work » Closed (fixed)

I don't see this in the 8.x or 7.x robots.txt file.