Providing a hook for checking some links against custom conditions

Could you provide example URL's, please? Expired links showing a message should send an 404 status code and not 200. If they do - the site admins need to be contacted to fix their broken servers...

Otherwise you can add this problematic domains to the exclusion list. Than the links are not verified.

Thank you for the example! Wget shows this is a broken server or webapp for the reason that the 200 status code need to be a 404 :-(

--xx:xx:xx--  http://rapidshare.com/files/85079037/koc.part6.rar
           => `koc.part6.rar'
Resolving rapidshare.com... 195.122.131.8, 195.122.131.9, 195.122.131.10, ...
Connecting to rapidshare.com|195.122.131.8|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6,945 (6.8K) [text/html]
200 OK

This feature becomes possible if #503040: Implement a trigger mechanism to call actions based on link checker test results has been implemented and you write a custom action for this specific functionality. If you can code, feel free to jump in.

It sounds like a good addition to me, too as many sites like rapidshare seems to have some incompetent web developers or system administrators who cannot configure their server in a correct way or cannot program. It's not the fault of this module that this sites are buggy and the module cannot fix buggy server configurations... this are the bad news. I'm sorry for this, but this is how it is. I have also send all sites behaving wrong - emails that they should fix their servers. Some fixed their servers others have had incompetent people and they haven't fixed it - but over all 7/10 fixed their servers. More often I've seen yet servers sending 500 to HEAD requests... this is also very bad.

Please send the support of rapidshare an email and inform them about their buggy server/webapp and they should fix their servers to be RFC compliant. Maybe point here.

Marking as a duplicate of #503040: Implement a trigger mechanism to call actions based on link checker test results.

Only keep in mind that we are not able to download the content of such a link as rapid share links are mostly binary and mega/gigabytes in size while running cron. If we are not able to verify the links with HEAD we are out of luck. As very last action - put them on the ignore list and never check binary file links like rapidshare with GET method or your Drupal may become unresponsive for long time!

Only a note about Logged-in/logged-out. In such a case they need to send a 403 (access denied) if they require a login to access the binaries... So again - another bug on their sites... :-).

They may need some consulting about http status codes and why there are status codes...

This is no incompetence of the web developers on these sites.

These no-file-found pages are ALL valid, and its a way for them to make money, so they cant set them 403 or 200 or 404.

For example, access http://rapidshare.com/RAMDOMPAGEHERE and you WILL get your 404 http status message. But the pages where you have rapidshare.com/files/xxxxx should not be 404 nor 403, even if the page takes you nowhere.

What has to be done is a way to download the page and preg_match its contents. It will require custom code for Rapidshare, Megaupload, and any other file sharing servers, so a plugin approach may be the best.

I see no relation in #503040: Implement a trigger mechanism to call actions based on link checker test results with this issue.

Let me know if i can help with anything.

Are you one of the developers at one of the sites? Great to have you here than... write me via contact tab.

We can schedule some consulting hours and I will explain You what status codes are available and usable in the special cases, but on the end of the day - what they are doing is wrong and there are other status codes for what the are doing.

Make it short - if a page is not found or an url is invalid (download has been removed from rapidshare) they are free to show a custom file not found page, but they need to send a 404 status code (this is technical only) what doesn't mean the page shows other content than today. If they require payment there is a status code 402. But in the above reported issues we have ONLY seen dead links where the files have been removed from rapidshare and the page got us an 200 OK for download not found. This is clearly WRONG and have nothing to do with their business model.

Here is the self explaining list:

    100 => 'Continue',
    101 => 'Switching Protocols',
    200 => 'OK',
    201 => 'Created',
    202 => 'Accepted',
    203 => 'Non-Authoritative Information',
    204 => 'No Content',
    205 => 'Reset Content',
    206 => 'Partial Content',
    300 => 'Multiple Choices',
    301 => 'Moved Permanently',
    302 => 'Found',
    303 => 'See Other',
    304 => 'Not Modified',
    305 => 'Use Proxy',
    307 => 'Temporary Redirect',
    400 => 'Bad Request',
    401 => 'Unauthorized',
    402 => 'Payment Required',
    403 => 'Forbidden',
    404 => 'Not Found',
    405 => 'Method Not Allowed',
    406 => 'Not Acceptable',
    407 => 'Proxy Authentication Required',
    408 => 'Request Time-out',
    409 => 'Conflict',
    410 => 'Gone',
    411 => 'Length Required',
    412 => 'Precondition Failed',
    413 => 'Request Entity Too Large',
    414 => 'Request-URI Too Large',
    415 => 'Unsupported Media Type',
    416 => 'Requested range not satisfiable',
    417 => 'Expectation Failed',
    500 => 'Internal Server Error',
    501 => 'Not Implemented',
    502 => 'Bad Gateway',
    503 => 'Service Unavailable',
    504 => 'Gateway Time-out',
    505 => 'HTTP Version not supported',

On the other side the referenced case is the solution as everyone will be able to create custom action handlers for buggy sites. It's clearly the best way to solve such issues in a general manner for all times and all the buggy sites around having incompetent admins and developers who have no understanding about the insights of the web/webservers and how it works.

I also know about a few people blocking HEAD requests... dumb psychos... they could save traffic and save server load... shit happens for them - we use GET in this cases and if we'd like we come as Internet Explorer - no problem at all we fake this, too. Amazon does not allow HEAD as one example, but they have good admins providing a status 405 that tells us - this method is not allowed and the module automatically switch over to GET and give their servers more load... but who cares if they like to learn it in a wasteful way!?

Understood.

;)

Sites such as rapidshare.com/filesonic.com/hotfile.com/megaupload.com/etc do not provide standard HTTP codes to hinder automation. They don't want automated systems checking their contents availability for whatever reason. This is clearly intentional as their content may be questionable, depends entirely on the uploader (ie registered user). They do this to prevent systems, such as Link Checker, from finding the current status of the URL in question.

I understand this is not common use case. Some custom parsing is required, beyond these mentioned hosting sites, for other potential developers needs. What is the best way to apply custom logic to handle these kinds of URLs, besides forking Link Checker altogether?

Thanks in advance for any advice on the matter,
.