Some nodes on our site represent information from other external pages. The content updating is done manually and tracking changes is a nightmare. For example when a bank changes its credit programm description on their site we must update a node tied to that bank on our site. We need to keep up to date about 300 nodes of 3 types, each node tied to 1-3 external pages. The idea is to check those pages automatically for changes and mark a node "need updating" if something changed on those pages.

The task is very similar to Link Checker module except for:
- the links to external pages need not to be extracted -- we may provide exact links for every node by hand;
- we need to track linked content updates, not only response code.

So I have 2 questions:
1. I haven't found any module available already addressing the exact task. Is there any?
2. If I'll implement it would it be better to add a feature to Link Checker, start a separate project using some of Link Checker code or write from scratch?

Comments

hass’s picture

hass commented
Status: Active » Closed (works as designed)

This is an extremely specific request and I think it's out of scope for linkchecker. Linkchecker does not download the content in normal 99.9% of all cases. We only use HEAD and we never see content nor do we save content today. This will change if we also check anchors, but not for now. GET is only used in edge cases (e.g broken servers) and can result in serious issues if you use it to check a "download" link. I'm also planing to implement "range" support to make GET more reliable, but it will not really solve your specific needs.

To implement a feature like you need it - in a reliable way you need HTMLDom and you need to parse HTML containers, build checksums and so on and so on. If you don't do so, you will get tons of differences if a side bar changes or ads or tracking codes or anything else. This is completely out of scope for this module.

sin’s picture

sin commented

Thank you for the feedback!

Autodetecting where content starts and ends is a complicated task so it will be done by hand once per link at the initial stages.

I plan to avoid parsing storing two pieces of the content per link (node) -- a starting and ending words of a page content we need to track, cutting the rest will drop sidebars etc, after that dropping script tags, using strip_tags() on the text between two content pieces and doing a hash should do the work.

The 'download' link is a special case indeed. For the Content-Type other than text/html a HEAD request should be sufficient. I need to store and track Etag, Last-Modified, Content-Length changes.

Implementing If-Modified-Since may worth it too.

Thanks again for your time, I'll start a separate project soon.