Content is more and more stolen via automated crawler and the used for cloaking.

On the SEO side this is very bad, because it produces duplicated content and often the orginal site gets a penalty by the search engine.

If you would implement it, that only the listed crawler are allowed, it would be a great wall for content stealers.
More control to the content produces and provider.

Comments

enikola’s picture

enikola commented
Assigned: Unassigned » enikola

Very good point. I'll look into it, either in the first stable version or in the next one.

bennos’s picture

bennos commented

More Security could be realized. Bind crawler on the IP or the IP Range where they come from.
A simple Cross Check and self written crawler have no chance.