Sometimes when I'm importing a bunch of feedapi items, which are being processed automatically by opencalais, I get an error like this:

Calais processing error:
403 Developer Over Qps

1. What does it mean?

2. How do I stop it?

3. How bad is it to be getting this error? Is this dangerous?

Thanks.

Comments

febbraro’s picture

febbraro commented

Basically this means that you have exceeded your quota of transactions per second enforced on the web service. If you have some way of throttling you can avoid the message/failure.

The Calais Bulk Processing implements a basic approach to throttling.

febbraro’s picture

febbraro commented

Missed one. To answer your 3rd question of is it bad... Well, there are no puppies killed when you get the error (thankfully), however it fails to apply Calais metadata to your node, so you miss out on the autotagging, etc. So nothing is hurt, but you miss out on some data.

bflora’s picture

bflora commented

Thanks. That's more or less what I assumed was happening. Sweet.

Also, dude, bulk processing.......it actually works! Congrats. After 6 months of working with drupal I've learned that modules work off-the-shelf about 40% of the time. When I read about the bulk processiong, I assumed it'd be the feature that just didn't work at all, but it did. And it even looks slick. Kudos.

febbraro’s picture

febbraro commented
Title: Calais processing error: 403 Developer Over Qps » Investigate throttling Calais calls for nodes being created via FeedAPI
Assigned: Unassigned » febbraro

Thanks bflora.

I can see Feed API being a very popular use of Calais integration, we should try to devise an approach for throttling feedapi based requests.

bflora’s picture

bflora commented

I would argue that it's the only really smart use of Calais. If you're going to take the trouble to type in a node, then you'll probably add your own meta data.

If you've got something that can generate metadata on the fly however, then you need to take advantage of that.

febbraro’s picture

febbraro commented
Category: support » feature
febbraro’s picture

febbraro commented
Status: Active » Postponed

I have added SemanticProxy integration with Calais specifically to work with FeedAPI. Basically it allows you to submit the Feed Items URL to Calais, SemanticProxy gets thd ontents of that URL and tries to intelligently extract the original content and submits that to Calais (instead of submitting the miniscule 1 sentence blurb). You also have the option of saving that resultant document text to your node too.

While this does not fix the quota issues, there is a bit more delay in using SemanticProxy and I have not seen these errors lately.

febbraro’s picture

febbraro commented
Status: Postponed » Closed (fixed)