When using drush solr-index to re-index a site, if the process takes longer than 15 minutes, the service will begin throwing 403s for every iteration of the indexing process.
Run drush solr-index on a site with a large-enough number of nodes that indexing it would take longer than 15 minutes (e.g. 10,000 nodes).
A quick skim of the code indicates that acquia_search_authenticator() uses REQUEST_TIME instead of time(). As a result, after the drush command has been running for over 15 minutes, the HMAC authenticator is no longer valid. This would be a likely place to start.
| Comment | File | Size | Author |
|---|---|---|---|
| #14 | 2077981-14.patch | 2.83 KB | nick_vh |
| #11 | acquia_search-solr-index-hmac-timeout-2077981-11.patch | 843 bytes | janusman |
| #4 | acquia_search-solr-index-hmac-timeout-2077981-4.patch | 843 bytes | jrbeeman |
| #2 | acquia_search-solr-index-hmac-timeout-2077981-1.patch | 544 bytes | jrbeeman |
Comments
Comment #1
nick_vhConfirming that this is an issue.
Comment #2
jrbeemanThe patch attached appears to have resolved the issue for me. Instead of using REQUEST_TIME, use time() when generating the authenticator string.
Comment #3
coltraneI recommend an explicit comment for why time() is being used instead of REQUEST_TIME so it doesn't get changed back by future developer.
Comment #4
jrbeemanAdded a comment to explain why time() is being used.
Comment #5
nick_vh*Agrees with coltrane*
Comment should be something like this :
"Due to the nature of drush and long running scripts it is required to refresh the timestamp value that is being sent. Acquia Search servers check for the timestamp and will discard requests if the timestamp that was sent is older or newer than 15 minutes."
We could get more complex also and refresh the value after 10 minutes but I suppose it is easier to just take the time() value.
Comment #6
nick_vhLet's test this, so we can get a green board and then get approval from coltrane to commit
Comment #7
nick_vhComment #8
cpliakas commentedCross-posting to https://drupal.org/node/1678152. Although it an issue for a different module, the bug is the same and thought it would be appropriate to link them.
Comment #9
coltraneCommitted http://drupalcode.org/project/acquia_connector.git/commit/9ff1452 Thanks!
Comment #11
janusman commentedYikes! The improvement was clobbered during a non-related commit :( We need to get it back in ASAP.
See: http://drupalcode.org/project/acquia_connector.git/blobdiff/9ff1452da650... for commit that undid the fix.
New patch from clean 7.x-2.x branch.
Comment #13
nick_vhYikes indeed! That's not good
Comment #14
nick_vhComment #15
nick_vhComment #17
coltraneI've pushed this http://drupalcode.org/project/acquia_connector.git/commit/8b9d40f. Thanks very much for the speedy issue and patch and I'm sorry this happened and went so unnoticed!