POST queries result in a 403 access denied error [#1678152]

There are reports that POST queries are returning 403 errors for operations such as index clears and reindexing. The specific error is listed below:

Apache_Solr_HttpTransportException: '403' Status: Forbidden in Apache_Solr_Service->_sendRawPost() (line 364 of sites/all/libraries/SolrPhpClient/Apache/Solr/Service.php).

This indicates that the POST queries aren't routed through SearchApiAcquiaSearchHttpTransport::performHttpRequest() causing 403 errors. (Thanks Nick_vh!) This method adds the access parameters to the query so that it authenticates using Acquia Search's HMAC authentication mechanism.

Comment	File	Size	Author
#6	indexing-403-1678152-6.patch	630 bytes	cpliakas
#6
#4	debug-1678152-4.patch	737 bytes	cpliakas
#4

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

cpliakas CreditAttribution: cpliakas commented 12 July 2012 at 14:59

Status:

Active

» Closed (cannot reproduce)

Closing as cannot reproduce. The reporter apparently was using something other than the code in this module.

Comment #2

kenorb CreditAttribution: kenorb commented 15 May 2013 at 08:51

Status:

Needs review

» Active

The same problems.

Solr retrurns a 403 at times. No pattern detected yet. Happened a couple of times during the 3 hours when testing the search. No configuration was changed during the period. No code changes either.

The error reads "SearchApiException: An error occurred while trying to search with Solr: '403' Status: Forbidden. in SearchApiSolrService->search() (line 607 of modules/search_api_solr/service.inc)." - on the search pages

At the same time when you try to index the content, it throws an error:

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=150&op=do StatusText: Service unavailable (with message) ResponseText: Apache_Solr_HttpTransportException: '403' Status: Forbidden in Apache_Solr_Service->_sendRawPost() (line 364 of libraries/SolrPhpClient/Apache/Solr/Service.php).

The Search API configuration page reports that the solr server is available and in good health.

Search API Modules:

$ drush pm-list | grep -e acquia -e search_api
Acquia agent (acquia_agent)                                           Module  Enabled        7.x-2.8      
Acquia search (acquia_search)                                         Module  Enabled  7.x-2.8      
Acquia Search for Search API (search_api_acquia)                      Module  Enabled        7.x-1.0-beta1
Database search (search_api_db)                                       Module  Not installed               
Search API (search_api)                                               Module  Enabled        7.x-1.4      
Search API autocomplete (search_api_autocomplete)                     Module  Not installed  7.x-1.0-beta2
Search API saved searches (search_api_saved_searches)                 Module  Enabled        7.x-1.0      
Search facets (search_api_facetapi)                                   Module  Enabled        7.x-1.4      
Search ranges (search_api_ranges)                                     Module  Enabled        7.x-1.4      
Search views (search_api_views)                                       Module  Enabled        7.x-1.4      
Solr search (search_api_solr)                                         Module  Enabled        7.x-1.0-rc2

Workaround:
Setting the number of content to index to 1, run the index and then set it back to 50.

Comment #3

ryan.gibson CreditAttribution: ryan.gibson commented 7 March 2013 at 15:12

Status:

Closed (cannot reproduce)

» Active

This is happening to me as well. Access denied on search.

Comment #4

cpliakas CreditAttribution: cpliakas commented 12 April 2013 at 02:58

Status:

Active

» Postponed (maintainer needs more info)

File	Size
debug-1678152-4.patch	737 bytes

Let's see if we can capture the pattern when searching. The attached patch will log a watchdog message with the appropriate info so we can detect the error. If you want to post the info here, make sure it is sanitized. Otherwise it might be safer to post it in a support ticket.

Comment #5

cpliakas CreditAttribution: cpliakas commented 14 May 2013 at 15:51

Status:

Postponed (maintainer needs more info)

» Active

After receiving some more info about this, it could be because Search API isn't updating the timestamp on post requests, so Acquia Search thinks that there is a replay attack. Just a guess, but looking at some Solr logs there is evidence to believe that this is the case. Investigating further.

Comment #6

cpliakas CreditAttribution: cpliakas commented 14 May 2013 at 16:14

Status:

Active

» Needs review

File	Size
indexing-403-1678152-6.patch	630 bytes

Kind of a shot in the dark, but if Search API isn't batching the requests then the timestamp might be stale. The attached patch swaps out the REQUEST_TIME constant for the time() function. I would appreciate testers who are experiencing this issue to privide feedback on whether this works or not. At the very least this will help us rule out this assumption.

Thanks,
Chris

Comment #7

rikvd CreditAttribution: rikvd commented 15 May 2013 at 09:44

Status:

Active

» Reviewed & tested by the community

the path to reproduce is to run a batch for massdeleting nodes that are indexed trough acquia SOLR using for example this script, for over 1500 nodes.

<?php
function _mymodule_node_delete($data, &$context) {
  $context['message'] = 'next batch';
  $query = new EntityFieldQuery;

  $result = $query->entityCondition('entity_type', 'node')
    ->range(0,250)
    ->execute();
  $nids = array_keys($result['node']);
  node_delete_multiple($nids);
}
?>

the patch mentioned in #6 fixed the problem.