Queries are using a write lock
| Project: | xapian |
| Version: | 5.x-1.x-dev |
| Component: | Code |
| Category: | bug report |
| Priority: | normal |
| Assigned: | Jeremy |
| Status: | active |
Jump to:
In using the Xapian module on the Drupal.org infrastructure, we've run into an issue where Xapian is trying to grab a write lock even when it simply wants to do what should be a read-only search. We first detected this because the following error was showing up very frequently:
DatabaseLockError: REMOTE:Unable to acquire database write lock on /var/lib/xapian: already locked
I provided a debug version of the Xapian module to better understand what was going on, and we traced the source of these errors coming from the xapian_query() function. The xapian_query() function calls _xapian_init_database() with no parameters, which means it should be requesting a read-only copy of the database.
I'm still unclear where the problem lies. A few ideas regarding what could be causing these problems:
- we do not initialize $database or $writable_database, so a stale version could be left sitting around
- the call to Xapian::remote_open() is for some reason opening a writable database
- there's some other flow bug in this function that I'm missing
#1
Another idea I'm exploring: if you launch "xapian-tcpsrv --writable", does this force all connections to take the writable lock? Do we need to connect to two different tcpsrv instances? One for writable connections, and one for non-writable connections?
#2
This bug report confirms that #1 is indeed the case. The bug is marked as fixed three months ago in the development version of Xapian (1.1), while we're using the latest stable version (1.0.6).
The solution is either 1) upgrade to the unstable development version of Xapian, or 2) update the Xapian module to require a two database definitions, one for writable queries and one for read-only queries.
I'm going with the second option for now, as I'd like to use the stable search engine code.
#3
Fix committed. It is now possible to (optionally) configure a separate write-only database. While this was implemented to work around a bug in the 1.0.x xapian-tcpsrv code, it also has the nice side effect of greatly increasing the potential scalability of Xapian powered search.
For example, it is now possible to have one server dedicated to indexing content with Xapian (one of your web servers, or any other server). Using rsync on a regular cron-job, synchronize these search indexes onto all of your web servers. Finally, configure each of your web servers to search using their local copy of the Xapian indexes, only sending writes to the remote write-only server. If at any time your index server fails, you can easily turn another server into your index server utilizing an existing copy of the indexes. In this way, the search load is spread across all of your web servers, and even if your index server fails search continues working and no indexes have to be re-generated.
#4
Automatically closed -- issue fixed for two weeks with no activity.
#5
This is not actually fixed in the current code base for both D5 and D6. The code that uses the settings is missing.