Opening this to have a discussion around the future...

Not used sphinxsearch for a while, but when I did I was able to index a site with millions of rows at the rate of about 240,000 nodes per hour on D5 + ubercart site.

When I did a similar test recently using apachesolr + apachesolr_ubercart + facetapi on D7, I was only getting about 30,000 nodes per hour.

However, apachesolr has a lot more modules/support for drupal right now. Not an exact scientific comparison, but gives a rough idea...

A few questions:
- Is indexing in sphinx inherently much faster than apachesolr? If so, how/why?

- Is there any support planned support for sphinx in search_api, facetapi?

Any thoughts? Thanks.

Comments

markus_petrux’s picture

markus_petrux commented

In regards to Search API... it seems to me that's the future, but I'm not sure I'll have the time to work on that direction, at least until a project requires me to do so. Anyone who wishes to work on that, I can give developer access to this project if that helps.

I have not compared performance of apachesolr, so I cannot really tell. One difference, I think is Apache Solr is Java, and Sphinx Search is compiled C. The later begin a bit more faster for that reason, I would guess, and probably easier to install and maintain.

Latest changes I introduced in Sphinx Search module allow to build the XMLPipe stream from the PHP CLI, and that's faster than before, when it was under the context of Apache. Also, latest versions of the Sphinx Search package are a lot more optimized than before. They are doing a really great job.

coderintherye’s picture

coderintherye commented

Do you foresee any option of this going away from XML-PIPE and instead towards allowing for multiple types, like straight database queries?

I've just completed hacking your D5 version of the module to bypass xmlpipe completely, and use the indexes I set up in the sphinx conf, which run over the node and content_type_* tables. I feel like with D7 being entity based that it could be fairly straight-forward to just get a list of entities via some of the Entity API and generate queries for those on a node type basis, then index accordingly. Easier/Faster for an end user than going through xmlpipe, in my opinion.

I'm open to contributing for D7. I probably won't post up my D5 code since it's a hack on your module, though I could post up the sphinx conf for doing things via MySQL.

coderintherye’s picture

coderintherye commented

Well, I'm going to be moving forward with a D7 port, I hope you'll give some feedback about bypassing the xml-pipe

I'm going to try interfacing with the Search API module.

zeezhao’s picture

zeezhao commented

I just came across this: http://drupal.org/project/search_api_sphinx
Any overlap?

coderintherye’s picture

coderintherye commented
Status: Active » Closed (won't fix)

Oh nice, good find! I'm going to close this in favor of using the search api sphinx module.

zeezhao’s picture

zeezhao commented

Let me know how it goes. I had a quick look and it appears needs php 5.3+ (see README) even though not a condition info file. Also was not sure what sphinx.conf should look like give that used xmlpipe_command previously. Maybe an sql_query?

markus_petrux’s picture

markus_petrux commented

Using sql_query may not work well when using a lot of fields and/or different content types. XMLPipe allows you to preprocess any kind of data.

coderintherye’s picture

coderintherye commented

Personally, when I got this running on 5.x, I had to remove the XML pipe stuff to get anything to work with sphinx. After, I removed it and instead put SQL queries, everything worked great, so for my case this new module will be fine.

zeezhao’s picture

zeezhao commented

I tried out a small test with php 5.2 using the default rt that came with the module. So no sql or XML pipe. It worked for default, then I had some problems when I added extra fields.

See my post here for details of patch for 5.2 and issue I had: http://drupal.org/node/1863990

In any case test was just for a few nodes, but it seems you can get it to work without writing any sql yourself as the configuration of search_api index/server generates a sphinx.conf.