Hi,

Trying to build the sparql index for a large database, I am running into scalability issues. I have 400,000 nodes. I can run the build rdf index ../drupal/sparql_endpoint_index via the cron using wget which is fine, but it ends prematurely after a few seconds. I see in ../drupal/sites/all/modules/sparql/sparql_endpoint/sparql_endpoint.module

line 34
->limit(500)
...which I can change to
->limit(400000)

but it then times out with a php memory limit. I raised this from 128MB to 2000MB (the limit
of the server) and it manages about 250,000 rows in the table sparql_store_site_endpoint_triple. At about 25 triples per record, that means it has indexed about 10k of my 400k records. Is there anything I can do to progress?

thanks!

Comments

scor’s picture

scor commented
Project: RDF Extensions » SPARQL
Version: 7.x-2.0-alpha4 » 7.x-2.x-dev
Category: support » feature

could you try the patch at #1389718: rebuild endpoint does not cover all entities and see if it helps?

bers’s picture

bers commented

Thanks for the tip. I installed the patch and ran the rebuild a couple of times. Each time it produced 25,756 triples before saying it was rebuilt. I then tried the latest version of that patch on

https://drupal.org /node/1389718#comment-6270218 This time

...it produced 25,777 and hung at 73%. The watchdog gave:

EntityMetadataWrapperException: Invalid data value given. Be sure it matches the required data type and format. in
EntityDrupalWrapper->set() (line 735 of /var/www/html/drupal/sites/all/modules/entity/includes/entity.wrapper.inc).

scor’s picture

scor commented

looks like you might have a entity that rdfx is not able to translate into RDF for some reason. You could try #1700652: rdfx_get_rdf_model() generates errors on stale references and see if it helps, but not 100% sure. The best thing to do otherwise would be to log the nids or entity_ids as sparql_endpoint indexes them and find out which entity is causing this error, then debug the RDF output of that entity. are you indexing nodes and other entity types?

bers’s picture

bers commented

Hi,

I only need node entities so I rolled back to patch 2 https://drupal.org/node/1389718#comment-5653104
This got rid of the error in #2 and appears to complete but is only putting 25,756 records in the
sparql_store_site_endpoint_triple mysql table and completes in about 2 mins which sounds unlikely.
If I run via wget, it does show:

Saving to: `sparql_endpoint_index'

0K .......... .......... .......... .......... .......... 228M
50K .......... .......... .......... .......... .......... 274M
100K .......... .......... .......... .......... .......... 298M
150K .......... .......... .......... .......... .......... 298M
200K .......... .......... .......... .......... .......... 296M
250K .......... .......... .......... .......... .......... 45.7M
300K .......... .......... .......... .......... .......... 321M
350K .......... .......... .......... .......... .......... 367M
400K .......... .......... .......... .... 305M=0.002s

2012-09-13 14:00:43 (181 MB/s) - `sparql_endpoint_index' saved [444734]

... but it seems it skips most of my rdf mappings. Is there anywhere where I can see which rdf mappings from the content type it is actually trying to use?

scor’s picture

scor commented

yes, run this query to see what mappings were used for a given node:

DESCRIBE <http://yoursite.com/node/1>

That should be the same as what you get at http://yoursite.com/node/1.rdf

bers’s picture

bers commented

If I run

select val from drupal.sparql_store_site_endpoint_id2val

I can see node, users and taxonomy. All still existing "nodes" up to 7931 are recorded.

So

DESCRIBE <http://oemlv01.ioe.ulcc.ac.uk/drupal/node/7931> gives rdf but...
DESCRIBE <http://oemlv01.ioe.ulcc.ac.uk/drupal/node/7932> gives nothing

Those two records were created the same way, same content type and are very similar. See

http://oemlv01.ioe.ulcc.ac.uk/drupal/node/7931
http://oemlv01.ioe.ulcc.ac.uk/drupal/node/7932

3597 node rows were created. Is there any way I can see what it is selecting as nodes to process before it builds as I can see no reason for it to think it has finished at 7931.

Note: My sprql endpoint is at http://oemlv01.ioe.ulcc.ac.uk/drupal/sparql

bers’s picture

bers commented

Just to add to this, I can verify that it actually creates 3300 nodes in chunks of 300 before deciding there are no more:

Processed 0 node entities.
Processed 300 node entities.
Processed 600 node entities.
Processed 900 node entities.
Processed 1200 node entities.
Processed 1500 node entities.
Processed 1800 node entities.
Processed 2100 node entities.
Processed 2400 node entities.
Processed 2700 node entities.
Processed 3000 node entities.
Processed 3300 node entities.
Processed 0 user entities.
Processed 0 taxonomy_term entities.
The RDF index of the site has been rebuilt. Browse to the SPARQL endpoint to query it.