Data storage overview
- xc_entity_properties
- xc_sql_metadata
- xc_location
- xc_source_locations
- xc_entity_relationships
- xc_oaiharvester_bridge_changes
xc_entity_properties table
Information about stored metadata entity instances. Defined in XC Metadata (xc_metadata) module. The central part of the metadata record storage on the MySQL part, because it stores the information about metadata (the metadata itself is stored in xc_sql_metadata table.
+----------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+------------------+------+-----+---------+----------------+
| metadata_id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| metadata_type | varchar(28) | NO | MUL | NULL | |
| node_id | int(10) unsigned | YES | MUL | NULL | |
| node_type | varchar(32) | NO | MUL | NULL | |
| identifier | varchar(255) | NO | UNI | NULL | |
| format | varchar(32) | NO | MUL | NULL | |
| source_id | varchar(32) | NO | MUL | NULL | |
| created | int(10) unsigned | NO | | NULL | |
| updated | int(10) unsigned | NO | | NULL | |
| locks | text | NO | | NULL | |
| locations | text | NO | | NULL | |
| properties | text | NO | | NULL | |
| built | tinyint(4) | NO | | 0 | |
| stored | tinyint(4) | NO | | 0 | |
| deleted | tinyint(4) | NO | | 0 | |
| identifier_int | int(10) unsigned | YES | MUL | 0 | |
+----------------+------------------+------+-----+---------+----------------+
- metadata_id
- Entity instance identifier. The main key of xc_entity_properties table, and the id of the XCEntity object. Same value is stored in Solr as metadata_id_i field.
- metadata_type
- Metadata entity type. The schema modules defined the the types. For Dublin Core the type is oai_dc, for XC schema records the types are work, expression, manifestation and holdings. The metadata type is the more common name of the type, while node type is a more technical name. Each node type matches to a metadata type - practically they contains the same information. The node type is used by node system, we use metadata type in several places in Drupal Toolkit including themes.
- node_id
- Related node identifier. If there is no related node, its value is 0.
- node_type
- Drupal node type. For XC schema records the node type's name is 'xc_' prefix plus the metadata type, like xc_work. With this prefix we can avoid conflicts with a 3rd party module's node type.
- identifier
- Entity identifier. Same as the identifier in the OAI response, typically a string like an URI, e.g. oai:mst.rochester.edu:marctoxctransformation/11656308.
- format
- Metadata format type. This field contain the name of the schema which contains the metadata type. The XC core modules provides oai_dc (for Dublin Core records) and xc (for all kinds of XC schema records)
- source_id
- Metadata source identifier. It refers to the xc_source table's source_id field.
- created
- The timestamp of the record's creation in Drupal.
- updated
- The timestamp of the record's last modification in Drupal.
- locks
- Metadata locks. It is an array in PHP serialized format. Currently unused, and contain only a serialized empty array. The intention was to contain some kind of flags to prevent access the data from other modules, but it was never implemented.
- locations
- Metadata locations if not associated with metadata source. It may contain an associative array of locations (with the location IDs as keys, and the xc_location record as value) in PHP serialized format. It is used when a record is imported individually by XC Export/Import module, but it is not used when the record is created as a result of batch import by OAI Harvester. In this later case the location is determined by the source_id value of the record (see xc_source, xc_source_locations and xc_location tables).
- properties
- Additional entity properities, which do not fit the table structure. They are stored in PHP serialized format. Currently the core XC modules do not store any values in this field, but it is possible that in the future we will use this, or other modules can use it for their's need.
- built
- Whether the entity has been built. "Build" is a process under which the Metadata module and the schema modules create an XCEntity object from the external XML source.
- stored
- Whether the entity has been stored. This means, that the metadata part of the entity is stored in a metadata location (in xc_sql_metadata table.
- deleted
- Whether the entity has been deleted. This means, that the metadata part of the entity is deleted from a location. It has only historical concerns, the Drupal Toolkit doesn't make use of it, and we will remove it in the future.
- identifier_int
- Normalized entity identifier. It means, that the module tries to exctract a numeric identifier from the OAI identifier.
xc_sql_metadata table
It stores the actual metadata of the entity record. The table is defined in XC SQL (xc_sql) module. I would like to call attention, that this table seems not normalized, because the connection between xc_entity_properties record and xc_sql_metadata record is 1:1. But the expreiences showed us, that to put a blob type field, the metadata into xc_entity_properties would slow down identifier lookup, and iterations over records. This is why we keep this separated structure.
The table structure:
+-------------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------+-------+
| metadata_id | int(10) unsigned | NO | PRI | NULL | |
| location_id | int(10) unsigned | NO | MUL | NULL | |
| metadata | longtext | NO | | NULL | |
| timestamp | int(10) unsigned | NO | | NULL | |
+-------------+------------------+------+-----+---------+-------+
- metadata_id
- Entity instance identifier. The main key of xc_entity_properties table, and theid of the XCEntity object. Same value is stored in Solr as metadata_id_i field.
- location_id
- Location identifier. The main key of xc_location table.
- metadata
- Metadata container. It contains the metadata part of the OAI record in PHP serialized format, which enables us to store any complex PHP data structure (object or array) in MySQL. Before storage the module convert the data to this textual format, when retrieves the records it converts back to PHP object.
- timestamp
- Entity instance last modified time.
xc_location table
Information about metadata storage locations. The table is defined in XC Metadata (xc_metadata) module. The XC module now provides two storage types: sql and solr, but with hooks it is possible to add more location types. If sql is enabled it means, that the record should be stored in the main Drupal database in the xc_sql_metadata table. If Solr is enabled XC stores records into a Solr server, you specified. The Solr servers are registered in xc_solr_servers table (the table's location_id field refers to this table location_id). You can administer the metadata locations at Administer > eXtensible Catalog (XC) > Metadata Storage Configuration and Utilities > Storage locations (admin/xc/metadata/location).
+-------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------+----------------+
| location_id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| name | text | NO | | NULL | |
| description | text | NO | | NULL | |
| types | text | NO | | NULL | |
+-------------+------------------+------+-----+---------+----------------+
- location_id
- Primary identifier for metadata storage location. This field is referenced in the xc_source_locations, and xc_solr_servers tables' location_id field.
- name
- Metadata storage location name.
- description
- Metadata storage location description.
- types
- Metadata storage types for this location. The field contains the list of enabled types in PHP serialized format. Currently the modules support the following types: sql and solr.
xc_source_locations table
Lookup table for metadata source and locations. The table is defined in XC Metadata (xc_metadata) module.
+-------------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------+-------+
| source_id | int(10) unsigned | NO | PRI | NULL | |
| location_id | int(10) unsigned | NO | PRI | NULL | |
| weight | int(11) | NO | PRI | 0 | |
+-------------+------------------+------+-----+---------+-------+
- source_id
- Metadata source identifier. This field is reference to the xc_source table's source_id field (the primary identifier of the xc_source).
- location_id
- Metadata storage location identifier. This field is reference to in the xc_location table's location_id field.
- weight
- Weight for ordering location priority.
xc_entity_relationships table
A table with parent-child pairs, which reflects the hierarchy between entities. The XC record schema makes use of OAI-PMH identifiers, and at the time of harvesting we can not has a lookup table to resolve another records identifier, so we store a derivated value from this identifier. Unfortunatelly OAI-PMH identifiers are usually strings, but searching strings in MySQL is much slower process than searching for numeric values, so we extract a numeric value from the identifier with the xc_util module's xc_util_get_identifier_int(). Usually the OAI identifiers match the following schema: [an OAI prefix][a number]. The OAI prefix part can be extract from the response to the Identify verb of the OAI-PMH standard, and Drupal Toolkit store this value. With this trick we can quickly find the parent-child relationships even in great number of records.
+--------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------+------------------+------+-----+---------+-------+
| parent | int(10) unsigned | NO | PRI | NULL | |
| child | int(10) unsigned | NO | PRI | NULL | |
+--------+------------------+------+-----+---------+-------+
- parent
- An identifier of an xc_entity_properties record. This field is reference to the xc_entity_properties table's identifier_int field (which is a unique identifier of an entity, and derivated by Drupal Toolkit from the OAI identifier of the record).
- child
- An identifier of an xc_entity_properties record. This field is reference to the xc_entity_properties table's identifier_int field (which is a unique identifier of an entity, and derivated by Drupal Toolkit from the OAI identifier of the record).
xc_oaiharvester_bridge_changes table
The purpose of this table to track changes during harvest for Solr. Since Solr contains a kind of rebuilt MARC, so each record belongs to one manifestation and this record merge all parent expressions and works of this manifestation, so if a work or an expression record is updated, we should recreate the Solr record as well. This table is not used in case of "initial harvest" (the first harvest of a data provider). The table has its own lifecycle.
- At the first phase of the harvest (in an implementation of
hook_oaiharvester_harvest_starting()), the table is emptied. - During the harvesting phase the table is fulfilled with references of each changed, deleted or new records (see the OAI Harvester Bridge module's
xc_oaiharvester_bridge_set_changes()function). Only manifestations have to be marked as deleted. Other records have to be marked as changed, since if a work is deleted, the manifestation might remain the same, so the Solr records should be recreated, but not deleted. - Because - as we mentioned - the Solr records are connected to manifestations, after the harvesting process, we have to normalize this table: to find the manifestations belonging to works end expressions. That's what happen when we import the CSV files into other tables, and this is what OAI Harvester Bridge module's
xc_oaiharvester_bridge_normalize_changes()function does. - The last phase is to apply the changes. When Solr indexer starts working, first it checks whether it is an initial harvest, or an incremental one. This information is stored in
xc_solr_iteration_strategiesDrupal variable. If this is an incremental harvest, then Solr have to process only the changes, which is stored in this table. First it iterates over the deletable records and deletes them, then iterates over the modified or newly created records, and (re)creates the necessary records.
Drupal Toolkit doesn't use this table after indexing.
+---------------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+------------------+------+-----+---------+-------+
| metadata_id | int(10) unsigned | NO | MUL | NULL | |
| metadata_type | tinyint(4) | NO | | 0 | |
| type | tinyint(4) | NO | | 0 | |
+---------------+------------------+------+-----+---------+-------+
- metadata_id
- Entity instance identifier. The main key of xc_entity_properties table, and the id of the XCEntity object. Same value is stored in Solr as metadata_id_i field.
- metadata_type
- Metadata entity type. The type is encoding by numbers. The possible values: 1 means work (in some places the PHP code use the constant XC_WORK), 2 means expression (XC_EXPRESSION), and 3 means manifestation (XC_MANIFESTATION).
- type
- The type of the change. The type is encoded by numbers. The possible values: 1 means deleted (in some places the PHP code use the constant XC_DELETED), 2 means updated (XC_UPDATED), and finally 3 means new record (XC_NEW)
Help improve this page
You can:
- Log in, click Edit, and edit this page
- Log in, click Discuss, update the Page status value, and suggest an improvement
- Log in and create a Documentation issue with your suggestion
