xc_installation 6.x-1.3

Version 6.x-1.3

The release has a primary focus on data integrity, namely being able to successfully process record updates on a schedule basis. This includes new additions, updates and deletions of records. This release includes some Solr integrity fixes submitted by Kyushu University. The installation process for release 1.3 has also been reworked to include an implementation option using Drush that makes the installation substantially easier.

Installation

The toolkit has four parts: the row of modules, a specific theme, a custom Apache Solr, and an Drupal distribution, which includes all necessary modules (XC and 3rd party ones), the current Drupal 6 core (6.27, with the necessary patch), and the theme. Previously the installation was quite complex, now we worked hard to make it easier.

The process is built on "drush", the Drupal command line tool. (see more: http://drupal.org/documentation/modules/drush).

The installation has two parts: a Drupal part, and a Solr part.

The Drupal part

If you have Drush:

$ cd /var/www    # the Apache httpd server's webroot directory
$ drush dl xc_installation-6.x-1.3 --drupal-project-rename=xc
$ cd xc
$ drush si xc_installation --db-url=mysqli://[mysql user]:[mysqlpw]@localhost/xc --site-name="eXtensible Catalog" --account-pass=admin
$ chmod a+w sites/default/files

Open http://localhost/xc in your browser. You can log in as admin/admin.

If you don't have Drush:

$ wget http://ftp.drupal.org/files/projects/xc_installation-6.x-1.3-core.tar.gz
$ tar -zxvf xc_installation-6.x-1.3-core.tar.gz
$ mv xc_installation-6.x-1.3 xc
$ mv xc /var/www/
$ cd /var/www/xc

From here you havet to follow the Drupal installation from Step 2.. You can get the same instructions in the INSTALL.txt and INSTALL.mysql.txt files in the current directory. Here we give you a very short list of command, which fits for the typical situations.

Create database:

$ mysqladmin -u username -p create xc
$ mysql -u username -p
$ GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES ON xc.* TO 'username'@'localhost' IDENTIFIED BY 'password';
$ FLUSH PRIVILEGES;
$ exit

Prepare files:

$ cp sites/default/default.settings.php sites/default/settings.php
$ chmod o+w sites/default/settings.php
$ chmod o+w sites/default
$ mkdir sites/default/files
$ chmod o+w sites/default/files

Open http://localhost/xc/install.php in your browser and follow the instructions. When it finished remove write access from the settings.php:

$ chmod a-w sites/default/settings.php

The Solr part:

$ cd ~/solr  # or any other convenient location
$ wget http://extensiblecatalog.org/apache-solr-3.6.1-xc-6.x-1.3.tar.gz
$ tar -xvzf apache-solr-3.6.1-xc-6.x-1.3.tar.gz
$ cd apache-solr-3.6.1-xc-6.x-1.3/
$ ./solr.start

It will start Solr at http://localhost:8983/solr/ with all necessary settings, and Java libraries the Drupal Toolkit requires. Of course you need Java runtime environment to run Apache Solr. (As you can see the package is double compressed, we will improve it [singe compression] in the final release.)

If you want to use the WordNet suggestions, you have to start a multicore Solr setup. To do that simply run

$ multicore-start.sh

instead of solr.start. It will start Solr as http://localhost:8983/solr/xc and http://localhost:8983/solr/wordnet. You have to modify Solr path the following way:
1) go to admin/xc/metadata/location/list
2) click on edit
3) go to next page
4) edit the "path" value to "/solr/xc"
5) save

To stop Solr run solr.stop (or multicore-stop.sh if you use multicore setup).

To import the harvested records into relational database the Toolkit can use CSV files, and LOAD DATA INFILE commands or the more traditionally INSERT command. The first is much more quicker, but it requires general (and not database) level privilege in MySQL, which is not available in all situation. If you chose this, be very careful with privilege settings. The INSERT is slower command, but it requires only database level privileges, so it is more secure. If you have millions of records, and time is important factor, consider using LOAD DATA INFILE with carefully, if you don't have as many record, or time is not an important factor, use the INSERT command. Some MySQL version supports the "LOCAL" modifier in LOAD DATA INFILE command. If you found problems with the LOCAL modifier, change to LOAD DATA INFILE (without the LOCAL modifier). You can select this settings at the second page of harvest schedule editing form (admin/xc/harvester/schedule/[schedule ID]/edit).

Changes in this release

Bug

• [DRUPAL-209] - Way to clear search and "Start" over
• [DRUPAL-273] - Stability of Drupal Toolkit - updates and deletes
• [DRUPAL-274] - Reconcile to stable MST - counts by record types
• [DRUPAL-275] - Reconcile after an "update" file has been processed.
• [DRUPAL-276] - Reconcile after a "delete" file has been processed.
• [DRUPAL-287] - Eliminate node creation when viewing the FRBR structure
• [DRUPAL-371] - Similar Items Number Display
• [DRUPAL-426] - Kyushu Solr enhancements (10 patches)
• [DRUPAL-433] - Cross Training for MT
• [DRUPAL-436] - The End Date property in schedule editing page is not working correctly
• [DRUPAL-437] - Make difference between the cases if there is no entity, and if there is no metadata for the entity
• [DRUPAL-438] - No harvest report created when it is launched by schedule
• [DRUPAL-439] - Documenting per harvest report on drupal.org
• [DRUPAL-443] - Put each OAI-PMH repositories into menu
• [DRUPAL-444] - Put each OAI schedules into menu
• [DRUPAL-445] - Rename menu item node/%node_id/xc/download to node/%node_id/xc/schema
• [DRUPAL-447] - Solving harvesting date recording problem
• [DRUPAL-449] - Metadata CSV file creation problem
• [DRUPAL-450] - Add link to Solr information
• [DRUPAL-451] - Change the 'per harvest report' to contain harvested record type statistics
• [DRUPAL-452] - Log Question
• [DRUPAL-453] - MST and Drupal summary stats reconcile
• [DRUPAL-455] - Delayed SQL Inserts - more details
• [DRUPAL-456] - Debugging and logging functions and variable - more details
• [DRUPAL-457] - Updates and deletes process - more details
• [DRUPAL-458] - Reconcile logs for deleted records
• [DRUPAL-459] - Metadata column question - why the numbers increase on an update
• [DRUPAL-460] - Documentation need: what is search_type and xc_search_type?
• [DRUPAL-461] - Documentation need: what does the solr param "XCNAME" do?
• [DRUPAL-462] - Documentation need: whats the difference between xc_search_form_validate and xc_search_form_search_validate?
• [DRUPAL-463] - Documentation need: how do you move back and forth between search results from the full record page?
• [DRUPAL-464] - Documentation need: what does xc_search_translate_field function do?
• [DRUPAL-465] - Documentation need: can you explain all the issues with apache mod_rewrite and why the "hacks" are needed for it?
• [DRUPAL-477] - Document on download page what skills are needed to succeed with install
• [DRUPAL-478] - Delete records in partial Voyager Pipeline
• [DRUPAL-485] - Search box, first try at "enter" results in no results, selecting "enter" again does
• [DRUPAL-486] - Update website to include the Windows XAMPP instructions
• [DRUPAL-488] - Security issues with release 1.2, need new step in our release process
• [DRUPAL-490] - Use of jquery, pulled out of issue 488
• [DRUPAL-495] - Update Solr schema and config to release 3.6.1, test that it works or not
• [DRUPAL-498] - Format facet does not contains "Book, text" value
• [DRUPAL-499] - Format facet wrongly behaves when user clicks on 'More...' link
• [DRUPAL-502] - XC Schema view not working
• [DRUPAL-504] - Solr indexing not completing
• [DRUPAL-505] - NCIP requests use old settings, even if admin changed the properties of the server
• [DRUPAL-506] - NCIP2 Test before DT release 1.3

Improvement

• [DRUPAL-480] - Server monitoring - what can be added that is helpful
• [DRUPAL-483] - Provide pre-configured Webserver Environment for use with Install Profile

New Feature

• [DRUPAL-48] - Record count - verification of exact numbers or records
• [DRUPAL-49] - Run end to end system - run records through entire process daily
• [DRUPAL-98] - Harvested records match MST
• [DRUPAL-99] - Harvested records internally consistent 0 Metadata ID, OAI ID, # of MySQL records
• [DRUPAL-100] - # of Nodes created compared to # Metadata IDs
• [DRUPAL-102] - Documentation on Drupal site
• [DRUPAL-216] - Document Data Storage
• [DRUPAL-217] - Document processing steps for 1 new record
• [DRUPAL-218] - Document Processing steps for batch of records
• [DRUPAL-220] - Document the module each processing step occurs in
• [DRUPAL-221] - Document updates of records
• [DRUPAL-222] - Document Deletes of records
• [DRUPAL-284] - If one result comes back, go to full record display
• [DRUPAL-286] - Document Kyushu url work

Task

• [DRUPAL-466] - DT Harvest Logs need to show total records by type after each harvest