Multicore Solr setup

Solr can be set up to have multiple configurations and search indexes on the same machine. Each configuration is called a Solr core. Having multiple cores is useful when you want different applications or different versions of CKAN to share the same Solr instance, each application can have its own Solr core so each can use a different schema.xml file. This is necessary, for example, if you want two CKAN instances to share the same Solr server and those two instances are running different versions of CKAN that require different schema.xml files, or if the two instances have different Solr schema customizations.

Each Solr core in a multicore setup will have a different URL, for example:

http://localhost:8983/solr/ckan_default
http://localhost:8983/solr/some_other_site

This section will show you how to create a multicore Solr setup and create your first core. If you already have a multicore setup and now you’ve setup a second CKAN instance on the same machine and want to create a second Solr core for it, see Creating another Solr core.

1. Create the file /usr/share/solr/solr.xml, with the following contents:

   <solr persistent="true" sharedLib="lib">
       <cores adminPath="/admin/cores">
           <core name="ckan_default" instanceDir="ckan_default">
               <property name="dataDir" value="/var/lib/solr/data/ckan_default" />
           </core>
       </cores>
   </solr>
   

   This file lists the different Solr cores, in this example we have just a single core called ckan_default.

2. Create the data directory for your Solr core, run this command in a terminal:

   sudo -u jetty mkdir /var/lib/solr/data/ckan_default
   

   This is the directory where Solr will store the search index files for our core.

3. Create the directory /etc/solr/ckan_default, and move the /etc/solr/conf directory into it:

   sudo mkdir /etc/solr/ckan_default
   sudo mv /etc/solr/conf /etc/solr/ckan_default/
   

   This directory holds the configuration files for your Solr core.

4. Replace the /etc/solr/ckan_default/schema.xml file with a symlink to CKAN’s schema.xml file:

   sudo mv /etc/solr/ckan_default/conf/schema.xml /etc/solr/ckan_default/conf/schema.xml.bak
   sudo ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml /etc/solr/ckan_default/conf/schema.xml
   

5. Edit /etc/solr/ckan_default/conf/solrconfig.xml and change the <dataDir> tag to this:

   <dataDir>${dataDir}</dataDir>
   

   This configures our ckan_default core to use the data directory you specified for it in solr.xml.

6. Create the directory /usr/share/solr/ckan_default and put a symlink to the conf directory in it:

   sudo mkdir /usr/share/solr/ckan_default
   sudo ln -s /etc/solr/ckan_default/conf /usr/share/solr/ckan_default/conf
   

7. Restart Solr:

   sudo service jetty restart
   

   You should now see your newly created ckan_default core if you open http://localhost:8983/solr/ckan_default/admin/ in your web browser. You can click on the schema link on this page to check that the core is using the right schema (you should see <schema name="ckan" version="2.0"> near the top of the schema.xml file). The http://localhost:8983/solr/ page will list all of your configured Solr cores.

8. Finally, change the solr_url setting in your /etc/ckan/default/development.ini or /etc/ckan/default/production.ini file to point to your new Solr core, for example:

   solr_url = http://127.0.0.1:8983/solr/ckan_default
   

If you have trouble when setting up Solr, see Multicore Solr setup troubleshooting below.

Creating another Solr core

In this example we’ll assume that:

1. You’ve followed the instructions in Multicore Solr setup to create a multicore setup and create your first core for your first CKAN instance.
2. You’ve installed a second instance of CKAN in a second virtual environment at /usr/lib/ckan/my-second-ckan-instance, and now want to setup a second Solr core for it.

You can of course follow these instructions again to setup further Solr cores.

1. Add the core to /usr/share/solr/solr.xml. This file should now list two cores. For example:

   <solr persistent="true" sharedLib="lib">
       <cores adminPath="/admin/cores">
           <core name="ckan_default" instanceDir="ckan_default">
               <property name="dataDir" value="/var/lib/solr/data/ckan_default" />
           </core>
           <core name="my-second-solr-core" instanceDir="my-second-solr-core">
               <property name="dataDir" value="/var/lib/solr/data/my-second-solr-core" />
           </core>
       </cores>
   </solr>
   

2. Create the data directory for your new core:

   sudo -u jetty mkdir /var/lib/solr/data/my-second-solr-core
   

3. Create the configuration directory for your new core, and copy the config from your first core into it:

   sudo mkdir /etc/solr/my-second-solr-core
   sudo cp -R /etc/solr/ckan_default/conf /etc/solr/my-second-solr-core/
   

4. Replace the /etc/solr/my-second-solr-core/schema.xml file with a symlink to the schema.xml file from your second CKAN instance:

   sudo rm /etc/solr/my-second-solr-core/conf/schema.xml
   sudo ln -s /usr/lib/ckan/my-second-ckan-instance/src/ckan/ckan/config/solr/schema.xml /etc/solr/my-second-solr-core/conf/schema.xml
   

5. Create the /usr/share/solr/my-second-solr-core directory and put a symlink to the conf directory in it:

   sudo mkdir /usr/share/solr/my-second-solr-core
   sudo ln -s /etc/solr/my-second-solr-core/conf /usr/share/solr/my-second-solr-core/conf
   

6. Restart Solr:

   sudo service jetty restart
   

   You should now see both your Solr cores when you open http://localhost:8983/solr/ in your web browser.

7. Finally, change the solr_url setting in your /etc/ckan/my-second-ckan-instance/development.ini or /etc/ckan/my-second-ckan-instance/production.ini file to point to your new Solr core:

   solr_url = http://127.0.0.1:8983/solr/my-second-solr-core
   

If you have trouble when setting up Solr, see Multicore Solr setup troubleshooting.

Multicore Solr setup troubleshooting

See also

Troubleshooting for single-core Solr setups

Most of these tips also apply to multi-core setups.

No cores shown on Solr index page

If no cores are shown when you visit the Solr index page, and the admin interface returns a 404 error, check the web server error log (/var/log/jetty/<date>.stderrout.log if you’re using Jetty, or /var/log/tomcat6/catalina.<date>.log for Tomcat). If you can find an error similar to this one:

WARNING: [iatiregistry.org] Solr index directory '/usr/share/solr/iatiregistry.org/data/index' doesn't exist. Creating new index...
07-Dec-2011 18:06:33 org.apache.solr.common.SolrException log
SEVERE: java.lang.RuntimeException: Cannot create directory: /usr/share/solr/iatiregistry.org/data/index
[...]

Then dataDir is not properly configured. With our setup the data directory should be under /var/lib/solr/data. Make sure that you defined the correct dataDir in the solr.xml file and that in the solrconfig.xml file you have the following configuration option:

<dataDir>${dataDir}</dataDir>