Understand and Write Extensions

If you want to extend CKAN core functionality, the best way to do so is by writing extensions.

Extensions allow you to customise CKAN for your own requirements, without interfering with the basic CKAN system.

To meet the need to customize CKAN efficiently, we have introduced the concepts of CKAN extensions and plugin interfaces. These work together to provide a simple mechanism to extend core CKAN functionality.

Warning

This is an advanced topic. We are working to make the most popular extensions more easily available as Debian packages.

Note

The terms extension and plugin interface have very precise meanings: the use of the generic word plugin to describe any way in which CKAN might be extended is deprecated.

CKAN Extensions

Extensions are implemented as namespace packages under the ckanext package which means that they can be imported like this:

$ python
>>> import ckanext.example

Individual CKAN extensions may implement one or more plugin interfaces to provide their functionality.

Creating CKAN Extensions

All CKAN extensions must start with the name ckanext-. You can create your own CKAN extension like this:

(pyenv)$ paster create -t ckanext ckanext-myextension

You’ll get prompted to complete a number of variables which will be used in your dataset. You change these later by editing the generated setup.py file. Here’s some example output:

Selected and implied templates:
  ckan#ckanext  CKAN extension project template

Variables:
  egg:      ckanext_myextension
  package:  ckanextmyextension
  project:  ckanext-myextension
Enter version (Version (like 0.1)) ['']: 0.4
Enter description (One-line description of the package) ['']: Great extension package
Enter author (Author name) ['']: James Gardner
Enter author_email (Author email) ['']: [email protected]
Enter url (URL of homepage) ['']: http://jimmyg.org
Enter license_name (License name) ['']: GPL
Creating template ckanext
Creating directory ./ckanext-myextension
  Directory ./ckanext-myextension exists
  Skipping hidden file pyenv/src/ckan/ckan/pastertemplates/template/.setup.py_tmpl.swp
  Recursing into ckanext
    Creating ./ckanext-myextension/ckanext/
    .svn/ does not exist; cannot add directory
    Recursing into +project+
      Creating ./ckanext-myextension/ckanext/myextension/
      .svn/ does not exist; cannot add directory
      Copying __init__.py to ./ckanext-myextension/ckanext/myextension/__init__.py
      .svn/ does not exist; cannot add file
    Copying __init__.py to ./ckanext-myextension/ckanext/__init__.py
    .svn/ does not exist; cannot add file
  Copying setup.py_tmpl to ./ckanext-myextension/setup.py
  .svn/ does not exist; cannot add file
Running pyenv/bin/python setup.py egg_info

Once you’ve run this, you should now install the extension in your virtual environment:

(pyenv)$ cd ckanext-myextension
(pyenv)$ python setup.py develop
(pyenv)$ python
Python 2.6.6 (r266:84292, Oct  6 2010, 16:19:55)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ckanext.myextension
>>>

Note

Running python setup.py develop will add a .egg-link file to your python site-packages directory (which is on your python path). This allows your extension to be imported and used, with any changes made to the extension source code showing up immediately without needing to be reinstalled, which is very useful during development.

To instead install a python package by copying all of the files to the site-packages directory run python setup.py install.

To build useful extensions you need to be able to “hook into” different parts of CKAN in order to extend its functionality. You do this using CKAN’s plugin architecture. We’ll look at this in the next section.

Plugins: An Overview

Plugin interfaces provide a specification which extensions can implement in order to “hook into” core CKAN functionality.

The CKAN plugin implementation is based on the PyUtilib component architecture (PCA). Here’s a quick summary, we’ll go through all this in much more detail in a minute:

  1. The CKAN core contains various plugin interfaces, each specifying a set of methods where plugins may hook into the software. For example a plugin wanting to hook into the SQLAlchemy mapping layer would need to implement the IMapperExtension interface.
  2. A plugin is a class that derives from ckan.plugins.Plugin or more commonly SingletonPlugin. It must also implement one of the plugin interfaces exposed in ckan.plugins.interfaces. The choice interface determines the functionality the plugin is expected to provide.
  3. Plugin objects must be registered as setuptools entry points. The ckan.plugins configuration directive is searched for names of plugin entry points to load and activate.

Here’s a list of some of the more commonly used plugin interfaces:

IDatasetForm
Provide a custom dataset form and schema.
IMapper
Listens and react to every database change.
IRoutes and IController
Provide an implementation to handle a particular URL.
IGenshiStreamFilter
Intercept template rendering to modify the output.
IDomainObjectModification
Listens for changes to CKAN domain objects.
IGroupController
Plugins for in the groups controller. These will usually be called just before committing or returning the respective object, i.e. all validation, synchronization and authorization setup are complete.
IConfigurable
Pass configuration to plugins and extensions.
IAuthorizer
Allows customisation of the default Authorization behaviour.

If you look in ckan/plugins/interfaces.py you will see the latest plugin interfaces. Alternatively see the Plugin API documentation below.

Example CKAN Extension

A example CKAN extension can be found at http://github.com/okfn/ckanext-example. Have a look at the README file for installation instructions.

Publishing Extensions

At this point you might want to share your extension with the public.

First check you have chosen an open source licence (e.g. the MIT licence) and then update the long_description variable in setup.py to explain what the extension does and which entry point names a user of the extension will need to add to their ckan.plugins configuration.

Once you are happy, run the following commands to register your extension on the Python Package Index:

python setup.py register
python setup.py sdist upload

You’ll then see your extension at http://pypi.python.org/pypi. Others will be able to install your plugin with pip.

Finally, please also add a summary of your extension and its entry points to the Extensions page at http://wiki.ckan.org/Extensions.

Writing a Plugin Interface

This describes how to add a plugin interface to make core CKAN code pluggable.

Suppose you have a class such as this:

class DataInput(object):

    def accept_new_data(self, data):
        self.data = data

And you want plugins to hook into accept_new_data to modify the data.

You would start by declaring an interface specifying the methods that plugin classes must provide. You would add the following code to ckan/plugins/interfaces.py:

class IDataMunger(Interface):

    def munge(self, data):
        return data

Now you can tell this class that its plugins are anything that implements IDataMunger like this:

from ckan.plugins import PluginImplementations, IDataMunger

class DataInput(object):

    plugins = PluginImplementations(IDataMunger)

    def accept_new_data(self, data):
       for plugin in self.plugins:
           data = plugin.munge(data)
       self.data = data

Any registered plugins that implement IDataMunger will then be available in your class via self.plugin.

See the pyutilib documentation for more information on creating interfaces and plugins. However, be aware that pyutilib uses slightly different terminology. It calls PluginImplementations ExtensionPoint and it calls instances of a plugin object a service.

Testing

Testing CKAN Extensions

CKAN extensions ordinarily have their own test.ini that refers to the CKAN test.ini, so you can run them in exactly the same way. For example:

cd ckanext-dgu
nosetests ckanext/dgu/tests --ckan
nosetests ckanext/dgu/tests --ckan --with-pylons=test-core.ini

To test your changes you’ll need to use the paster serve command from the ckan directory:

paster serve --reload -c <path to your CKAN config file>

You should also make sure that your CKAN installation passes the developer tests, as described in Testing for Developers.

Testing Plugins

When writing tests for your plugin code you will need setup and teardown code similar to the following to ensure that your plugin is loaded while testing:

from ckan import plugins

class TestMyPlugin(TestCase):

   @classmethod
   def setup_class(cls):
       # Use the entry point name of your plugin as declared
       # in your package's setup.py
       plugins.load('my_plugin')

   @classmethod
   def teardown_class(cls):
       plugins.reset()

The exception to using plugins.load() is for when your plug-in is for routes. In this case, the plugin must be configured before the WSGI app is started. Here is an example test set-up:

from paste.deploy import appconfig
import paste.fixture
from ckan.config.middleware import make_app
from ckan.tests import conf_dir

class TestMyRoutesPlugin(TestCase):

    @classmethod
    def setup_class(cls):
        config = appconfig('config:test.ini', relative_to=conf_dir)
        config.local_conf['ckan.plugins'] = 'my_routes_plugin'
        wsgiapp = make_app(config.global_conf, **config.local_conf)
        cls.app = paste.fixture.TestApp(wsgiapp)

At this point you should be able to write your own plugins and extensions together with their tests.

Ordering of Extensions

Caution

The order in which extensions are initially loaded is different to the order that their plugins are run.

The order in which extensions are initially loaded is as follows:

  1. System plugins (in setup.py under ckan.system_plugins).
  2. In order of the plugins specified in the config file: plugins =.
  3. If more than one module has a plugin with the same name specified in the config, then all those are loaded, in the order the modules appear in sys.path.

The order that a plugins are run in, for example the order that IRoutes extensions have their before_map method run, is alphabetical by the plugin class.

e.g. here is the order for these four extensions: <Plugin DguInventoryPlugin>, <Plugin FormApiPlugin>, <Plugin StatsPlugin>, <Plugin WalesThemePlugin>

(This alphabetical ordering is done by pyutilib.component.core:ExtensionPoint.extensions())

Plugin API Documentation

Libraries Available To Extensions

As well as using the variables made available to them by implementing various plugin hooks, extensions will likely want to be able to use parts of the CKAN core library. To allow this, CKAN provides a stable set of modules that extensions can use safe in the knowledge the interface will remain stable, backward-compatible and with clear deprecation guidelines as development of CKAN core progresses. This interface is available in ckan.plugins.toolkit.toolkit.

Guidelines for writing extensions:

  • Use the plugins toolkit, described above.
  • Extensions should use actions where possible via get_action().
  • No foreign key constraints into core as these cause problems.
  • The Session object is available through the toolkit.

Core Plugin Reference

Provides plugin services to the CKAN

ckan.plugins.core.PluginImplementations

alias of ExtensionPoint

ckan.plugins.core.implements(interface, namespace=None, inherit=False, service=True)

Can be used in the class definition of Plugin subclasses to declare the extension points that are implemented by this interface class.

If the inherits option is True, then this Plugin class inherits from the interface class.

exception ckan.plugins.core.PluginNotFoundException

Raised when a requested plugin cannot be found.

class ckan.plugins.core.Plugin(**kwargs)

Base class for plugins which require multiple instances.

Unless you need multiple instances of your plugin object you should probably use SingletonPlugin.

class ckan.plugins.core.SingletonPlugin(**kwargs)

Base class for plugins which are singletons (ie most of them)

One singleton instance of this class will be created when the plugin is loaded. Subsequent calls to the class constructor will always return the same singleton instance.

ckan.plugins.core.load(plugin)

Load a single plugin, given a plugin name, class or instance

ckan.plugins.core.load_all(config)

Load all plugins listed in the ‘ckan.plugins’ config directive.

ckan.plugins.core.unload(plugin)

Unload a single plugin, given a plugin name, class or instance

ckan.plugins.core.unload_all()

Unload (deactivate) all loaded plugins

ckan.plugins.core.reset()

Clear and reload all configured plugins

CKAN Interface Reference

Interfaces for plugins system See doc/plugins.rst for more information

class ckan.plugins.interfaces.IGenshiStreamFilter

Hook into template rendering. See ckan.lib.base.py:render

filter(stream)

Return a filtered Genshi stream. Called when any page is rendered.

Parameters:stream – Genshi stream of the current output document
Returns:filtered Genshi stream
class ckan.plugins.interfaces.IRoutes

Plugin into the setup of the routes map creation.

after_map(map)

Called after routes map is set up. after_map can be used to add fall-back handlers.

Parameters:map – Routes map object
Returns:Modified version of the map object
before_map(map)

Called before the routes map is generated. before_map is before any other mappings are created so can override all other mappings.

Parameters:map – Routes map object
Returns:Modified version of the map object
class ckan.plugins.interfaces.IMapper

A subset of the SQLAlchemy mapper extension hooks. See http://www.sqlalchemy.org/docs/05/reference/orm/interfaces.html#sqlalchemy.orm.interfaces.MapperExtension

Example:

>>> class MyPlugin(SingletonPlugin):
...
...     implements(IMapper)
...
...     def after_update(self, mapper, connection, instance):
...         log("Updated: %r", instance)
after_delete(mapper, connection, instance)

Receive an object instance after that instance is DELETEed.

after_insert(mapper, connection, instance)

Receive an object instance after that instance is INSERTed.

after_update(mapper, connection, instance)

Receive an object instance after that instance is UPDATEed.

before_delete(mapper, connection, instance)

Receive an object instance before that instance is DELETEed.

before_insert(mapper, connection, instance)

Receive an object instance before that instance is INSERTed into its table.

before_update(mapper, connection, instance)

Receive an object instance before that instance is UPDATEed.

class ckan.plugins.interfaces.ISession

A subset of the SQLAlchemy session extension hooks.

after_begin(session, transaction, connection)

Execute after a transaction is begun on a connection

after_commit(session)

Execute after a commit has occured.

after_flush(session, flush_context)

Execute after flush has completed, but before commit has been called.

after_rollback(session)

Execute after a rollback has occured.

before_commit(session)

Execute right before commit is called.

before_flush(session, flush_context, instances)

Execute before flush process has started.

class ckan.plugins.interfaces.IMiddleware

Hook into Pylons middleware stack

make_middleware(app, config)

Return an app configured with this middleware

class ckan.plugins.interfaces.IAuthFunctions

Allow customisation of default Authorization implementation

get_auth_functions()

Returns a dict of all the authorization functions which the implementation overrides

class ckan.plugins.interfaces.IDomainObjectModification

Receives notification of new, changed and deleted datesets.

class ckan.plugins.interfaces.IGroupController

Hook into the Group controller. These will usually be called just before committing or returning the respective object, i.e. all validation, synchronization and authorization setup are complete.

before_view(pkg_dict)

Extensions will recieve this before the group gets displayed. The dictionary passed will be the one that gets sent to the template.

class ckan.plugins.interfaces.IPackageController

Hook into the package controller. (see IGroupController)

Extensions will receive the search results, as well as the search parameters, and should return a modified (or not) object with the same structure:

{‘count’: ‘’, ‘results’: ‘’, ‘facets’: ‘’}

Note that count and facets may need to be adjusted if the extension changed the results for some reason.

search_params will include an extras dictionary with all values from fields starting with ext_, so extensions can receive user input from specific fields.

before_index(pkg_dict)

Extensions will receive what will be given to the solr for indexing. This is essentially a flattened dict (except for multli-valued fields such as tags) of all the terms sent to the indexer. The extension can modify this by returning an altered version.

Extensions will receive a dictionary with the query parameters, and should return a modified (or not) version of it.

search_params will include an extras dictionary with all values from fields starting with ext_, so extensions can receive user input from specific fields.

before_view(pkg_dict)

Extensions will recieve this before the dataset gets displayed. The dictionary passed will be the one that gets sent to the template.

class ckan.plugins.interfaces.IPluginObserver

Plugin to the plugin loading mechanism

after_load(service)

Called after a plugin has been loaded. This method is passed the instantiated service object.

after_unload(service)

Called after a plugin has been unloaded. This method is passed the instantiated service object.

before_load(plugin)

Called before a plugin is loaded This method is passed the plugin class.

before_unload(plugin)

Called before a plugin is loaded This method is passed the plugin class.

class ckan.plugins.interfaces.IConfigurable

Pass configuration to plugins and extensions

configure(config)

Called by load_environment

class ckan.plugins.interfaces.IConfigurer

Configure CKAN (pylons) environment via the pylons.config object

update_config(config)

Called by load_environment at earliest point when config is available to plugins. The config should be updated in place.

Parameters:configpylons.config object
class ckan.plugins.interfaces.IAuthorizer

Allow customisation of default Authorization implementation

get_authorization_groups(username)

Called by Authorizer to extend the list of groups to which a user belongs. Should return a list of AuthorizationGroups.

get_roles(username, domain_obj)

Called by Authorizer to extend the list of roles which a user has in the context of the supplied object. Should return a list of strings which are the names of valid UserObjectRoles.

is_authorized(username, action, domain_obj)

Called by Authorizer to assert that a user `username` can perform action` on `domain_obj`.

Should return True or False. A value of False will allow other Authorizers to run; True will shortcircuit and return.

class ckan.plugins.interfaces.IActions

Allow adding of actions to the logic layer.

get_actions()

Should return a dict, the keys being the name of the logic function and the values being the functions themselves.

By decorating a function with the ckan.logic.side_effect_free decorator, the associated action will be made available by a GET request (as well as the usual POST request) through the action API.

class ckan.plugins.interfaces.IResourceUrlChange

Receives notification of changed urls.

class ckan.plugins.interfaces.IDatasetForm

Allows customisation of the package controller as a plugin.

The behaviour of the plugin is determined by 5 method hooks:

  • package_form(self)
  • form_to_db_schema(self)
  • db_to_form_schema(self)
  • check_data_dict(self, data_dict, schema=None)
  • setup_template_variables(self, context, data_dict)

Furthermore, there can be many implementations of this plugin registered at once. With each instance associating itself with 0 or more package type strings. When a package controller action is invoked, the package type determines which of the registered plugins to delegate to. Each implementation must implement two methods which are used to determine the package-type -> plugin mapping:

  • is_fallback(self)
  • package_types(self)

Implementations might want to consider mixing in ckan.lib.plugins.DefaultDatasetForm which provides default behaviours for the 5 method hooks.

check_data_dict(data_dict, schema=None)

Check if the return data is correct.

raise a DataError if not.

comments_template()

Returns a string representing the location of the template to be rendered for the comments page

db_to_form_schema()

Returns the schema for mapping package data from the database into a format suitable for the form (optional)

form_to_db_schema()

Returns the schema for mapping package data from a form to a format suitable for the database.

history_template()

Returns a string representing the location of the template to be rendered for the history page

is_fallback()

Returns true iff this provides the fallback behaviour, when no other plugin instance matches a package’s type.

There must be exactly one fallback controller defined, any attempt to register more than one will throw an exception at startup. If there’s no fallback registered at startup the ckan.lib.plugins.DefaultDatasetForm is used as the fallback.

new_template()

Returns a string representing the location of the template to be rendered for the new page

package_form()

Returns a string representing the location of the template to be rendered. e.g. “package/new_package_form.html”.

package_types()

Returns an iterable of package type strings.

If a request involving a package of one of those types is made, then this plugin instance will be delegated to.

There must only be one plugin registered to each package type. Any attempts to register more than one plugin instance to a given package type will raise an exception at startup.

read_template()

Returns a string representing the location of the template to be rendered for the read page

search_template()

Returns a string representing the location of the template to be rendered for the search page (if present)

setup_template_variables(context, data_dict)

Add variables to c just prior to the template being rendered.

class ckan.plugins.interfaces.IGroupForm

Allows customisation of the group controller as a plugin.

The behaviour of the plugin is determined by 5 method hooks:

  • package_form(self)
  • form_to_db_schema(self)
  • db_to_form_schema(self)
  • check_data_dict(self, data_dict)
  • setup_template_variables(self, context, data_dict)

Furthermore, there can be many implementations of this plugin registered at once. With each instance associating itself with 0 or more package type strings. When a package controller action is invoked, the package type determines which of the registered plugins to delegate to. Each implementation must implement two methods which are used to determine the package-type -> plugin mapping:

  • is_fallback(self)
  • package_types(self)

Implementations might want to consider mixing in ckan.lib.plugins.DefaultGroupForm which provides default behaviours for the 5 method hooks.

check_data_dict(data_dict)

Check if the return data is correct.

raise a DataError if not.

db_to_form_schema()

Returns the schema for mapping group data from the database into a format suitable for the form (optional)

edit_template()

Returns a string representing the location of the template to be rendered for the edit page

form_to_db_schema()

Returns the schema for mapping group data from a form to a format suitable for the database.

group_types()

Returns an iterable of group type strings.

If a request involving a package of one of those types is made, then this plugin instance will be delegated to.

There must only be one plugin registered to each group type. Any attempts to register more than one plugin instance to a given group type will raise an exception at startup.

history_template()

Returns a string representing the location of the template to be rendered for the history page

index_template()

Returns a string representing the location of the template to be rendered for the index page. Uses the default_group_type configuration option to determine which plugin to use the template from.

is_fallback()

Returns true iff this provides the fallback behaviour, when no other plugin instance matches a package’s type.

There must be exactly one fallback controller defined, any attempt to register more than one will throw an exception at startup. If there’s no fallback registered at startup the ckan.lib.plugins.DefaultGroupForm used as the fallback.

new_template()

Returns a string representing the location of the template to be rendered for the ‘new’ page. Uses the default_group_type configuration option to determine which plugin to use the template from.

package_form()

Returns a string representing the location of the template to be rendered. e.g. “group/new_group_form.html”.

read_template()

Returns a string representing the location of the template to be rendered for the read page

setup_template_variables(context, data_dict)

Add variables to c just prior to the template being rendered.

class ckan.plugins.interfaces.ITagController

Hook into the Tag controller. These will usually be called just before committing or returning the respective object, i.e. all validation, synchronization and authorization setup are complete.

before_view(tag_dict)

Extensions will recieve this before the tag gets displayed. The dictionary passed will be the one that gets sent to the template.

class ckan.plugins.interfaces.ITemplateHelpers

Allow adding extra template functions available via h variable

get_helpers()

Should return a dict, the keys being the name of the helper function and the values being the functions themselves.