Writing Extensions

Note

A CKAN extension is a Python package that contains one or more plugins. A plugin is a class that implements one or more of CKAN’s plugin interfaces to customize CKAN or add new features.

Plugins: An Overview

Plugins are created as classes inheriting from either the Plugin or SingletonPlugin base classes. Most Extensions use the SingletonPlugin base class and we advise you to use this if possible.

Having created your class you need to inherit from one or more plugin interfaces to allow CKAN to interact with your extension. When specifying the interfaces that will be implemented you must remember to either (a) define all methods required by the interface or (b) use the inherits=True parameter which will use the interfaces default methods for any that you have not defined.

Note

When writing extensions it is important to keep your code separate from CKAN by not importing ckan modules, so that internal CKAN changes do not break your code between releases. You can however import ckan.plugins without this risk.

Libraries Available To Extensions

As well as using the variables made available to them by implementing various plugin hooks, extensions will likely want to be able to use parts of the CKAN core library. To allow this, CKAN provides a stable set of modules that extensions can use safe in the knowledge the interface will remain stable, backward-compatible and with clear deprecation guidelines as development of CKAN core progresses. This interface is available in Plugins Toolkit.

Example Extension

# Example Extension
# This extension adds a new template helper function `hello_world` when
# enabled templates can `{{ h.hello_world() }}` to add this html snippet.

import ckan.plugins as p

class HelloWorldPlugin(p.SingletonPlugin):

    p.implements(p.ITemplateHelpers)

    @staticmethod
    def hello_world():
        # This is our simple helper function.
        html = '<span>Hello World</span>'
        return p.toolkit.literal(html)

    def get_helpers(self):
        # This method is defined in the ITemplateHelpers interface and
        # is used to return a dict of named helper functions.
        return {'hello_world': hello_world}

Guidelines for writing extensions

  • Use the plugins Plugins Toolkit.
  • Extensions should use actions via get_action(). This function is available in the toolkit.
  • No foreign key constraints into core as these cause problems.

Creating CKAN Extensions

All CKAN extensions must start with the name ckanext-. You can create your own CKAN extension like this (you must be in your CKAN pyenv):

(pyenv)$ paster create -t ckanext ckanext-myextension

You’ll get prompted to complete a number of variables which will be used in your dataset. You change these later by editing the generated setup.py file.

Once you’ve run this, you should now install the extension in your virtual environment:

(pyenv)$ cd ckanext-myextension
(pyenv)$ python setup.py develop

Note

Running python setup.py develop will add a .egg-link file to your python site-packages directory (which is on your python path). This allows your extension to be imported and used, with any changes made to the extension source code showing up immediately without needing to be reinstalled, which is very useful during development.

To instead install a python package by copying all of the files to the site-packages directory run python setup.py install.

Testing

Testing CKAN Extensions

CKAN extensions ordinarily have their own test.ini that refers to the CKAN test.ini, so you can run them in exactly the same way. For example:

cd ckanext-dgu
nosetests ckanext/stats/tests --ckan
nosetests ckanext/stats/tests --ckan --with-pylons=test-core.ini

Testing Plugins

When writing tests for your plugin code you will need setup and teardown code similar to the following to ensure that your plugin is loaded while testing:

from ckan import plugins

class TestMyPlugin(TestCase):

   @classmethod
   def setup_class(cls):
       # Use the entry point name of your plugin as declared
       # in your package's setup.py
       plugins.load('my_plugin')

   @classmethod
   def teardown_class(cls):
       plugins.reset()

The exception to using plugins.load() is for when your plug-in is for routes. In this case, the plugin must be configured before the WSGI app is started. Here is an example test set-up:

from pylons import config
import paste.fixture
from ckan.config.middleware import make_app

class TestMyRoutesPlugin(TestCase):

    @classmethod
    def setup_class(cls):
        cls._original_config = config.copy()
        config['ckan.plugins'] = 'my_routes_plugin'
        wsgiapp = make_app(config['global_conf'], **config.local_conf)
        cls.app = paste.fixture.TestApp(wsgiapp)

    @classmethod
    def teardown_class(cls):
        config.clear()
        config.update(cls._original_config)

At this point you should be able to write your own plugins and extensions together with their tests.

Plugin API Documentation

Core Plugin Reference

Provides plugin services to the CKAN

class ckan.plugins.core.Plugin(**kwargs)

Base class for plugins which require multiple instances.

Unless you need multiple instances of your plugin object you should probably use SingletonPlugin.

class ckan.plugins.core.SingletonPlugin(**kwargs)

Base class for plugins which are singletons (ie most of them)

One singleton instance of this class will be created when the plugin is loaded. Subsequent calls to the class constructor will always return the same singleton instance.

ckan.plugins.core.implements(interface, namespace=None, inherit=False, service=True)

Can be used in the class definition of Plugin subclasses to declare the extension points that are implemented by this interface class.

If the inherits option is True, then this Plugin class inherits from the interface class.

CKAN Interface Reference

Interfaces for plugins system

class ckan.plugins.interfaces.IMiddleware

Hook into Pylons middleware stack

make_middleware(app, config)

Return an app configured with this middleware

class ckan.plugins.interfaces.IGenshiStreamFilter

Hook into template rendering. See ckan.lib.base.py:render

filter(stream)

Return a filtered Genshi stream. Called when any page is rendered.

Parameters:stream – Genshi stream of the current output document
Returns:filtered Genshi stream
class ckan.plugins.interfaces.IRoutes

Plugin into the setup of the routes map creation.

before_map(map)

Called before the routes map is generated. before_map is before any other mappings are created so can override all other mappings.

Parameters:map – Routes map object
Returns:Modified version of the map object
after_map(map)

Called after routes map is set up. after_map can be used to add fall-back handlers.

Parameters:map – Routes map object
Returns:Modified version of the map object
class ckan.plugins.interfaces.IMapper

A subset of the SQLAlchemy mapper extension hooks. See http://www.sqlalchemy.org/docs/05/reference/orm/interfaces.html#sqlalchemy.orm.interfaces.MapperExtension

Example:

>>> class MyPlugin(SingletonPlugin):
...
...     implements(IMapper)
...
...     def after_update(self, mapper, connection, instance):
...         log("Updated: %r", instance)
before_insert(mapper, connection, instance)

Receive an object instance before that instance is INSERTed into its table.

before_update(mapper, connection, instance)

Receive an object instance before that instance is UPDATEed.

before_delete(mapper, connection, instance)

Receive an object instance before that instance is DELETEed.

after_insert(mapper, connection, instance)

Receive an object instance after that instance is INSERTed.

after_update(mapper, connection, instance)

Receive an object instance after that instance is UPDATEed.

after_delete(mapper, connection, instance)

Receive an object instance after that instance is DELETEed.

class ckan.plugins.interfaces.ISession

A subset of the SQLAlchemy session extension hooks.

after_begin(session, transaction, connection)

Execute after a transaction is begun on a connection

before_flush(session, flush_context, instances)

Execute before flush process has started.

after_flush(session, flush_context)

Execute after flush has completed, but before commit has been called.

before_commit(session)

Execute right before commit is called.

after_commit(session)

Execute after a commit has occured.

after_rollback(session)

Execute after a rollback has occured.

class ckan.plugins.interfaces.IDomainObjectModification

Receives notification of new, changed and deleted datesets.

class ckan.plugins.interfaces.IResourceUrlChange

Receives notification of changed urls.

class ckan.plugins.interfaces.IResourcePreview

Hook into the resource previews in helpers.py. This lets you create custom previews for example for xml files.

can_preview(data_dict)

Return True if the extension can preview the resource. The data_dict contains the resource and the package.

Make sure to ckeck the on_same_domain value of the resource or the url if your preview requires the resource to be on the same domain because of the same origin policy.

setup_template_variables(context, data_dict)

Add variables to c just prior to the template being rendered. The data_dict contains the resource and the package.

Change the url to a proxied domain if necessary.

preview_template(context, data_dict)

Returns a string representing the location of the template to be rendered for the read page. The data_dict contains the resource and the package.

class ckan.plugins.interfaces.ITagController

Hook into the Tag controller. These will usually be called just before committing or returning the respective object, i.e. all validation, synchronization and authorization setup are complete.

before_view(tag_dict)

Extensions will recieve this before the tag gets displayed. The dictionary passed will be the one that gets sent to the template.

class ckan.plugins.interfaces.IGroupController

Hook into the Group controller. These will usually be called just before committing or returning the respective object, i.e. all validation, synchronization and authorization setup are complete.

before_view(pkg_dict)

Extensions will recieve this before the group gets displayed. The dictionary passed will be the one that gets sent to the template.

class ckan.plugins.interfaces.IOrganizationController

Hook into the Organization controller. These will usually be called just before committing or returning the respective object, i.e. all validation, synchronization and authorization setup are complete.

before_view(pkg_dict)

Extensions will recieve this before the organization gets displayed. The dictionary passed will be the one that gets sent to the template.

class ckan.plugins.interfaces.IPackageController

Hook into the package controller. (see IGroupController)

after_create(context, pkg_dict)

Extensions will receive the validated data dict after the package has been created (Note that the create method will return a package domain object, which may not include all fields). Also the newly created package id will be added to the dict.

after_update(context, pkg_dict)

Extensions will receive the validated data dict after the package has been updated (Note that the edit method will return a package domain object, which may not include all fields).

after_delete(context, pkg_dict)

Extensions will receive the data dict (tipically containing just the package id) after the package has been deleted.

after_show(context, pkg_dict)

Extensions will receive the validated data dict after the package is ready for display (Note that the read method will return a package domain object, which may not include all fields).

Extensions will receive a dictionary with the query parameters, and should return a modified (or not) version of it.

search_params will include an extras dictionary with all values from fields starting with ext_, so extensions can receive user input from specific fields.

Extensions will receive the search results, as well as the search parameters, and should return a modified (or not) object with the same structure:

{‘count’: ‘’, ‘results’: ‘’, ‘facets’: ‘’}

Note that count and facets may need to be adjusted if the extension changed the results for some reason.

search_params will include an extras dictionary with all values from fields starting with ext_, so extensions can receive user input from specific fields.

before_index(pkg_dict)

Extensions will receive what will be given to the solr for indexing. This is essentially a flattened dict (except for multli-valued fields such as tags) of all the terms sent to the indexer. The extension can modify this by returning an altered version.

before_view(pkg_dict)

Extensions will recieve this before the dataset gets displayed. The dictionary passed will be the one that gets sent to the template.

class ckan.plugins.interfaces.IPluginObserver

Plugin to the plugin loading mechanism

before_load(plugin)

Called before a plugin is loaded This method is passed the plugin class.

after_load(service)

Called after a plugin has been loaded. This method is passed the instantiated service object.

before_unload(plugin)

Called before a plugin is loaded This method is passed the plugin class.

after_unload(service)

Called after a plugin has been unloaded. This method is passed the instantiated service object.

class ckan.plugins.interfaces.IConfigurable

Pass configuration to plugins and extensions

configure(config)

Called by load_environment

class ckan.plugins.interfaces.IConfigurer

Configure CKAN (pylons) environment via the pylons.config object

update_config(config)

Called by load_environment at earliest point when config is available to plugins. The config should be updated in place.

Parameters:configpylons.config object
class ckan.plugins.interfaces.IActions

Allow adding of actions to the logic layer.

get_actions()

Should return a dict, the keys being the name of the logic function and the values being the functions themselves.

By decorating a function with the ckan.logic.side_effect_free decorator, the associated action will be made available by a GET request (as well as the usual POST request) through the action API.

class ckan.plugins.interfaces.IAuthFunctions

Allow customisation of default Authorization implementation

get_auth_functions()

Returns a dict of all the authorization functions which the implementation overrides

class ckan.plugins.interfaces.ITemplateHelpers

Add custom template helper functions.

By implementing this plugin interface plugins can provide their own template helper functions, which custom templates can then access via the h variable.

See ckanext/example_itemplatehelpers for an example plugin.

get_helpers()

Return a dict mapping names to helper functions.

The keys of the dict should be the names with which the helper functions will be made available to templates, and the values should be the functions themselves. For example, a dict like: {'example_helper': example_helper} allows templates to access the example_helper function via h.example_helper().

Function names should start with the name of the extension providing the function, to prevent name clashes between extensions.

class ckan.plugins.interfaces.IDatasetForm

Customize CKAN’s dataset (package) schemas and forms.

By implementing this interface plugins can customise CKAN’s dataset schema, for example to add new custom fields to datasets.

Multiple IDatasetForm plugins can be used at once, each plugin associating itself with different package types using the package_types() and is_fallback() methods below, and then providing different schemas and templates for different types of dataset. When a package controller action is invoked, the type field of the package will determine which IDatasetForm plugin (if any) gets delegated to.

When implementing IDatasetForm, you can inherit from ckan.plugins.toolkit.DefaultDatasetForm, which provides default implementations for each of the methods defined in this interface.

See ckanext/example_idatasetform for an example plugin.

package_types()

Return an iterable of package types that this plugin handles.

If a request involving a package of one of the returned types is made, then this plugin instance will be delegated to.

There cannot be two IDatasetForm plugins that return the same package type, if this happens then CKAN will raise an exception at startup.

Return type:iterable of strings
is_fallback()

Return True if this plugin is the fallback plugin.

When no IDatasetForm plugin’s package_types() match the type of the package being processed, the fallback plugin is delegated to instead.

There cannot be more than one IDatasetForm plugin whose is_fallback() method returns True, if this happens CKAN will raise an exception at startup.

If no IDatasetForm plugin’s is_fallback() method returns True, CKAN will use DefaultDatasetForm as the fallback.

Return type:boolean
create_package_schema()

Return the schema for validating new dataset dicts.

CKAN will use the returned schema to validate and convert data coming from users (via the dataset form or API) when creating new datasets, before entering that data into the database.

If it inherits from ckan.plugins.toolkit.DefaultDatasetForm, a plugin can call DefaultDatasetForm‘s create_package_schema() method to get the default schema and then modify and return it.

CKAN’s convert_to_tags() or convert_to_extras() functions can be used to convert custom fields into dataset tags or extras for storing in the database.

See ckanext/example_idatasetform for examples.

Returns:a dictionary mapping dataset dict keys to lists of validator and converter functions to be applied to those keys
Return type:dictionary
update_package_schema()

Return the schema for validating updated dataset dicts.

CKAN will use the returned schema to validate and convert data coming from users (via the dataset form or API) when updating datasets, before entering that data into the database.

If it inherits from ckan.plugins.toolkit.DefaultDatasetForm, a plugin can call DefaultDatasetForm‘s update_package_schema() method to get the default schema and then modify and return it.

CKAN’s convert_to_tags() or convert_to_extras() functions can be used to convert custom fields into dataset tags or extras for storing in the database.

See ckanext/example_idatasetform for examples.

Returns:a dictionary mapping dataset dict keys to lists of validator and converter functions to be applied to those keys
Return type:dictionary
show_package_schema()

Return a schema to validate datasets before they’re shown to the user.

CKAN will use the returned schema to validate and convert data coming from the database before it is returned to the user via the API or passed to a template for rendering.

If it inherits from ckan.plugins.toolkit.DefaultDatasetForm, a plugin can call DefaultDatasetForm‘s show_package_schema() method to get the default schema and then modify and return it.

If you have used convert_to_tags() or convert_to_extras() in your create_package_schema() and update_package_schema() then you should use convert_from_tags() or convert_from_extras() in your show_package_schema() to convert the tags or extras in the database back into your custom dataset fields.

See ckanext/example_idatasetform for examples.

Returns:a dictionary mapping dataset dict keys to lists of validator and converter functions to be applied to those keys
Return type:dictionary
setup_template_variables(context, data_dict)

Add variables to the template context for use in templates.

This function is called before a dataset template is rendered. If you have custom dataset templates that require some additional variables, you can add them to the template context ckan.plugins.toolkit.c here and they will be available in your templates. See ckanext/example_idatasetform for an example.

new_template()

Return the path to the template for the new dataset page.

The path should be relative to the plugin’s templates dir, e.g. 'package/new.html'.

Return type:string
read_template()

Return the path to the template for the dataset read page.

The path should be relative to the plugin’s templates dir, e.g. 'package/read.html'.

Return type:string
edit_template()

Return the path to the template for the dataset edit page.

The path should be relative to the plugin’s templates dir, e.g. 'package/edit.html'.

Return type:string
comments_template()

Return the path to the template for the dataset comments page.

The path should be relative to the plugin’s templates dir, e.g. 'package/comments.html'.

Return type:string
search_template()

Return the path to the template for use in the dataset search page.

This template is used to render each dataset that is listed in the search results on the dataset search page.

The path should be relative to the plugin’s templates dir, e.g. 'package/search.html'.

Return type:string
history_template()

Return the path to the template for the dataset history page.

The path should be relative to the plugin’s templates dir, e.g. 'package/history.html'.

Return type:string
package_form()

Return the path to the template for the dataset form.

The path should be relative to the plugin’s templates dir, e.g. 'package/form.html'.

Return type:string
class ckan.plugins.interfaces.IGroupForm

Allows customisation of the group controller as a plugin.

The behaviour of the plugin is determined by 5 method hooks:

  • package_form(self)
  • form_to_db_schema(self)
  • db_to_form_schema(self)
  • check_data_dict(self, data_dict)
  • setup_template_variables(self, context, data_dict)

Furthermore, there can be many implementations of this plugin registered at once. With each instance associating itself with 0 or more package type strings. When a package controller action is invoked, the package type determines which of the registered plugins to delegate to. Each implementation must implement two methods which are used to determine the package-type -> plugin mapping:

  • is_fallback(self)
  • package_types(self)

Implementations might want to consider mixing in ckan.lib.plugins.DefaultGroupForm which provides default behaviours for the 5 method hooks.

is_fallback()

Returns true iff this provides the fallback behaviour, when no other plugin instance matches a package’s type.

There must be exactly one fallback controller defined, any attempt to register more than one will throw an exception at startup. If there’s no fallback registered at startup the ckan.lib.plugins.DefaultGroupForm used as the fallback.

group_types()

Returns an iterable of group type strings.

If a request involving a package of one of those types is made, then this plugin instance will be delegated to.

There must only be one plugin registered to each group type. Any attempts to register more than one plugin instance to a given group type will raise an exception at startup.

new_template()

Returns a string representing the location of the template to be rendered for the ‘new’ page. Uses the default_group_type configuration option to determine which plugin to use the template from.

index_template()

Returns a string representing the location of the template to be rendered for the index page. Uses the default_group_type configuration option to determine which plugin to use the template from.

read_template()

Returns a string representing the location of the template to be rendered for the read page

history_template()

Returns a string representing the location of the template to be rendered for the history page

edit_template()

Returns a string representing the location of the template to be rendered for the edit page

package_form()

Returns a string representing the location of the template to be rendered. e.g. “group/new_group_form.html”.

form_to_db_schema()

Returns the schema for mapping group data from a form to a format suitable for the database.

db_to_form_schema()

Returns the schema for mapping group data from the database into a format suitable for the form (optional)

check_data_dict(data_dict)

Check if the return data is correct.

raise a DataError if not.

setup_template_variables(context, data_dict)

Add variables to c just prior to the template being rendered.

class ckan.plugins.interfaces.IFacets

Allows specify which facets are displayed and also the names used.

facet_dicts are in the form {‘facet_name’: ‘dispaly name’, ...} to allow translatable dispaly names use _(...) eg {‘facet_name’: _(‘dispaly name’), ...} and ensure that this is created each time the function is called.

The dict supplied is actually an ordered dict.

dataset_facets(facets_dict, package_type)

Update the facets_dict and return it.

group_facets(facets_dict, group_type, package_type)

Update the facets_dict and return it.

organization_facets(facets_dict, organization_type, package_type)

Update the facets_dict and return it.