Testing coding standards

All new code, or changes to existing code, should have new or updated tests before being merged into master. This document gives some guidelines for developers who are writing tests or reviewing code for CKAN.

Guidelines for writing tests

We want the tests in ckan.tests to be:

Fast

Don’t share setup code between tests (e.g. in test class setup() or setup_class() methods, saved against the self attribute of test classes, or in test helper modules).

Instead use fixtures that create test objects and pass them as parameters, and inject into every method only the required fixtures.
Where appropriate, use the monkeypatch fixture to avoid pulling in other parts of CKAN (especially the database).

Independent

Each test module, class and method should be able to be run on its own.
Tests shouldn’t be tightly coupled to each other, changing a test shouldn’t affect other tests.

Clear

It should be quick and easy to see what went wrong when a test fails, or to see what a test does and how it works if you have to debug or update a test. If you think the test or helper method isn’t clear by itself, add docstrings.

You shouldn’t have to figure out what a complex test method does, or go and look up a lot of code in other files to understand a test method.

Tests should follow the canonical form for a pytest, see Recipe for a test method.
Write lots of small, simple test methods not a few big, complex tests.
Each test method should test just One Thing.
The name of a test method should clearly explain the intent of the test. See Naming test methods.

Easy to find

It should be easy to know where to add new tests for some new or changed code, or to find the existing tests for some code.

See How should tests be organized?
See Naming test methods.

Easy to write

Writing lots of small, clear and simple tests that all follow similar recipes and organization should make tests easy to write, as well as easy to read.

The follow sections give some more specific guidelines and tips for writing CKAN tests.

How should tests be organized?

The organization of test modules in ckan.tests mirrors the organization of the source modules in ckan:

ckan/
  tests/
    controllers/
      test_package.py <-- Tests for ckan/controllers/package.py
      ...
    lib/
      test_helpers.py <-- Tests for ckan/lib/helpers.py
      ...
    logic/
      action/
        test_get.py
        ...
      auth/
        test_get.py
        ...
      test_converters.py
      test_validators.py
    migration/
      versions/
        test_001_add_existing_tables.py
        ...
    model/
      test_package.py
      ...
    ...

There are a few exceptional test modules that don’t fit into this structure, for example PEP8 tests and coding standards tests. These modules can just go in the top-level ckan/tests/ directory. There shouldn’t be too many of these.

Naming test methods

The name of a test method should clearly explain the intent of the test.

Test method names are printed out when tests fail, so the user can often see what went wrong without having to look into the test file. When they do need to look into the file to debug or update a test, the test name helps to clarify the test.

Do this even if it means your method name gets really long, since we don’t write code that calls our test methods there’s no advantage to having short test method names.

Some modules in CKAN contain large numbers of loosely related functions. For example, ckan.logic.action.update contains all functions for updating things in CKAN. This means that ckan.tests.logic.action.test_update is going to contain an even larger number of test functions.

So as well as the name of each test method explaining the intent of the test, tests should be grouped by a test class that aggregates tests against a model entity or action type, for instance:

class TestPackageCreate(object):
    # ...
    def test_it_validates_name(self):
        # ...

    def test_it_validates_url(self):
        # ...


class TestResourceCreate(object)
    # ...
    def test_it_validates_package_id(self):
        # ...

# ...

Good test names:

TestUserUpdate.test_update_with_id_that_does_not_exist
TestUserUpdate.test_update_with_no_id
TestUserUpdate.test_update_with_invalid_name

Bad test names:

test_user_update
test_update_pkg_1
test_package

Recipe for a test method

The Pylons Unit Testing Guidelines give the following recipe for all unit test methods to follow:

Set up the preconditions for the method / function being tested.
Call the method / function exactly one time, passing in the values established in the first step.
Make assertions about the return value, and / or any side effects.
Do absolutely nothing else.

Most CKAN tests should follow this form. Here’s an example of a simple action function test demonstrating the recipe:

    def test_user_update_name(self):
        """Test that updating a user's name works successfully."""

        # The canonical form of a test has four steps:
        # 1. Setup any preconditions needed for the test.
        # 2. Call the function that's being tested, once only.
        # 3. Make assertions about the return value and/or side-effects of
        #    of the function that's being tested.
        # 4. Do nothing else!

        # 1. Setup.
        user = factories.User()
        user["name"] = "updated"

        # 2. Make assertions about the return value and/or side-effects.
        with pytest.raises(logic.ValidationError):
            helpers.call_action("user_update", **user)

How detailed should tests be?

Generally, what we’re trying to do is test the interfaces between modules in a way that supports modularization: if you change the code within a function, method, class or module, if you don’t break any of that code’s tests you should be able to expect that CKAN as a whole will not be broken.

As a general guideline, the tests for a function or method should:

Test for success:
- Test the function with typical, valid input values
- Test with valid, edge-case inputs
- If the function has multiple parameters, test them in different combinations
Test for failure:
- Test that the function fails correctly (e.g. raises the expected type of exception) when given likely invalid inputs (for example, if the user passes an invalid user_id as a parameter)
- Test that the function fails correctly when given bizarre input
Test that the function behaves correctly when given unicode characters as input
Cover the interface of the function: test all the parameters and features of the function

Creating test objects: `ckan.tests.factories`

This is a collection of factory classes for building CKAN users, datasets, etc.

Factories can be either used directly or via corresponding pytest fixtures to create any objects that are needed for the tests. These factories are written using factory_boy:

https://factoryboy.readthedocs.org/en/latest/

These are not meant to be used for the actual testing, e.g. if you’re writing a test for the user_create() function then call call_action(), don’t test it via the User factory or user_factory() fixture.

Usage:

# Create a user with the factory's default attributes, and get back a
# user dict:
def test_creation():
    user_dict = factories.User()

# or

def test_creation(user_factory):
    user_dict = user_factory()

# You can create a second user the same way. For attributes that can't be
# the same (e.g. you can't have two users with the same name) a new value
# will be generated each time you use the factory:
def test_creation():
    user_dict = factories.User()
    another_user_dict = factories.User()

# Create a user and specify your own user name and email (this works
# with any params that CKAN's user_create() accepts):
def test_creation():
    custom_user_dict = factories.User(name='bob', email='bob@bob.com')

# Get a user dict containing the attributes (name, email, password, etc.)
# that the factory would use to create a user, but without actually
# creating the user in CKAN:
def test_creation():
    user_attributes_dict = vars(factories.User.stub())

# If you later want to create a user using these attributes, just pass them
# to the factory:
def test_creation():
    user = factories.User(**user_attributes_dict)

# If you just need random user, you can get ready-to-use dictionary inside
# your test by requiring `user` fixture (just drop `_factory` suffix):
def test_creation(user):
    assert isinstance(user, dict)
    assert "name" in user

# If you need SQLAlchemy model object instead of the plain dictionary, call
# `model` method of the corresponding factory. All arguments has the same
# effect as if they were passed directly to the factory:
def test_creation():
    user = factories.User.model(name="bob")
    assert isinstance(user, model.User)


# In order to create your own factory:
# * inherit from :py:class:`~ckan.tests.factories.CKANFactory`
# * create `Meta` class inside it, with the two properties:
#   * model: corresponding SQLAlchemy model
#   * action: API action that can create instances of the model
# * define any extra attributes
# * register factory as a fixture using :py:func:`~pytest_factoryboy.register`
import factory
from pytest_factoryboy import register
from ckan.tests.factories import CKANFactory

@register
class RatingFactory(CKANFactory):

    class Meta:
        model = ckanext.ext.model.Rating
        action = "rating_create"

    # These are the default params that will be used to create new ratings
    value = factory.Faker("pyint")
    comment = factory.Faker("text")
    approved = factory.Faker("boolean")

Factory-fixtures are generated using pytest-factoryboy:

https://pytest-factoryboy.readthedocs.io/en/latest/

class ckan.tests.factories.CKANOptions

CKANFactory options.

Parameters:

action – name of the CKAN API action used for entity creation
primary_key – name of the entity’s property that can be used for retrieving entity object from database

class ckan.tests.factories.CKANFactory(**kwargs)

Extension of SQLAlchemy factory.

Creates entities via CKAN API using an action specified by the Meta.action.

Provides model method that returns created model object instead of the plain dictionary.

Check factoryboy’s documentation for more details: https://factoryboy.readthedocs.io/en/stable/orms.html#sqlalchemy

classmethod api_create(data_dict): Create entity via API call.

classmethod model(**kwargs): Create entity via API and retrieve result directly from the DB.

class ckan.tests.factories.User(**kwargs): A factory class for creating CKAN users.

class ckan.tests.factories.Resource(**kwargs): A factory class for creating CKAN resources.

class ckan.tests.factories.ResourceView(**kwargs)

A factory class for creating CKAN resource views.

Note: if you use this factory, you need to load the image_view plugin on your test class (and unload it later), otherwise you will get an error.

Example:

@pytest.mark.ckan_config("ckan.plugins", "image_view")
@pytest.mark.usefixtures("with_plugins")
def test_resource_view_factory():
    ...

class ckan.tests.factories.Sysadmin(**kwargs): A factory class for creating sysadmin users.

class ckan.tests.factories.Group(**kwargs): A factory class for creating CKAN groups.

class ckan.tests.factories.Organization(**kwargs): A factory class for creating CKAN organizations.

class ckan.tests.factories.Dataset(**kwargs): A factory class for creating CKAN datasets.

class ckan.tests.factories.Vocabulary(**kwargs): A factory class for creating tag vocabularies.

class ckan.tests.factories.Tag(**kwargs): A factory class for creating tag vocabularies.

class ckan.tests.factories.MockUser(**kwargs): A factory class for creating mock CKAN users using the mock library.

class ckan.tests.factories.SystemInfo(**kwargs): A factory class for creating SystemInfo objects (config objects stored in the DB).

class ckan.tests.factories.APIToken(**kwargs): A factory class for creating CKAN API Tokens

class ckan.tests.factories.UserWithToken(**kwargs): A factory class for creating CKAN users with an associated API token.

class ckan.tests.factories.SysadminWithToken(**kwargs): A factory class for creating CKAN sysadmin users with an associated API token.

Test helper functions: `ckan.tests.helpers`

This is a collection of helper functions for use in tests.

We want to avoid sharing test helper functions between test modules as much as possible, and we definitely don’t want to introduce a complex hierarchy of test class subclasses, etc.

We want to reduce the amount of “travel” that a reader needs to undertake to understand a test method – reducing the number of other files they need to go and read to understand what the test code does. And we want to avoid tightly coupling test modules to each other by having them share code.

But some test helper functions just increase the readability of tests so much and make writing tests so much easier, that it’s worth having them despite the potential drawbacks.

New in CKAN 2.9: Consider using Pytest fixtures whenever possible for setting up the initial state of a test or to create helpers objects like client apps.

ckan.tests.helpers.reset_db()

Reset CKAN’s database.

Rather than use this function directly, use the clean_db fixture either for all tests in a class:

@pytest.mark.usefixtures("clean_db")
class TestExample(object):

    def test_example(self):

or for a single test:

class TestExample(object):

    @pytest.mark.usefixtures("clean_db")
    def test_example(self):

If a test class uses the database, then it may call this function in its setup() method to make sure that it has a clean database to start with (nothing left over from other test classes or from previous test runs).

If a test class doesn’t use the database (and most test classes shouldn’t need to) then it doesn’t need to call this function.

Returns:: None

ckan.tests.helpers.call_action(action_name: str, context=None, **kwargs)

Call the named ckan.logic.action function and return the result.

This is just a nicer way for user code to call action functions, nicer than either calling the action function directly or via ckan.logic.get_action().

For example:

user_dict = call_action('user_create', name='seanh',
                        email='seanh@seanh.com', password='pass')

Any keyword arguments given will be wrapped in a dict and passed to the action function as its data_dict argument.

Note: this skips authorization! It passes ‘ignore_auth’: True to action functions in their context dicts, so the corresponding authorization functions will not be run. This is because ckan.tests.logic.action tests only the actions, the authorization functions are tested separately in ckan.tests.logic.auth. See the testing guidelines for more info.

This function should eventually be moved to ckan.logic.call_action() and the current ckan.logic.get_action() function should be deprecated. The tests may still need their own wrapper function for ckan.logic.call_action(), e.g. to insert 'ignore_auth': True into the context dict.

Parameters:

action_name (string) – the name of the action function to call, e.g. 'user_update'
context (dict) – the context dict to pass to the action function (optional, if no context is given a default one will be supplied)

Returns:

the dict or other value that the action function returns

ckan.tests.helpers.call_auth(auth_name: str, context, **kwargs) → bool

Call the named ckan.logic.auth function and return the result.

This is just a convenience function for tests in ckan.tests.logic.auth to use.

Usage:

result = helpers.call_auth('user_update', context=context,
                           id='some_user_id',
                           name='updated_user_name')

Parameters:

auth_name (string) – the name of the auth function to call, e.g. 'user_update'
context (dict) – the context dict to pass to the auth function, must contain 'user' and 'model' keys, e.g. {'user': 'fred', 'model': my_mock_model_object}

Returns:

the ‘success’ value of the authorization check, e.g. {'success': True} or {'success': False, msg: 'important error message'} or just {'success': False}

Return type:

bool

class ckan.tests.helpers.CKANCliRunner(charset: str = 'utf-8', env: Mapping[str, str | None] | None = None, echo_stdin: bool = False, mix_stderr: bool = True)

invoke(*args, **kwargs)

Invokes a command in an isolated environment. The arguments are forwarded directly to the command line script, the extra keyword arguments are passed to the main() function of the command.

This returns a Result object.

Parameters:

cli – the command to invoke
args – the arguments to invoke. It may be given as an iterable or a string. When given as string it will be interpreted as a Unix shell command. More details at shlex.split().
input – the input data for sys.stdin.
env – the environment overrides.
catch_exceptions – Whether to catch any other exceptions than SystemExit.
extra – the keyword arguments to pass to main().
color – whether the output should contain color codes. The application can still override this explicitly.

Changed in version 8.0: The result object has the return_value attribute with the value returned from the invoked command.

Changed in version 4.0: Added the color parameter.

Changed in version 3.0: Added the catch_exceptions parameter.

Changed in version 3.0: The result object has the exc_info attribute with the traceback if available.

class ckan.tests.helpers.CKANResponse(response: Iterable[bytes] | bytes | Iterable[str] | str | None = None, status: int | str | HTTPStatus | None = None, headers: Mapping[str, str | Iterable[str]] | Iterable[tuple[str, str]] | None = None, mimetype: str | None = None, content_type: str | None = None, direct_passthrough: bool = False)

class ckan.tests.helpers.CKANTestApp(app)

A wrapper around flask.testing.Client

It adds some convenience methods for CKAN

class ckan.tests.helpers.CKANTestClient(application: WSGIApplication, response_wrapper: type[Response] | None = None, use_cookies: bool = True, allow_subdomain_redirects: bool = False)

open(*args, **kwargs)

Generate an environ dict from the given arguments, make a request to the application using it, and return the response.

Parameters:

args – Passed to EnvironBuilder to create the environ for the request. If a single arg is passed, it can be an existing EnvironBuilder or an environ dict.
buffered – Convert the iterator returned by the app into a list. If the iterator has a close() method, it is called automatically.
follow_redirects – Make additional requests to follow HTTP redirects until a non-redirect status is returned. TestResponse.history lists the intermediate responses.

Changed in version 2.1: Removed the as_tuple parameter.

Changed in version 2.0: The request input stream is closed when calling response.close(). Input streams for redirects are automatically closed.

Changed in version 0.5: If a dict is provided as file in the dict for the data parameter the content type has to be called content_type instead of mimetype. This change was made for consistency with werkzeug.FileWrapper.

Changed in version 0.5: Added the follow_redirects parameter.

class ckan.tests.helpers.FunctionalTestBase

A base class for functional test classes to inherit from.

Deprecated: Use the app, clean_db, ckan_config and with_plugins ref:fixtures as needed to create functional test classes, eg:

@pytest.mark.ckan_config('ckan.plugins', 'image_view')
@pytest.mark.usefixtures('with_plugins')
@pytest.mark.usefixtures('clean_db')
class TestDatasetSearch(object):

    def test_dataset_search(self, app):

        url = h.url_for('dataset.search')
        response = app.get(url)

Allows configuration changes by overriding _apply_config_changes and resetting the CKAN config after your test class has run. It creates a CKANTestApp at self.app for your class to use to make HTTP requests to the CKAN web UI or API. Also loads plugins defined by _load_plugins in the class definition.

If you’re overriding methods that this class provides, like setup_class() and teardown_class(), make sure to use super() to call this class’s methods at the top of yours!

setup(): Reset the database and clear the search indexes.

class ckan.tests.helpers.RQTestBase

Base class for tests of RQ functionality.

all_jobs(): Get a list of all RQ jobs.

enqueue(job=None, *args, **kwargs): Enqueue a test job.

class ckan.tests.helpers.FunctionalRQTestBase: Base class for functional tests of RQ functionality.

ckan.tests.helpers.change_config(key, value)

Decorator to temporarily change CKAN’s config to a new value

This allows you to easily create tests that need specific config values to be set, making sure it’ll be reverted to what it was originally, after your test is run.

Usage:

@helpers.change_config('ckan.site_title', 'My Test CKAN')
def test_ckan_site_title(self):
    assert config['ckan.site_title'] == 'My Test CKAN'

Parameters:

key (string) – the config key to be changed, e.g. 'ckan.site_title'
value (string) – the new config key’s value, e.g. 'My Test CKAN'

Pytest fixtures

This is a collection of pytest fixtures for use in tests.

All fixtures below available anywhere under the root of CKAN repository. Any external CKAN extension should be able to include them by adding next lines under root conftest.py

# -*- coding: utf-8 -*-

pytest_plugins = [
    u'ckan.tests.pytest_ckan.ckan_setup',
    u'ckan.tests.pytest_ckan.fixtures',
]

There are three type of fixtures available in CKAN:

Fixtures that have some side-effect. They don’t return any useful value and generally should be injected via pytest.mark.usefixtures. Ex.: with_plugins, clean_db, clean_index.
Fixtures that provide value. Ex. app
Fixtures that provide factory function. They are rarely needed, so prefer using ‘side-effect’ or ‘value’ fixtures. Main use-case when one may use function-fixture - late initialization or repeatable execution(ex.: cleaning database more than once in a single test). But presence of these fixtures in test usually signals that is’s a good time to refactor this test.

Deeper explanation can be found in official documentation

class ckan.tests.pytest_ckan.fixtures.UserFactory(**kwargs)

class ckan.tests.pytest_ckan.fixtures.ResourceFactory(**kwargs)

class ckan.tests.pytest_ckan.fixtures.ResourceViewFactory(**kwargs)

class ckan.tests.pytest_ckan.fixtures.GroupFactory(**kwargs)

class ckan.tests.pytest_ckan.fixtures.PackageFactory(**kwargs)

class ckan.tests.pytest_ckan.fixtures.VocabularyFactory(**kwargs)

class ckan.tests.pytest_ckan.fixtures.TagFactory(**kwargs)

class ckan.tests.pytest_ckan.fixtures.SystemInfoFactory(**kwargs)

class ckan.tests.pytest_ckan.fixtures.APITokenFactory(**kwargs)

class ckan.tests.pytest_ckan.fixtures.SysadminFactory(**kwargs)

class ckan.tests.pytest_ckan.fixtures.OrganizationFactory(**kwargs)

ckan.tests.pytest_ckan.fixtures.ckan_config(request, monkeypatch)

Allows to override the configuration object used by tests

Takes into account config patches introduced by the ckan_config mark.

If you just want to set one or more configuration options for the scope of a test (or a test class), use the ckan_config mark:

@pytest.mark.ckan_config('ckan.auth.create_unowned_dataset', True)
def test_auth_create_unowned_dataset():

    # ...

To use the custom config inside a test, apply the ckan_config mark to it and inject the ckan_config fixture:

@pytest.mark.ckan_config(u"some.new.config", u"exists")
def test_ckan_config_mark(ckan_config):
    assert ckan_config[u"some.new.config"] == u"exists"

If the change only needs to be applied locally, use the monkeypatch fixture

@pytest.mark.usefixtures("with_request_context")
def test_deleting_a_key_delets_it_on_flask_config(monkeypatch, ckan_config):
    monkeypatch.setitem(ckan_config, u"ckan.site_title", u"Example title")
    del ckan_config[u"ckan.site_title"]
    assert u"ckan.site_title" not in flask.current_app.config

ckan.tests.pytest_ckan.fixtures.make_app(ckan_config)

Factory for client app instances.

Unless you need to create app instances lazily for some reason, use the app fixture instead.

ckan.tests.pytest_ckan.fixtures.app(make_app)

Returns a client app instance to use in functional tests

To use it, just add the app parameter to your test function signature:

def test_dataset_search(self, app):

    url = h.url_for('dataset.search')

    response = app.get(url)

ckan.tests.pytest_ckan.fixtures.cli(ckan_config)

Provides object for invoking CLI commands from tests.

This is subclass of click.testing.CliRunner, so all examples from Click docs are valid for it.

ckan.tests.pytest_ckan.fixtures.reset_db()

Callable for resetting the database to the initial state.

If possible use the clean_db fixture instead.

ckan.tests.pytest_ckan.fixtures.reset_index()

Callable for cleaning search index.

If possible use the clean_index fixture instead.

ckan.tests.pytest_ckan.fixtures.reset_queues()

Callable for emptying and deleting the queues.

If possible use the clean_queues fixture instead.

ckan.tests.pytest_ckan.fixtures.reset_redis()

Callable for removing all keys from Redis.

Accepts redis key-pattern for narrowing down the list of items to remove. By default removes everything.

This fixture removes all the records from Redis on call:

def test_redis_is_empty(reset_redis):
    redis = connect_to_redis()
    redis.set("test", "test")

    reset_redis()
    assert not redis.get("test")

If only specific records require removal, pass a pattern to the fixture:

def test_redis_is_empty(reset_redis):
    redis = connect_to_redis()
    redis.set("AAA-1", 1)
    redis.set("AAA-2", 2)
    redis.set("BBB-3", 3)

    reset_redis("AAA-*")
    assert not redis.get("AAA-1")
    assert not redis.get("AAA-2")

    assert redis.get("BBB-3") is not None

ckan.tests.pytest_ckan.fixtures.clean_redis(reset_redis)

Remove all keys from Redis.

This fixture removes all the records from Redis:

@pytest.mark.usefixtures("clean_redis")
def test_redis_is_empty():
    assert redis.keys("*") == []

If test requires presence of some initial data in redis, make sure that data producer applied after clean_redis:

@pytest.mark.usefixtures(
    "clean_redis",
    "fixture_that_adds_xxx_key_to_redis"
)
def test_redis_has_one_record():
    assert redis.keys("*") == [b"xxx"]

ckan.tests.pytest_ckan.fixtures.clean_db(reset_db)

Resets the database to the initial state.

This can be used either for all tests in a class:

@pytest.mark.usefixtures("clean_db")
class TestExample(object):

    def test_example(self):

or for a single test:

class TestExample(object):

    @pytest.mark.usefixtures("clean_db")
    def test_example(self):

ckan.tests.pytest_ckan.fixtures.clean_queues(reset_queues)

Empties and deleted all queues.

This can be used either for all tests in a class:

@pytest.mark.usefixtures("clean_queues")
class TestExample(object):

    def test_example(self):

or for a single test:

class TestExample(object):

    @pytest.mark.usefixtures("clean_queues")
    def test_example(self):

ckan.tests.pytest_ckan.fixtures.migrate_db_for()

Apply database migration defined by plugin.

In order to use models defined by extension extra tables may be required. In such cases database migrations(that were generated by ckan generate migration -p PLUGIN_NAME) can be applied as per example below:

@pytest.mark.usefixtures("clean_db")
def test_migrations_applied(migrate_db_for):
    migrate_db_for("my_plugin")
    assert model.Session.bind.has_table("my_plugin_custom_table")

ckan.tests.pytest_ckan.fixtures.clean_index(reset_index): Clear search index before starting the test.

ckan.tests.pytest_ckan.fixtures.with_plugins(ckan_config)

Load all plugins specified by the ckan.plugins config option at the beginning of the test(and disable any plugin which is not listed inside ckan.plugins). When the test ends (including fail), it will unload all the plugins.

@pytest.mark.ckan_config("ckan.plugins", "image_view")
@pytest.mark.usefixtures("non_clean_db", "with_plugins")
def test_resource_view_factory():
    resource_view1 = factories.ResourceView()
    resource_view2 = factories.ResourceView()
    assert resource_view1[u"id"] != resource_view2[u"id"]

Use this fixture if test relies on CKAN plugin infrastructure. For example, if test calls an action or helper registered by plugin XXX:

@pytest.mark.ckan_config("ckan.plugins", "XXX")
@pytest.mark.usefixtures("with_plugin")
def test_action_and_helper():
    assert call_action("xxx_action")
    assert tk.h.xxx_helper()

It will not work without with_plugins. If XXX plugin is not loaded, xxx_action and xxx_helper do not exist in CKAN registries.

But if the test above use direct imports instead, with_plugins is optional:

def test_action_and_helper():
    from ckanext.xxx.logic.action import xxx_action
    from ckanext.xxx.helpers import xxx_helper

    assert xxx_action()
    assert xxx_helper()

Keep in mind, that generally it’s a bad idea to import helpers and actions directly. If every test of extension requires standard set of plugins, specify these plugins inside test config file(test.ini):

ckan.plugins = essential_plugin another_plugin_required_by_every_test

And create an autouse-fixture that depends on with_plugins inside the main conftest.py (ckanext/ext/tests/conftest.py):

@pytest.fixture(autouse=True)
def load_standard_plugins(with_plugins):
    ...

This will automatically enable with_plugins for every test, even if it’s not required explicitely.

ckan.tests.pytest_ckan.fixtures.test_request_context(app): Provide function for creating Flask request context.

ckan.tests.pytest_ckan.fixtures.with_request_context(test_request_context): Execute test inside requests context

ckan.tests.pytest_ckan.fixtures.mail_server(monkeypatch): Catch all outcome mails.

ckan.tests.pytest_ckan.fixtures.with_test_worker(monkeypatch): Worker that doesn’t create forks.

ckan.tests.pytest_ckan.fixtures.with_extended_cli(ckan_config, monkeypatch)

Enables effects of IClick.

Without this fixture, only CLI command that came from plugins specified in real config file are available. When this fixture enabled, changing ckan.plugins on test level allows to update list of available CLI command.

ckan.tests.pytest_ckan.fixtures.reset_db_once(reset_db): Internal fixture that cleans DB only the first time it’s used.

ckan.tests.pytest_ckan.fixtures.non_clean_db(reset_db_once)

Guarantees that DB is initialized.

This fixture either initializes DB if it hasn’t been done yet or does nothing otherwise. If there is some data in DB, it stays intact. If your tests need empty database, use clean_db instead, which is much slower, but guarantees that there are no data left from the previous test session.

Example:

@pytest.mark.usefixtures("non_clean_db")
def test_example():
    assert factories.User()

class ckan.tests.pytest_ckan.fixtures.FakeFileStorage(stream: IO[bytes], filename: str)

ckan.tests.pytest_ckan.fixtures.create_with_upload(clean_db, ckan_config, monkeypatch, tmpdir)

Shortcut for creating resource/user/org with upload.

Requires content and name for newly created object. By default is using resource_create action, but it can be changed by passing named argument action.

Upload field if configured by passing upload_field_name named argument. Default value: upload.

In addition, accepts named argument context which will be passed to ckan.tests.helpers.call_action and arbitrary number of additional named arguments, that will be used as resource properties.

Example:

def test_uploaded_resource(create_with_upload):
    dataset = factories.Dataset()
    resource = create_with_upload(
        "hello world", "file.txt", url="http://data",
        package_id=dataset["id"])
    assert resource["url_type"] == "upload"
    assert resource["format"] == "TXT"
    assert resource["size"] == 11

Mocking: the `mock` library

We use the mock library to replace parts of CKAN with mock objects. This allows a CKAN function to be tested independently of other parts of CKAN or third-party libraries that the function uses. This generally makes the test simpler and faster (especially when ckan.model is mocked out so that the tests don’t touch the database). With mock objects we can also make assertions about what methods the function called on the mock object and with which arguments.

Note

Overuse of mocking is discouraged as it can make tests difficult to understand and maintain. Mocking can be useful and make tests both faster and simpler when used appropriately. Some rules of thumb:

Don’t mock out more than one or two objects in a single test method.
Don’t use mocking in more functional-style tests. For example the action function tests in ckan.tests.logic.action and the frontend tests in ckan.tests.controllers are functional tests, and probably shouldn’t do any mocking.
Do use mocking in more unit-style tests. For example the authorization function tests in ckan.tests.logic.auth, the converter and validator tests in ckan.tests.logic.auth, and most (all?) lib tests in ckan.tests.lib are unit tests and should use mocking when necessary (often it’s possible to unit test a method in isolation from other CKAN code without doing any mocking, which is ideal).

In these kind of tests we can often mock one or two objects in a simple and easy to understand way, and make the test both simpler and faster.

A mock object is a special object that allows user code to access any attribute name or call any method name (and pass any parameters) on the object, and the code will always get another mock object back:

>>> import unittest.mock as mock
>>> my_mock = mock.MagicMock()
>>> my_mock.foo
<MagicMock name='mock.foo' id='56032400'>
>>> my_mock.bar
<MagicMock name='mock.bar' id='54093968'>
>>> my_mock.foobar()
<MagicMock name='mock.foobar()' id='54115664'>
>>> my_mock.foobar(1, 2, 'barfoo')
<MagicMock name='mock.foobar()' id='54115664'>

When a test needs a mock object to actually have some behavior besides always returning other mock objects, it can set the value of a certain attribute on the mock object, set the return value of a certain method, specify that a certain method should raise a certain exception, etc.

You should read the mock library’s documentation to really understand what’s going on, but here’s an example of a test from ckan.tests.logic.auth.test_update that tests the user_update() authorization function and mocks out ckan.model:

def test_user_update_user_cannot_update_another_user():
    """Users should not be able to update other users' accounts."""

    # 1. Setup.

    # Make a mock ckan.model.User object, Fred.
    fred = factories.MockUser()

    # Make a mock ckan.model object.
    mock_model = mock.MagicMock()
    # model.User.get(user_id) should return Fred.
    mock_model.User.get.return_value = fred

    # Put the mock model in the context.
    # This is easier than patching import ckan.model.
    context = {"model": mock_model}

    # The logged-in user is going to be Bob, not Fred.
    context["user"] = "bob"

    # 2. Call the function that's being tested, once only.

    # Make Bob try to update Fred's user account.
    params = {"id": fred.id, "name": "updated_user_name"}

    # 3. Make assertions about the return value and/or side-effects.

    with pytest.raises(logic.NotAuthorized):
        helpers.call_auth("user_update", context=context, **params)

    # 4. Do nothing else!

The following sections will give specific guidelines and examples for writing tests for each module in CKAN.

Note

When we say that all functions should have tests in the sections below, we mean all public functions that the module or class exports for use by other modules or classes in CKAN or by extensions or templates.

Private helper methods (with names beginning with _) never have to have their own tests, although they can have tests if helpful.

Writing `ckan.logic.action` tests

All action functions should have tests.

Most action function tests will be high-level tests that both test the code in the action function itself, and also indirectly test the code in ckan.lib, ckan.model, ckan.logic.schema etc. that the action function calls. This means that most action function tests should not use mocking.

Tests for action functions should use the ckan.tests.helpers.call_action() function to call the action functions.

One thing call_action() does is to add ignore_auth: True into the context dict that’s passed to the action function, so that CKAN will not call the action function’s authorization function. The tests for an action function don’t need to cover authorization, because the authorization functions have their own tests in ckan.tests.logic.auth. But action function tests do need to cover validation, more on that later.

Action function tests should test the logic of the actions themselves, and should test validation (e.g. that various kinds of valid input work as expected, and invalid inputs raise the expected exceptions).

Here’s an example of a simple ckan.logic.action test:

    def test_user_update_name(self):
        """Test that updating a user's name works successfully."""

        # The canonical form of a test has four steps:
        # 1. Setup any preconditions needed for the test.
        # 2. Call the function that's being tested, once only.
        # 3. Make assertions about the return value and/or side-effects of
        #    of the function that's being tested.
        # 4. Do nothing else!

        # 1. Setup.
        user = factories.User()
        user["name"] = "updated"

        # 2. Make assertions about the return value and/or side-effects.
        with pytest.raises(logic.ValidationError):
            helpers.call_action("user_update", **user)

Todo

Insert the names of all tests for ckan.logic.action.update.user_update, for example, to show what level of detail things should be tested in.

Writing `ckan.logic.auth` tests

All auth functions should have tests.

Most auth function tests should be unit tests that test the auth function in isolation, without bringing in other parts of CKAN or touching the database. This requires using the mock library to mock ckan.model, see Mocking: the mock library.

Tests for auth functions should use the ckan.tests.helpers.call_auth() function to call auth functions.

Here’s an example of a simple ckan.logic.auth test:

def test_user_update_user_cannot_update_another_user():
    """Users should not be able to update other users' accounts."""

    # 1. Setup.

    # Make a mock ckan.model.User object, Fred.
    fred = factories.MockUser()

    # Make a mock ckan.model object.
    mock_model = mock.MagicMock()
    # model.User.get(user_id) should return Fred.
    mock_model.User.get.return_value = fred

    # Put the mock model in the context.
    # This is easier than patching import ckan.model.
    context = {"model": mock_model}

    # The logged-in user is going to be Bob, not Fred.
    context["user"] = "bob"

    # 2. Call the function that's being tested, once only.

    # Make Bob try to update Fred's user account.
    params = {"id": fred.id, "name": "updated_user_name"}

    # 3. Make assertions about the return value and/or side-effects.

    with pytest.raises(logic.NotAuthorized):
        helpers.call_auth("user_update", context=context, **params)

    # 4. Do nothing else!

Writing converter and validator tests

All converter and validator functions should have unit tests.

Although these converter and validator functions are tested indirectly by the action function tests, this may not catch all the converters and validators and all their options, and converters and validators are not only used by the action functions but are also available to plugins. Having unit tests will also help to clarify the intended behavior of each converter and validator.

CKAN’s action functions call ckan.lib.navl.dictization_functions.validate() to validate data posted by the user. Each action function passes a schema from ckan.logic.schema to validate(). The schema gives validate() lists of validation and conversion functions to apply to the user data. These validation and conversion functions are defined in ckan.logic.validators, ckan.logic.converters and ckan.lib.navl.validators.

Most validator and converter tests should be unit tests that test the validator or converter function in isolation, without bringing in other parts of CKAN or touching the database. This requires using the mock library to mock ckan.model, see Mocking: the mock library.

When testing validators, we often want to make the same assertions in many tests: assert that the validator didn’t modify the data dict, assert that the validator didn’t modify the errors dict, assert that the validator raised Invalid, etc. Decorator functions are defined at the top of validator test modules like ckan.tests.logic.test_validators to make these common asserts easy. To use one of these decorators you have to:

Define a nested function inside your test method, that simply calls the validator function that you’re trying to test.
Apply the decorators that you want to this nested function.
Call the nested function.

Here’s an example of a simple validator test that uses this technique:

def test_user_name_validator_with_non_string_value():
    """user_name_validator() should raise Invalid if given a non-string
    value.

    """
    non_string_values = [
        13,
        23.7,
        100,
        1.0j,
        None,
        True,
        False,
        ("a", 2, False),
        [13, None, True],
        {"foo": "bar"},
        lambda x: x ** 2,
    ]

    # Mock ckan.model.
    mock_model = mock.MagicMock()
    # model.User.get(some_user_id) needs to return None for this test.
    mock_model.User.get.return_value = None

    key = ("name",)
    for non_string_value in non_string_values:
        data = validator_data_dict()
        data[key] = non_string_value
        errors = validator_errors_dict()
        errors[key] = []

        @t.does_not_modify_data_dict
        @raises_invalid
        def call_validator(*args, **kwargs):
            return validators.user_name_validator(*args, **kwargs)

        call_validator(key, data, errors, context={"model": mock_model})

No tests for `ckan.logic.schema.py`

We don’t write tests for the schemas defined in ckan.logic.schema. The validation done by the schemas is instead tested indirectly by the action function tests. The reason for this is that CKAN actually does validation in multiple places: some validation is done using schemas, some validation is done in the action functions themselves, some is done in dictization, and some in the model. By testing all the different valid and invalid inputs at the action function level, we catch it all in one place.

Writing `ckan.controllers` tests

Controller tests probably shouldn’t use mocking.

Todo

Write the tests for one controller, figuring out the best way to write controller tests. Then fill in this guidelines section, using the first set of controller tests as an example.

Some things have been decided already:

All controller methods should have tests
Controller tests should be high-level tests that work by posting simulated HTTP requests to CKAN URLs and testing the response. So the controller tests are also testing CKAN’s templates and rendering - these are CKAN’s front-end tests.

For example, maybe we use a testapp and then use beautiful soup to parse the HTML?
In general the tests for a controller shouldn’t need to be too detailed, because there shouldn’t be a lot of complicated logic and code in controller classes. The logic should be handled in other places such as ckan.logic and ckan.lib, where it can be tested easily and also shared with other code.
The tests for a controller should:
- Make sure that the template renders without crashing.
- Test that the page contents seem basically correct, or test certain important elements in the page contents (but don’t do too much HTML parsing).
- Test that submitting any forms on the page works without crashing and has the expected side-effects.
- When asserting side-effects after submitting a form, controller tests should user the ckan.tests.helpers.call_action() function. For example after creating a new user by submitting the new user form, a test could call the user_show() action function to verify that the user was created with the correct values.

Warning

Some CKAN controllers do contain a lot of complicated logic code. These controllers should be refactored to move the logic into ckan.logic or ckan.lib where it can be tested easily. Unfortunately in cases like this it may be necessary to write a lot of controller tests to get this code’s behavior into a test harness before it can be safely refactored.

Writing `ckan.model` tests

All model methods should have tests.

Todo

Write the tests for one ckan.model module, figuring out the best way to write model tests. Then fill in this guidelines section, using the first set of model tests as an example.

Writing `ckan.lib` tests

All lib functions should have tests.

Todo

Write the tests for one ckan.lib module, figuring out the best way to write lib tests. Then fill in this guidelines section, using the first

We probably want to make these unit tests rather than high-level tests and mock out ckan.model, so the tests are really fast and simple.

Note that some things in lib are particularly important, e.g. the functions in ckan.lib.helpers are exported for templates (including extensions) to use, so all of these functions should really have tests and docstrings. It’s probably worth focusing on these modules first.

Writing `ckan.plugins` tests

The plugin interfaces in ckan.plugins.interfaces are not directly testable because they don’t contain any code, but:

Each plugin interface should have an example plugin in ckan.ckanext and the example plugin should have its own functional tests.
The tests for the code that calls the plugin interface methods should test that the methods are called correctly.

For example ckan.logic.action.get.package_show() calls ckan.plugins.interfaces.IDatasetForm.read(), so the package_show() tests should include tests that read() is called at the right times and with the right parameters.

Everything in ckan.plugins.toolkit should have tests, because these functions are part of the API for extensions to use. But toolkit imports most of these functions from elsewhere in CKAN, so the tests should be elsewhere also, in the test modules for the modules where the functions are defined.

Other than the plugin interfaces and plugins toolkit, any other code in ckan.plugins should have tests.

Writing `ckan.ckanext` tests

Within extensions, follow the same guidelines as for CKAN core. For example if an extension adds an action function then the action function should have tests, etc.

Testing coding standards

Guidelines for writing tests

How should tests be organized?

Naming test methods

Recipe for a test method

How detailed should tests be?

Creating test objects: ckan.tests.factories

Test helper functions: ckan.tests.helpers

Pytest fixtures

Mocking: the mock library

Writing ckan.logic.action tests

Writing ckan.logic.auth tests

Writing converter and validator tests

No tests for ckan.logic.schema.py

Writing ckan.controllers tests

Writing ckan.model tests

Writing ckan.lib tests

Writing ckan.plugins tests

Writing ckan.ckanext tests

Creating test objects: `ckan.tests.factories`

Test helper functions: `ckan.tests.helpers`

Mocking: the `mock` library

Writing `ckan.logic.action` tests

Writing `ckan.logic.auth` tests

No tests for `ckan.logic.schema.py`

Writing `ckan.controllers` tests

Writing `ckan.model` tests

Writing `ckan.lib` tests

Writing `ckan.plugins` tests

Writing `ckan.ckanext` tests