Testing coding standards¶
All new code, or changes to existing code, should have new or updated tests before being merged into master. This document gives some guidelines for developers who are writing tests or reviewing code for CKAN.
See also
- Testing CKAN
- How to set up your development environment to run CKAN’s test suite
- Testing code that uses background jobs
- How to handle asynchronous background jobs in your tests
Transitioning from legacy to new tests¶
CKAN is an old code base with a large legacy test suite in
ckan.tests.legacy
. The legacy tests are difficult to maintain and
extend, but are too many to be replaced all at once in a single effort. So
we’re following this strategy:
- A new test suite has been started in
ckan.tests
. - For now, we’ll run both the legacy tests and the new tests before merging something into the master branch.
- Whenever we add new code, or change existing code, we’ll add new-style tests for it.
- If you change the behavior of some code and break some legacy tests, consider adding new tests for that code and deleting the legacy tests, rather than updating the legacy tests.
- Now and then, we’ll write a set of new tests to cover part of the code, and delete the relevant legacy tests. For example if you want to refactor some code that doesn’t have good tests, write a set of new-style tests for it first, refactor, then delete the relevant legacy tests.
In this way we can incrementally extend the new tests to cover CKAN one “island
of code” at a time, and eventually we can delete the legacy ckan.tests
directory entirely.
Guidelines for writing new-style tests¶
We want the tests in ckan.tests
to be:
- Fast
Don’t share setup code between tests (e.g. in test class
setup()
orsetup_class()
methods, saved against theself
attribute of test classes, or in test helper modules).Instead write helper functions that create test objects and return them, and have each test method call just the helpers it needs to do the setup that it needs.
Where appropriate, use the
mock
library to avoid pulling in other parts of CKAN (especially the database), see Mocking: the mock library.
- Independent
- Each test module, class and method should be able to be run on its own.
- Tests shouldn’t be tightly coupled to each other, changing a test shouldn’t affect other tests.
- Clear
It should be quick and easy to see what went wrong when a test fails, or to see what a test does and how it works if you have to debug or update a test. If you think the test or helper method isn’t clear by itself, add docstrings.
You shouldn’t have to figure out what a complex test method does, or go and look up a lot of code in other files to understand a test method.
- Tests should follow the canonical form for a unit test, see Recipe for a test method.
- Write lots of small, simple test methods not a few big, complex tests.
- Each test method should test just One Thing.
- The name of a test method should clearly explain the intent of the test. See Naming test methods.
- Easy to find
It should be easy to know where to add new tests for some new or changed code, or to find the existing tests for some code.
- Easy to write
- Writing lots of small, clear and simple tests that all follow similar recipes and organization should make tests easy to write, as well as easy to read.
The follow sections give some more specific guidelines and tips for writing CKAN tests.
How should tests be organized?¶
The organization of test modules in ckan.tests
mirrors the
organization of the source modules in ckan
:
ckan/
tests/
controllers/
test_package.py <-- Tests for ckan/controllers/package.py
...
lib/
test_helpers.py <-- Tests for ckan/lib/helpers.py
...
logic/
action/
test_get.py
...
auth/
test_get.py
...
test_converters.py
test_validators.py
migration/
versions/
test_001_add_existing_tables.py
...
model/
test_package.py
...
...
There are a few exceptional test modules that don’t fit into this structure,
for example PEP8 tests and coding standards tests. These modules can just go in
the top-level ckan/tests/
directory. There shouldn’t be too many of these.
Naming test methods¶
The name of a test method should clearly explain the intent of the test.
Test method names are printed out when tests fail, so the user can often see what went wrong without having to look into the test file. When they do need to look into the file to debug or update a test, the test name helps to clarify the test.
Do this even if it means your method name gets really long, since we don’t write code that calls our test methods there’s no advantage to having short test method names.
Some modules in CKAN contain large numbers of loosely related functions.
For example, ckan.logic.action.update
contains all functions for
updating things in CKAN. This means that
ckan.tests.logic.action.test_update
is going to contain an even larger
number of test functions.
So as well as the name of each test method explaining the intent of the test, tests should be grouped by a test class that aggregates tests against a model entity or action type, for instance:
class TestPackageCreate(object):
# ...
def test_it_validates_name(self):
# ...
def test_it_validates_url(self):
# ...
class TestResourceCreate(object)
# ...
def test_it_validates_package_id(self):
# ...
# ...
Good test names:
TestUserUpdate.test_update_with_id_that_does_not_exist
TestUserUpdate.test_update_with_no_id
TestUserUpdate.test_update_with_invalid_name
Bad test names:
test_user_update
test_update_pkg_1
test_package
Recipe for a test method¶
The Pylons Unit Testing Guidelines give the following recipe for all unit test methods to follow:
- Set up the preconditions for the method / function being tested.
- Call the method / function exactly one time, passing in the values established in the first step.
- Make assertions about the return value, and / or any side effects.
- Do absolutely nothing else.
Most CKAN tests should follow this form. Here’s an example of a simple action function test demonstrating the recipe:
def test_user_update_name(self):
'''Test that updating a user's name works successfully.'''
# The canonical form of a test has four steps:
# 1. Setup any preconditions needed for the test.
# 2. Call the function that's being tested, once only.
# 3. Make assertions about the return value and/or side-effects of
# of the function that's being tested.
# 4. Do nothing else!
# 1. Setup.
user = factories.User()
user['name'] = 'updated'
# 2. Make assertions about the return value and/or side-effects.
assert_raises(logic.ValidationError,
helpers.call_action, 'user_update',
**user)
One common exception is when you want to use a for
loop to call the
function being tested multiple times, passing it lots of different arguments
that should all produce the same return value and/or side effects. For example,
this test from ckan.tests.logic.action.test_update
:
def test_user_update_with_invalid_name(self):
user = factories.User()
invalid_names = ('', 'a', False, 0, -1, 23, 'new', 'edit', 'search',
'a' * 200, 'Hi!', 'i++%')
for name in invalid_names:
user['name'] = name
assert_raises(logic.ValidationError,
helpers.call_action, 'user_update',
**user)
The behavior of user_update()
is the same
for every invalid value.
We do want to test user_update()
with lots
of different invalid names, but we obviously don’t want to write a dozen
separate test methods that are all the same apart from the value used for the
invalid user name. We don’t really want to define a helper method and a dozen
test methods that call it either. So we use a simple loop. Technically this
test calls the function being tested more than once, but there’s only one line
of code that calls it.
How detailed should tests be?¶
Generally, what we’re trying to do is test the interfaces between modules in a way that supports modularization: if you change the code within a function, method, class or module, if you don’t break any of that code’s tests you should be able to expect that CKAN as a whole will not be broken.
As a general guideline, the tests for a function or method should:
- Test for success:
- Test the function with typical, valid input values
- Test with valid, edge-case inputs
- If the function has multiple parameters, test them in different combinations
- Test for failure:
- Test that the function fails correctly (e.g. raises the expected type of exception) when given likely invalid inputs (for example, if the user passes an invalid user_id as a parameter)
- Test that the function fails correctly when given bizarre input
- Test that the function behaves correctly when given unicode characters as input
- Cover the interface of the function: test all the parameters and features of the function
Creating test objects: ckan.tests.factories
¶
This is a collection of factory classes for building CKAN users, datasets, etc.
These are meant to be used by tests to create any objects that are needed for
the tests. They’re written using factory_boy
:
http://factoryboy.readthedocs.org/en/latest/
These are not meant to be used for the actual testing, e.g. if you’re writing
a test for the user_create()
function then
call call_action()
, don’t test it via the
User
factory below.
Usage:
# Create a user with the factory's default attributes, and get back a
# user dict:
user_dict = factories.User()
# You can create a second user the same way. For attributes that can't be
# the same (e.g. you can't have two users with the same name) a new value
# will be generated each time you use the factory:
another_user_dict = factories.User()
# Create a user and specify your own user name and email (this works
# with any params that CKAN's user_create() accepts):
custom_user_dict = factories.User(name='bob', email='bob@bob.com')
# Get a user dict containing the attributes (name, email, password, etc.)
# that the factory would use to create a user, but without actually
# creating the user in CKAN:
user_attributes_dict = factories.User.attributes()
# If you later want to create a user using these attributes, just pass them
# to the factory:
user = factories.User(**user_attributes_dict)
-
class
ckan.tests.factories.
User
¶ A factory class for creating CKAN users.
-
class
ckan.tests.factories.
Resource
¶ A factory class for creating CKAN resources.
-
class
ckan.tests.factories.
ResourceView
¶ A factory class for creating CKAN resource views.
Note: if you use this factory, you need to load the image_view plugin on your test class (and unload it later), otherwise you will get an error.
Example:
class TestSomethingWithResourceViews(object): @classmethod def setup_class(cls): if not p.plugin_loaded('image_view'): p.load('image_view') @classmethod def teardown_class(cls): p.unload('image_view')
-
class
ckan.tests.factories.
Sysadmin
¶ A factory class for creating sysadmin users.
-
class
ckan.tests.factories.
Group
¶ A factory class for creating CKAN groups.
-
class
ckan.tests.factories.
Organization
¶ A factory class for creating CKAN organizations.
-
class
ckan.tests.factories.
Dataset
¶ A factory class for creating CKAN datasets.
-
class
ckan.tests.factories.
MockUser
¶ A factory class for creating mock CKAN users using the mock library.
-
FACTORY_FOR
¶ alias of
mock.mock.MagicMock
-
-
class
ckan.tests.factories.
SystemInfo
¶ A factory class for creating SystemInfo objects (config objects stored in the DB).
-
ckan.tests.factories.
validator_data_dict
()¶ Return a data dict with some arbitrary data in it, suitable to be passed to validator functions for testing.
-
ckan.tests.factories.
validator_errors_dict
()¶ Return an errors dict with some arbitrary errors in it, suitable to be passed to validator functions for testing.
-
class
ckan.tests.factories.
Vocabulary
¶ A factory class for creating tag vocabularies.
Test helper functions: ckan.tests.helpers
¶
This is a collection of helper functions for use in tests.
We want to avoid sharing test helper functions between test modules as much as possible, and we definitely don’t want to share test fixtures between test modules, or to introduce a complex hierarchy of test class subclasses, etc.
We want to reduce the amount of “travel” that a reader needs to undertake to understand a test method – reducing the number of other files they need to go and read to understand what the test code does. And we want to avoid tightly coupling test modules to each other by having them share code.
But some test helper functions just increase the readability of tests so much and make writing tests so much easier, that it’s worth having them despite the potential drawbacks.
This module is reserved for these very useful functions.
-
ckan.tests.helpers.
reset_db
()¶ Reset CKAN’s database.
If a test class uses the database, then it should call this function in its
setup()
method to make sure that it has a clean database to start with (nothing left over from other test classes or from previous test runs).If a test class doesn’t use the database (and most test classes shouldn’t need to) then it doesn’t need to call this function.
Returns: None
-
ckan.tests.helpers.
call_action
(action_name, context=None, **kwargs)¶ Call the named
ckan.logic.action
function and return the result.This is just a nicer way for user code to call action functions, nicer than either calling the action function directly or via
ckan.logic.get_action()
.For example:
user_dict = call_action('user_create', name='seanh', email='seanh@seanh.com', password='pass')
Any keyword arguments given will be wrapped in a dict and passed to the action function as its
data_dict
argument.Note: this skips authorization! It passes ‘ignore_auth’: True to action functions in their
context
dicts, so the corresponding authorization functions will not be run. This is because ckan.tests.logic.action tests only the actions, the authorization functions are tested separately in ckan.tests.logic.auth. See the testing guidelines for more info.This function should eventually be moved to
ckan.logic.call_action()
and the currentckan.logic.get_action()
function should be deprecated. The tests may still need their own wrapper function forckan.logic.call_action()
, e.g. to insert'ignore_auth': True
into thecontext
dict.Parameters: - action_name (string) – the name of the action function to call, e.g.
'user_update'
- context (dict) – the context dict to pass to the action function (optional, if no context is given a default one will be supplied)
Returns: the dict or other value that the action function returns
- action_name (string) – the name of the action function to call, e.g.
-
ckan.tests.helpers.
call_auth
(auth_name, context, **kwargs)¶ Call the named
ckan.logic.auth
function and return the result.This is just a convenience function for tests in
ckan.tests.logic.auth
to use.Usage:
result = helpers.call_auth('user_update', context=context, id='some_user_id', name='updated_user_name')
Parameters: - auth_name (string) – the name of the auth function to call, e.g.
'user_update'
- context (dict) – the context dict to pass to the auth function, must
contain
'user'
and'model'
keys, e.g.{'user': 'fred', 'model': my_mock_model_object}
Returns: the dict that the auth function returns, e.g.
{'success': True}
or{'success': False, msg: '...'}
or just{'success': False}
Return type: dict
- auth_name (string) – the name of the auth function to call, e.g.
-
class
ckan.tests.helpers.
CKANTestApp
(app, extra_environ=None, relative_to=None, use_unicode=True)¶ A wrapper around webtest.TestApp
It adds some convenience methods for CKAN
-
class
ckan.tests.helpers.
FunctionalTestBase
¶ A base class for functional test classes to inherit from.
Allows configuration changes by overriding _apply_config_changes and resetting the CKAN config after your test class has run. It creates a webtest.TestApp at self.app for your class to use to make HTTP requests to the CKAN web UI or API. Also loads plugins defined by _load_plugins in the class definition.
If you’re overriding methods that this class provides, like setup_class() and teardown_class(), make sure to use super() to call this class’s methods at the top of yours!
-
setup
()¶ Reset the database and clear the search indexes.
-
-
class
ckan.tests.helpers.
RQTestBase
¶ Base class for tests of RQ functionality.
-
setup
()¶ Delete all RQ queues and jobs.
-
all_jobs
()¶ Get a list of all RQ jobs.
-
enqueue
(job=None, *args, **kwargs)¶ Enqueue a test job.
-
-
class
ckan.tests.helpers.
FunctionalRQTestBase
¶ Base class for functional tests of RQ functionality.
-
setup
()¶ Reset the database and clear the search indexes.
-
-
ckan.tests.helpers.
submit_and_follow
(app, form, extra_environ=None, name=None, value=None, **args)¶ Call webtest_submit with name/value passed expecting a redirect and return the response from following that redirect.
-
ckan.tests.helpers.
webtest_submit
(form, name=None, index=None, value=None, **args)¶ backported version of webtest.Form.submit that actually works for submitting with different submit buttons.
We’re stuck on an old version of webtest because we’re stuck on an old version of webob because we’re stuck on an old version of Pylons. This prolongs our suffering, but on the bright side it lets us have functional tests that work.
-
ckan.tests.helpers.
webtest_submit_fields
(form, name=None, index=None, submit_value=None)¶ backported version of webtest.Form.submit_fields that actually works for submitting with different submit buttons.
-
ckan.tests.helpers.
webtest_maybe_follow
(response, **kw)¶ Follow all redirects. If this response is not a redirect, do nothing. Returns another response object.
(backported from WebTest 2.0.1)
-
ckan.tests.helpers.
change_config
(key, value)¶ Decorator to temporarily change CKAN’s config to a new value
This allows you to easily create tests that need specific config values to be set, making sure it’ll be reverted to what it was originally, after your test is run.
Usage:
@helpers.change_config('ckan.site_title', 'My Test CKAN') def test_ckan_site_title(self): assert config['ckan.site_title'] == 'My Test CKAN'
Parameters: - key (string) – the config key to be changed, e.g.
'ckan.site_title'
- value (string) – the new config key’s value, e.g.
'My Test CKAN'
See also
The context manager
changed_config()
- key (string) – the config key to be changed, e.g.
-
ckan.tests.helpers.
changed_config
(*args, **kwds)¶ Context manager for temporarily changing a config value.
Allows you to temporarily change the value of a CKAN configuration option. The original value is restored once the context manager is left.
Usage:
with changed_config(u'ckan.site_title', u'My Test CKAN'): assert config[u'ckan.site_title'] == u'My Test CKAN'
See also
The decorator
change_config()
-
ckan.tests.helpers.
mock_auth
(auth_function_path)¶ - Decorator to easily mock a CKAN auth method in the context of a test
- function
- It adds a mock object for the provided auth_function_path as a parameter to
- the test function.
- Essentially it makes sure that ckan.authz.clear_auth_functions_cache is
- called before and after to make sure that the auth functions pick up the newly changed values.
Usage:
@helpers.mock_auth('ckan.logic.auth.create.package_create') def test_mock_package_create(self, mock_package_create): from ckan import logic mock_package_create.return_value = {'success': True} # package_create is mocked eq_(logic.check_access('package_create', {}), True) assert mock_package_create.called
Parameters: action_name (string) – the full path to the auth function to be mocked, e.g. ckan.logic.auth.create.package_create
-
ckan.tests.helpers.
mock_action
(action_name)¶ Decorator to easily mock a CKAN action in the context of a test function
It adds a mock object for the provided action as a parameter to the test function. The mock is discarded at the end of the function, even if there is an exception raised.
Note that this mocks the action both when it’s called directly via
ckan.logic.get_action
and viackan.plugins.toolkit.get_action
.Usage:
@mock_action('user_list') def test_mock_user_list(self, mock_user_list): mock_user_list.return_value = 'hi' # user_list is mocked eq_(helpers.call_action('user_list', {}), 'hi') assert mock_user_list.called
Parameters: action_name (string) – the name of the action to be mocked, e.g. package_create
-
ckan.tests.helpers.
set_extra_environ
(key, value)¶ Decorator to temporarily changes a single request environemnt value
Create a new test app and use the a side effect of making a request to set an extra_environ value. Reset the value to ‘’ after the test.
Usage:
@helpers.extra_environ('SCRIPT_NAME', '/myscript') def test_ckan_thing_affected_by_script_name(self): # ...
Parameters: - key (string) – the extra_environ key to be changed, e.g.
'SCRIPT_NAME'
- value (string) – the new extra_environ key’s value, e.g.
'/myscript'
- key (string) – the extra_environ key to be changed, e.g.
-
ckan.tests.helpers.
recorded_logs
(*args, **kwds)¶ Context manager for recording log messages.
Parameters: - logger – The logger to record messages from. Can either be a
logging.Logger
instance or a string with the logger’s name. Defaults to the root logger. - level (int) – Temporary log level for the target logger while
the context manager is active. Pass
None
if you don’t want the level to be changed. The level is automatically reset to its original value when the context manager is left. - override_disabled (bool) – A logger can be disabled by setting
its
disabled
attribute. By default, this context manager sets that attribute toFalse
at the beginning of its execution and resets it when the context manager is left. Setoverride_disabled
toFalse
to keep the current value of the attribute. - override_global_level (bool) – The
logging.disable
function allows one to install a global minimum log level that takes precedence over a logger’s own level. By default, this context manager makes sure that the global limit is at mostlevel
, and reduces it if necessary during its execution. Setoverride_global_level
toFalse
to keep the global limit.
Returns: A recording log handler that listens to
logger
during the execution of the context manager.Return type: Example:
import logging logger = logging.getLogger(__name__) with recorded_logs(logger) as logs: logger.info(u'Hello, world!') logs.assert_log(u'info', u'world')
- logger – The logger to record messages from. Can either be a
-
class
ckan.tests.helpers.
RecordingLogHandler
(*args, **kwargs)¶ Log handler that records log messages for later inspection.
You can inspect the recorded messages via the
messages
attribute (a dict that maps log levels to lists of messages) or by usingassert_log
.This class is rarely useful on its own, instead use
recorded_logs()
to temporarily record log messages.-
emit
(record)¶ Do whatever it takes to actually log the specified logging record.
This version is intended to be implemented by subclasses and so raises a NotImplementedError.
-
assert_log
(level, pattern, msg=None)¶ Assert that a certain message has been logged.
Parameters: - pattern (string) – A regex which the message has to match.
The match is done using
re.search
. - level (string) – The message level (
'debug'
, …). - msg (string) – Optional failure message in case the expected log message was not logged.
Raises: AssertionError – If the expected message was not logged.
- pattern (string) – A regex which the message has to match.
The match is done using
-
clear
()¶ Clear all captured log messages.
-
Mocking: the mock
library¶
We use the mock library to
replace parts of CKAN with mock objects. This allows a CKAN
function to be tested independently of other parts of CKAN or third-party
libraries that the function uses. This generally makes the test simpler and
faster (especially when ckan.model
is mocked out so that the tests
don’t touch the database). With mock objects we can also make assertions about
what methods the function called on the mock object and with which arguments.
Note
Overuse of mocking is discouraged as it can make tests difficult to understand and maintain. Mocking can be useful and make tests both faster and simpler when used appropriately. Some rules of thumb:
Don’t mock out more than one or two objects in a single test method.
Don’t use mocking in more functional-style tests. For example the action function tests in
ckan.tests.logic.action
and the frontend tests inckan.tests.controllers
are functional tests, and probably shouldn’t do any mocking.Do use mocking in more unit-style tests. For example the authorization function tests in
ckan.tests.logic.auth
, the converter and validator tests inckan.tests.logic.auth
, and most (all?) lib tests inckan.tests.lib
are unit tests and should use mocking when necessary (often it’s possible to unit test a method in isolation from other CKAN code without doing any mocking, which is ideal).In these kind of tests we can often mock one or two objects in a simple and easy to understand way, and make the test both simpler and faster.
A mock object is a special object that allows user code to access any attribute name or call any method name (and pass any parameters) on the object, and the code will always get another mock object back:
>>> import mock
>>> my_mock = mock.MagicMock()
>>> my_mock.foo
<MagicMock name='mock.foo' id='56032400'>
>>> my_mock.bar
<MagicMock name='mock.bar' id='54093968'>
>>> my_mock.foobar()
<MagicMock name='mock.foobar()' id='54115664'>
>>> my_mock.foobar(1, 2, 'barfoo')
<MagicMock name='mock.foobar()' id='54115664'>
When a test needs a mock object to actually have some behavior besides always returning other mock objects, it can set the value of a certain attribute on the mock object, set the return value of a certain method, specify that a certain method should raise a certain exception, etc.
You should read the mock library’s documentation to really understand what’s
going on, but here’s an example of a test from
ckan.tests.logic.auth.test_update
that tests the
user_update()
authorization function and mocks
out ckan.model
:
def test_user_update_user_cannot_update_another_user(self):
'''Users should not be able to update other users' accounts.'''
# 1. Setup.
# Make a mock ckan.model.User object, Fred.
fred = factories.MockUser(name='fred')
# Make a mock ckan.model object.
mock_model = mock.MagicMock()
# model.User.get(user_id) should return Fred.
mock_model.User.get.return_value = fred
# Put the mock model in the context.
# This is easier than patching import ckan.model.
context = {'model': mock_model}
# The logged-in user is going to be Bob, not Fred.
context['user'] = 'bob'
# 2. Call the function that's being tested, once only.
# Make Bob try to update Fred's user account.
params = {
'id': fred.id,
'name': 'updated_user_name',
}
# 3. Make assertions about the return value and/or side-effects.
nose.tools.assert_raises(logic.NotAuthorized, helpers.call_auth,
'user_update', context=context, **params)
# 4. Do nothing else!
The following sections will give specific guidelines and examples for writing tests for each module in CKAN.
Note
When we say that all functions should have tests in the sections below, we mean all public functions that the module or class exports for use by other modules or classes in CKAN or by extensions or templates.
Private helper methods (with names beginning with _
) never have to
have their own tests, although they can have tests if helpful.
Writing ckan.logic.action
tests¶
All action functions should have tests.
Most action function tests will be high-level tests that both test the code in
the action function itself, and also indirectly test the code in
ckan.lib
, ckan.model
, ckan.logic.schema
etc. that the
action function calls. This means that most action function tests should not
use mocking.
Tests for action functions should use the
ckan.tests.helpers.call_action()
function to call the action
functions.
One thing call_action()
does is to add
ignore_auth: True
into the context
dict that’s passed to the action
function, so that CKAN will not call the action function’s authorization
function. The tests for an action function don’t need to cover
authorization, because the authorization functions have their own tests in
ckan.tests.logic.auth
. But action function tests do need to cover
validation, more on that later.
Action function tests should test the logic of the actions themselves, and should test validation (e.g. that various kinds of valid input work as expected, and invalid inputs raise the expected exceptions).
Here’s an example of a simple ckan.logic.action
test:
def test_user_update_name(self):
'''Test that updating a user's name works successfully.'''
# The canonical form of a test has four steps:
# 1. Setup any preconditions needed for the test.
# 2. Call the function that's being tested, once only.
# 3. Make assertions about the return value and/or side-effects of
# of the function that's being tested.
# 4. Do nothing else!
# 1. Setup.
user = factories.User()
user['name'] = 'updated'
# 2. Make assertions about the return value and/or side-effects.
assert_raises(logic.ValidationError,
helpers.call_action, 'user_update',
**user)
Todo
Insert the names of all tests for ckan.logic.action.update.user_update
,
for example, to show what level of detail things should be tested in.
Writing ckan.logic.auth
tests¶
All auth functions should have tests.
Most auth function tests should be unit tests that test the auth function in
isolation, without bringing in other parts of CKAN or touching the database.
This requires using the mock
library to mock ckan.model
, see
Mocking: the mock library.
Tests for auth functions should use the
ckan.tests.helpers.call_auth()
function to call auth functions.
Here’s an example of a simple ckan.logic.auth
test:
def test_user_update_user_cannot_update_another_user(self):
'''Users should not be able to update other users' accounts.'''
# 1. Setup.
# Make a mock ckan.model.User object, Fred.
fred = factories.MockUser(name='fred')
# Make a mock ckan.model object.
mock_model = mock.MagicMock()
# model.User.get(user_id) should return Fred.
mock_model.User.get.return_value = fred
# Put the mock model in the context.
# This is easier than patching import ckan.model.
context = {'model': mock_model}
# The logged-in user is going to be Bob, not Fred.
context['user'] = 'bob'
# 2. Call the function that's being tested, once only.
# Make Bob try to update Fred's user account.
params = {
'id': fred.id,
'name': 'updated_user_name',
}
# 3. Make assertions about the return value and/or side-effects.
nose.tools.assert_raises(logic.NotAuthorized, helpers.call_auth,
'user_update', context=context, **params)
# 4. Do nothing else!
Writing converter and validator tests¶
All converter and validator functions should have unit tests.
Although these converter and validator functions are tested indirectly by the action function tests, this may not catch all the converters and validators and all their options, and converters and validators are not only used by the action functions but are also available to plugins. Having unit tests will also help to clarify the intended behavior of each converter and validator.
CKAN’s action functions call
ckan.lib.navl.dictization_functions.validate()
to validate data posted
by the user. Each action function passes a schema from
ckan.logic.schema
to
validate()
. The schema gives
validate()
lists of validation
and conversion functions to apply to the user data. These validation and
conversion functions are defined in ckan.logic.validators
,
ckan.logic.converters
and ckan.lib.navl.validators
.
Most validator and converter tests should be unit tests that test the validator
or converter function in isolation, without bringing in other parts of CKAN or
touching the database. This requires using the mock
library to mock
ckan.model
, see Mocking: the mock library.
When testing validators, we often want to make the same assertions in many
tests: assert that the validator didn’t modify the data
dict, assert that
the validator didn’t modify the errors
dict, assert that the validator
raised Invalid
, etc. Decorator functions are defined at the top of
validator test modules like ckan.tests.logic.test_validators
to
make these common asserts easy. To use one of these decorators you have to:
- Define a nested function inside your test method, that simply calls the validator function that you’re trying to test.
- Apply the decorators that you want to this nested function.
- Call the nested function.
Here’s an example of a simple validator test that uses this technique:
def test_user_name_validator_with_non_string_value(self):
'''user_name_validator() should raise Invalid if given a non-string
value.
'''
non_string_values = [
13,
23.7,
100,
1.0j,
None,
True,
False,
('a', 2, False),
[13, None, True],
{'foo': 'bar'},
lambda x: x ** 2,
]
# Mock ckan.model.
mock_model = mock.MagicMock()
# model.User.get(some_user_id) needs to return None for this test.
mock_model.User.get.return_value = None
key = ('name',)
for non_string_value in non_string_values:
data = factories.validator_data_dict()
data[key] = non_string_value
errors = factories.validator_errors_dict()
errors[key] = []
@t.does_not_modify_data_dict
@raises_Invalid
def call_validator(*args, **kwargs):
return validators.user_name_validator(*args, **kwargs)
call_validator(key, data, errors, context={'model': mock_model})
No tests for ckan.logic.schema.py
¶
We don’t write tests for the schemas defined in ckan.logic.schema
.
The validation done by the schemas is instead tested indirectly by the action
function tests. The reason for this is that CKAN actually does validation in
multiple places: some validation is done using schemas, some validation is done
in the action functions themselves, some is done in dictization, and some in
the model. By testing all the different valid and invalid inputs at the action
function level, we catch it all in one place.
Writing ckan.controllers
tests¶
Controller tests probably shouldn’t use mocking.
Todo
Write the tests for one controller, figuring out the best way to write controller tests. Then fill in this guidelines section, using the first set of controller tests as an example.
Some things have been decided already:
All controller methods should have tests
Controller tests should be high-level tests that work by posting simulated HTTP requests to CKAN URLs and testing the response. So the controller tests are also testing CKAN’s templates and rendering - these are CKAN’s front-end tests.
For example, maybe we use a webtests testapp and then use beautiful soup to parse the HTML?
In general the tests for a controller shouldn’t need to be too detailed, because there shouldn’t be a lot of complicated logic and code in controller classes. The logic should be handled in other places such as
ckan.logic
andckan.lib
, where it can be tested easily and also shared with other code.The tests for a controller should:
- Make sure that the template renders without crashing.
- Test that the page contents seem basically correct, or test certain important elements in the page contents (but don’t do too much HTML parsing).
- Test that submitting any forms on the page works without crashing and has the expected side-effects.
- When asserting side-effects after submitting a form, controller tests
should user the
ckan.tests.helpers.call_action()
function. For example after creating a new user by submitting the new user form, a test could call theuser_show()
action function to verify that the user was created with the correct values.
Warning
Some CKAN controllers do contain a lot of complicated logic code. These
controllers should be refactored to move the logic into ckan.logic
or
ckan.lib
where it can be tested easily. Unfortunately in cases like
this it may be necessary to write a lot of controller tests to get this
code’s behavior into a test harness before it can be safely refactored.
Writing ckan.model
tests¶
All model methods should have tests.
Todo
Write the tests for one ckan.model
module, figuring out the best way
to write model tests. Then fill in this guidelines section, using the first
set of model tests as an example.
Writing ckan.lib
tests¶
All lib functions should have tests.
Todo
Write the tests for one ckan.lib
module, figuring out the best way
to write lib tests. Then fill in this guidelines section, using the first
We probably want to make these unit tests rather than high-level tests and
mock out ckan.model
, so the tests are really fast and simple.
Note that some things in lib are particularly important, e.g. the functions
in ckan.lib.helpers
are exported for templates (including
extensions) to use, so all of these functions should really have tests and
docstrings. It’s probably worth focusing on these modules first.
Writing ckan.plugins
tests¶
The plugin interfaces in ckan.plugins.interfaces
are not directly
testable because they don’t contain any code, but:
- Each plugin interface should have an example plugin in
ckan.ckanext
and the example plugin should have its own functional tests. - The tests for the code that calls the plugin interface methods should test that the methods are called correctly.
For example ckan.logic.action.get.package_show()
calls
ckan.plugins.interfaces.IDatasetForm.read()
, so the
package_show()
tests should include tests
that read()
is called at the
right times and with the right parameters.
Everything in ckan.plugins.toolkit
should have tests, because these
functions are part of the API for extensions to use. But
toolkit
imports most of these functions from elsewhere
in CKAN, so the tests should be elsewhere also, in the test modules for the
modules where the functions are defined.
Other than the plugin interfaces and plugins toolkit, any other code in
ckan.plugins
should have tests.
Writing ckan.migration
tests¶
All migration scripts should have tests.
Todo
Write some tests for a migration script, and then use them as an example to fill out this guidelines section.
Writing ckan.ckanext
tests¶
Within extensions, follow the same guidelines as for CKAN core. For example if an extension adds an action function then the action function should have tests, etc.