This document explains some of the changes done for the CKAN Surrey extension, mainly to customize the apparence of CKAN, as well as some configuration changed.
Most of the changes done for the Surrey implementation are self-contained in the ckanext-surrey extension. Working in the extention allow to differentiate the custom code from the CKAN base code and thus allows a better maintenance.
In its current state, the Surrey CKAN extension does the following:
- Support custom CSS stylesheet as well and other resources like images needed in the layout,
- Change the layout and content thanks to custom templates
- Add new fields at the package/dataset level
- Add/modify the licenses supported
The plugin.py file contains the core features of the extention. First, it lists the various directories used by the extension and their usage:
plugins.toolkit.add_template_directory(config, 'templates')
plugins.toolkit.add_public_directory(config, 'public')
plugins.toolkit.add_resource('fanstatic_library', 'ckanext-surrey')
add_template_directorytells CKAN to use a directory (in our casetemplates) to search for custom templates.add_public_directorytells CKAN that a given directory (in our casepublic) will contain files that have to be made public, mainly images.add_resourcedeclares a directory (in our casefanstatic_library) that will contain some resources to be included in the templates, mainly CSS and Javacript files.
Like in any other web framework, templating is the fact of creating/deciding the (HTML) content generated by the framework.
CKAN is delivered with a complete set of template so that the tool is working out of the box (locate in /usr/lib/ckan/default/src/ckan/ckan/templates/). However, each implementation has specific needs like adding new fields, changing some HTML structure or adding custom CSS files. All this needs some changes in the template. However, for maintenant purposes, the original templates (as well as any other original file) must not be modified. If the original templates are modified, those changes will be erased and lost when CKAN is upgraded/patched. So all changes will be located in the extension.
As explained previously, the extension contains some declaration to specify where the custome templates will be:
plugins.implements(plugins.ITemplateHelpers)
def update_config(self, config):
plugins.toolkit.add_template_directory(config, 'templates')
The custom template files in the extension must mimic the original template file structure.
For example, the content of the package modification form is managed by the file package/edit_base.html in the original CKAN templates. So in order to modify this page, one will have to create the same package/edit_base.html in the templates directory of the extension.
CKAN uses a templating system named Jinja2. It allows to define most of the elements as independant blocks that can be accessed/included by parent templates.
In order to have a complete understanding of the Jinja2 templating system, one should browse the Jinga2 documentation. The following section shows how to do some basic template modification (e.g replacing a block of HTML by another one.)
As explained in the CKAN documentation about templating, Jinja2 works by declaring blocks of code. A block is defined as follow:
{% block my_block %}
<p>This will be print something in HTML</p>
{% endblock %}
All the building blocks are declared in the original template files. In order to do our own design, we will override in the extension template files the block we want to change.
One point to know before doing so: In order to be able to use some of the templating features built in CKAN, all the template files must start with
{% ckan_extends %}
Now, let's say we want to add some content in the package page, right after the table containing the metadata. When opening the package/read.html template, we see that the metadata table is managed by the additionnal_info template: package/snippets/additional_info.html.
{% block package_additional_info %}
{% snippet "package/snippets/additional_info.html", pkg_dict=pkg %}
{% endblock %}
We have 2 options here: either we want to completely replace the additional info section or we want to add some content.
In order to replace the additionnal information, one just have to replace the content of the package_additional_info block in the read.html file (if the content is simple) or replace the content of the snippet package/snippets/additional_info.html
If one replaces the content of the block, the read.html will end up like that:
{% block package_additional_info %}
<p>This will replace the additionnal info</p>
{% endblock %}
If one changes the snippet file content, the package/snippets/additional_info.html could look like this:
{% ckan_extends %}
<!-- Simplified version of the table with only the 'extra' fileds-->
<h3>Custom info</h3>
<table>
{% for extra in h.sorted_extras(pkg_dict.extras) %}
{% set key, value = extra %}
<tr>
<th>{{ key }}</th>
<td>{{ value }}</td>
</tr>
{% endfor %}
</table>
In many cases, one will want to add some content of an existing block. Instead of redefining the complete block just to add one or two item, CKAN/Jinja2 allows to call insert the original template content and add some content after of before thanks to the super() function.
Example: the following example will add a new line at the beginning of the metadata table in the
{% ckan_extends %}
{% block package_additional_info %}
<tr>
<td>A field</td>
<td>Some content</td>
</td>
{{ super() }}
{% endblock %}
In order to change the overall design of CKAN, the most straigh-forward option is to change the CSS. Directly changing the CKAN core CSS will cretes the maintenance issues: when installing a new patch/version of CKAN, the changes will be erased by the new version. The solution provided by CKAN is to create a custom CSS file in the extension that will override the CKAN CSS directives. This is a 2 step process.
As explained previously, custome CSS files are managed as resources, so they have to be created in the resources directory. In our case, it will be fanstatic_library/css/surrey.css.
The file is a regular CSS file and can contain any declaration.
In order to use the custom CSS file, it should be declared in the template to appear in the generated HTML files.
For this we use the overall templating process: the headers where CSS files should be declared are located in the base.html file in the styles block, so in our extension we create a templates/base.html file and add the following content:
`` {% ckan_extends %}
{% block styles %} {{ super() }} {% resource 'ckanext-surrey/css/surrey.css' %} {% endblock %} ``
In some cases, it could be useful to add in the template some custom images, for exemple on the landing page. As explained previously, CKAN provides a mechanism to publicly expose some files of the extension. In our case, in the public directory of the extension. If we want to add an image on the landing page, here is how to do it.
Copy the image in the public directory or in a subdirectory, for example public/images/homepage-graphic.jpg. Now, the image become available
Now we have to include the image in the template. The image become accessible thanks to the url_for_static function in the templates. The image in the landing page is controlled in the home/index.html template by the home_image_content block. So in our extension we create a file templates/home/index.html and add the following content:
{% block home_image_content %}
<img src="{% url_for_static('/images/homepage-graphic.jpg') %}" width="479" height="298" />
{% endblock %}
By default CKAN allows to add some custom fields at the dataset/package level. However, since the name of the field is free text, it increases the risk of inconsistency of the data and does not allow to have any control on the data. CKAN allows to add some pre-defined custom fields. The CKAN documentation explains in detail how to manage extra fields. The following section just explain how to add new fields in the existing extension.
The fields have to be declared in the plugin.py file:
- In the
_modify_package_schemafunction
schema.update({
'coordinate_system': [tk.get_validator('ignore_missing'),
tk.get_converter('convert_to_extras')]
})
- In the
show_package_schemafunction
schema.update({
'coordinate_system': [tk.get_converter('convert_from_extras'),
tk.get_validator('ignore_missing')]
})
These declaration will let CKAN that the package schema (e.g the data structure) contains a new field.
In order to be able to fill the field, it has to be added in the dataset creation/modification form. This form is controlled in the templates/package/snippets/package_metadata_fields.html file by the package_metadata_fields_custom block. We also have to remove the existing free text custom field, managed by the custom_fields block, since CKAN is not able to manage both. We create the file templates/package/snippets/package_metadata_fields.html with the following content:
{% ckan_extends %}
{% block custom_fields %}
{% endblock %}
{% block package_metadata_fields_custom %}
{{ form.input('coordinate_system', label=_('Coordinate system'), id='field-coordinate_system', placeholder=_('WGS84, etc.'), value=data.coordinate_system, error=errors.coordinate_system, classes=['control-medium']) }}
{{ super() }}
{% endblock %}
Now, users who visit the site must also see the new fields. This is part is controlled by the templates/package/snippets/additional_info.html template. In our case, in order to get full control of the content and order of the fields, we completely override the template file. In order to add a new field, the follwing snippet has be used:
{% if pkg_dict.coordinate_system %}
<tr>
<th scope="row" class="dataset-label">{{ _("Coordinate system") }}</th>
<td class="dataset-details">{{ pkg_dict.coordinate_system }}</td>
</tr>
{% endif %}
CKAN is bundled with a set of generic licenses, however most of the implementation needs customs licenses. In this case, a custom license file must be defined in the configutation file production.ini (or development.ini):
licenses_group_url = file:///path/to/licence/licences.json
The licence file is an array of licence items as defined by opendefinition:
{
"domain_content": false,
"domain_data": true,
"domain_software": false,
"family": "",
"id": "OGL-Surrey",
"is_okd_compliant": true,
"is_osi_compliant": false,
"maintainer": "City of Surrey",
"status": "active",
"title": "Open Government License - City of Surrey",
"url": "http://www.surrey.ca/files/Open_Government_License_-_City_of_Surrey_v1.pdf"
}
The ckan-page extension allows to create static files with not interaction like the FAQ page. But in order to support dynamic feature like forms, one need to integrate the new pages to the existing page management system implemented by CKAN. This can be done with the iRoutes plugin.
In the surrey extension, the pages follow (that explain how to create an account), suggest (to suggest a new dataset) and contact are managed using this way of working.
In plugin.py, the declaration of these new pages in managed by class SurreyExtraPagesPlugin. All the code to support the new pages is in controller.py and the templates are in the corresponding template directories.
Each instance has its own configuration file (.ini). When setting up a new instance, the following items should be filled (on top of those required in the regular installation)
site_url (some links rely on this value)
ckan.site_url = http://....
Authorization parameters should be set to the following to allow any user to create an account but without any right
ckan.auth.anon_create_dataset = false
ckan.auth.create_unowned_dataset = false
ckan.auth.create_dataset_if_not_in_organization = false
ckan.auth.user_create_groups = false
ckan.auth.user_create_organizations = false
ckan.auth.user_delete_groups = false
ckan.auth.user_delete_organizations = false
ckan.auth.create_user_via_api = false
ckan.auth.create_user_via_web = true
ckan.auth.roles_that_cascade_to_sub_groups = admin
The following items controls some public items of the site:
ckan.site_title = my title
ckan.site_logo = /base/images/ckan-logo.png
ckan.site_description = This is a description
ckan.favicon = /images/surrey.ico
The Atom feeds are published in the meta data, so the feed parameters should be filled:
ckan.feeds.authority_name = City of Surrey
ckan.feeds.date = 2014-03-01
ckan.feeds.author_name = City of Surrey
ckan.feeds.author_link = http://surrey.ca/
In order to upload large files, it could be useful to change the max size parameter (by default set to 10MB. Note that the web server -apache2, nginx, etc.- might also have its own limitation)
ckan.max_resource_size = 200
Enable email notifications and setup the email configuration
ckan.activity_streams_email_notifications = true
email_to = [email protected]
error_email_from = [email protected]
smtp.server = smtp.test.ca
smtp.mail_from = [email protected]
To disable preview for some formats, like json, one has to remove the format from ckan.preview.loadable and instruct the textpreview plugin not to treat this format (see http://docs.ckan.org/en/847-new-theming-docs/data-viewer.html)
ckan.preview.loadable = html htm rdf+xml owl+xml xml n3 n-triples turtle plain atom csv tsv rss txt
ckan.preview.json_formats = -
ckan.preview.jsonp_formats = -