# Technical documentation for data portal ## The Data portal The data portal consists of two services: [WordPress](https://wordpress.org/) for content management and [CKAN](https://ckan.org/) publishing data. Services are built in such way that the front page, news, blogs and articles are made with WordPress and URLs starting with /data/ are made with CKAN. The demo environment is available at with login to each service with its own account. WordPress login is available at and CKAN is at . Access to services can be asked from the developer team. ## Setting up development environment ### Requirements: * Virtualbox (tested with 6.1.12) * Vagrant (tested with 2.2.9) The development environment can be built on any machine that has required applications installed. Setting up development environment is done automatically when following commands are executed: ```bash git clone https://github.com/6aika/sixodp.git cd sixodp git submodule update --init --recursive npm install npm run build vagrant up ``` {% hint style="info" %} Setting up the environment can take half an hour. {% endhint %} Once ansible has installed everything inside the virtual machine, the environment is available at . The default password in development environment are admin / admin in WordPress and admin / adminadmin in CKAN. # General The portal consists of two services: WordPress and CKAN, which are installed side by side behind NGINX server. Application architecture is depicted in the following figure.
WordPress is executed with [PHP FPM](https://www.php.net/manual/en/install.fpm.php) and CKAN with [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/). Additional to these, the service has multiple databases: WordPress requires MySQL and CKAN requires PostgreSQL with additionally required Redis-database and Solr search index. # AWS The environments are hosted on AWS where they are built using [Cloudformation templates](https://aws.amazon.com/cloudformation/) and CDK code. Each environment consists multiple stack, templates for these are available in and . ### Database stack The database stack contains components with irreplaceable data. If the stack is destroyed, the data is lost. The stack contains the following resources: * Dataset bucket (S3) * Policies of dataset bucket The stack takes the following parameters: * Environment name (pre-defined list) The stack outputs following parameters: * Dataset bucket ### CDK based stacks Rest of the stacks are built using cdk and contains following components: * Route53 zone for DNS * WAF firefall for managing access to VPC network * Load balancer to manage access to multiple web server * Nat gateway for access to the internet * 1-2 web servers depending on configuration, these host wordpress and ckan * Background server for backgound processes like crontab, solr and redis * 2 databases: Mysql for wordpress and postgresql for ckan * Image files are hosted in EFS and dataset files in S3 The stacks are also depicted on the following figure: ![](/files/Sg8U0wUwsVlY5HoWYoAY) {% hint style="info" %} Additionally to the stacks, each application stack requires access to secrets S3 bucket which is configured within the stack template. {% endhint %} # WordPress WordPress is used with custom built theme available in . Following list on plugins are installed with the theme: * [Advanced Custom Fields](https://www.advancedcustomfields.com/), adds new fields to posts. * [Polylang](https://polylang.pro/), handles localization of content to different languages. * [ReCaptcha Integration for WordPress](https://wordpress.org/plugins/wp-recaptcha-integration/), adds recaptcha checks to anonymous data and showcase requests. * [WP User Avatars](https://fi.wordpress.org/plugins/wp-user-avatars/), adds avatar images on articles. * [Classic Editor](https://fi.wordpress.org/plugins/classic-editor/), adds the old wysiwyg editor. * [Yoast SEO](https://fi.wordpress.org/plugins/wordpress-seo/), adds search engine optimization features. * [Wordfence](https://wordpress.org/plugins/wordfence/), securing WordPress. * [WP Mail SMTP](https://wordpress.org/plugins/wp-mail-smtp/), configuring WP to send emails via service ### Installation WordPress is installed by running ansible (). Installation is done with the help of WP CLI. # CKAN CKAN is installed from source via ansible. Current version is 2.9.8. The installed ckan is modified with bugfixes or other needs not yet present in the current release. These are available as patch files in . When CKAN is upgraded, installation of these patches most likely will fail. The developer needs to go through them and port necessary ones to new CKAN. ## Extensions Most of the actual customization is done via ckan extensions. Some of them are implemented specifically for 6aika data portal, some are built by other with various degrees of modifications done in 6aika data portal. Below are lists of used extensions divided by responsible party of the extension. ### Maintained by 6aika developers: * [Sixodp](https://github.com/6aika/sixodp/tree/master/ckanext/ckanext-sixodp), Most of the custom theme, routing and schema changes. * [Sixodp\_showcase](https://github.com/6aika/sixodp/tree/master/ckanext/ckanext-sixodp_showcase), extends showcase extension with customized meta model schema and UI. * [Sixodp\_showcasesubmit](https://github.com/6aika/sixodp/tree/master/ckanext/ckanext-sixodp_showcasesubmit), provides anonymous submitting of showcases to 6aika showcase schema. * [Collection](https://github.com/6aika/ckanext-collection), creates a duplicate feature of groups, so that there are similar collections. Groups are used as categories. * [Reminder](https://github.com/6aika/ckanext-reminder), enables sending notifications from dataset updates to anonymous subscribers and reminders to dataset maintainers when they have to update their datasets. * [Datasetcopy](https://github.com/6aika/ckanext-datasetcopy), enables creating new datasets from existing ones by filling the form with existing values. * [Statistics](https://github.com/6aika/sixodp/tree/master/ckanext/ckanext-statistics), creates visualization on the browser from the ckan api. The visualizations are implemented with d3.js and probably has scaling issues when amount of datasets or organizations grow. * [Editor](https://github.com/6aika/ckanext-editor), provides user interface to modify multiple datasets at the same time with the same values. * [Datasubmitter](https://github.com/6aika/sixodp/tree/master/ckanext/ckanext-datasubmitter), similar to showcase submit, enables submitting datasets anonymously. ### Maintained by others but might have modification in 6aika dataportal * [Showcase](https://github.com/6aika/ckanext-showcase), provides showcasing appilications. * [Report](https://github.com/6aika/ckanext-report), provides reporting infrastructure. * [QA](https://github.com/6aika/ckanext-qa), provides quality analysis for the uploaded data. * [Archiver](https://github.com/6aika/ckanext-archiver), required by QA, mostly used for checking data link validity. * [Scheming](https://github.com/6aika/ckanext-scheming), enables customizing dataset schemas via json files. * [Fluent](https://github.com/ckan/ckanext-fluent), adds multilingual capabilities to scheming. * [Hierarchy](https://github.com/6aika/ckanext-hierarchy), adds hierarchies to organizations. * [Cloudstorage](https://github.com/6aika/ckanext-cloudstorage), enables storing dataset data to AWS S3. * [Matomo](https://github.com/6aika/ckanext-matomo), provides analytics and reporting of usage on dataset pages # Transifex Translations are managed in [Transifex](https://www.transifex.com/6aika-dataportal/sixodp/). Each CKAN extensions has its own resource in Transifex and in addition, wordpress has a single resource which covers all of the wordpress translations. Translations from codebase to Transifex and back are handled via .po files which are uploaded and downloaded with the help of [Transifex client](https://docs.transifex.com/client/introduction). ## CKAN Translations for CKAN consist of translations of extensions and translations of the CKAN [itself](https://www.transifex.com/okfn/ckan/). If the translation shown on UI does not exist in projects transifex resources, it comes from CKAN core translations. If someone wishes to change the translation, he needs to add translated key to the extension, where it overrides the core translation. If multiple extensions have the same translated key in their .po files, all need to have the same translation. ### Generating translation files Translations in CKAN extensions mostly follow the [official documentation](https://docs.ckan.org/en/2.8/extensions/translating-extensions.html). In summary, following commands should cover most of the cases. ``` // Activate virtual environment . /usr/lib/ckan/default/bin/activate // Run rest of commands for single extension cd /vagrant/ckanext/ckanext-sixodp/ // Extract messages for extension python setup.py extract_messages ``` ### Translating in Transifex The newly generated translation files can be uploaded and downloaded to transifex with the following commands: ``` // Upload to transifex tx push -s // After translated in transifex, download the translated files tx pull ``` ### Compiling translations Translation files can be compiled by running following commands: ``` // Activate virtual environment . /usr/lib/ckan/default/bin/activate // Run rest of commands for single extension cd /vagrant/ckanext/ckanext-sixodp/ // Compile translation files python setup.py compile_catalog -f ``` ## WordPress Generating translations for WordPress is done manually with [makepot.php](https://github.com/6aika/sixodp/blob/master/sixodp/tools/makepot.php). Translation tools in wp-cli cannot be used as the version is use is too old for them and it cannot be updated until automating polylang is replaced with something as polylang-cli was [abandoned](https://github.com/diggy/polylang-cli/issues/118). makepot.php assumes specific folder structure and following commands should generate translation files. ``` // Run in vagrant home /home/vagrant cp -r /vagrant/sixodp/ sixodp cp -r /opt/wordpress/ src cd sixodp/tools/ php makepot.php wp-theme ../ /vagrant/sixodp/i18n/sixodp.pot ``` After being pushed and pulled from Transifex, following commands will compile the translations files: ``` msgfmt /vagrant/sixodp/i18n/en_GB/LC_MESSAGES/sixodp.po -o /vagrant/sixodp/i18n/en_GB/LC_MESSAGES/sixodp-en_GB.mo msgfmt /vagrant/sixodp/i18n/fi/LC_MESSAGES/sixodp.po -o /vagrant/sixodp/i18n/fi/LC_MESSAGES/sixodp-fi.mo msgfmt /vagrant/sixodp/i18n/sv/LC_MESSAGES/sixodp.po -o /vagrant/sixodp/i18n/sv/LC_MESSAGES/sixodp-sv.mo ``` ## Other notes There are some tools which may or may not work in for CKAN and in for WordPress. # CKAN Doing a CKAN version upgrade starts by changing the version number [here](https://github.com/6aika/sixodp/blob/master/ansible/roles/ckan-install/vars/main.yml). Next the developer should run the ansible playbook and check if it will succeed or not. Those [patch files](https://github.com/6aika/sixodp/tree/master/ansible/roles/ckan-install-patches/files/patches) that doesn't change anything should be removed and those that fail, should be ported to newer CKAN version. If the playbook succeeds, the following list of features should be checked that they still work. There are no automated tests. * Creating, modifying and deleting user * Requesting password reset for user * Creating, modifying and deleting dataset * Setting a reminder date to dataset and checking that the reminder is actually sent * Subscribing to dataset updates anonymously and checking that notifications are sent * Creating, modifying and deleting organization * Creating, modifying and deleting category (group in ckan terms) * Creating, modifying and deleting showcase * Creating, modifying and deleting collection * Submitting dataset anonymously * Submitting showcase anonymously * Modifying different fields in dataset via mass editor * Fetching google analytics data via paster cli command # WordPress WordPress and its plugins should be upgraded automatically to latest version every time ansible is executed. Patch updates should be installed automatically by WordPress itself. If the upgrade fails for any reason, usually it is a permission issue on the server. After the upgrade, the following list of features should be checked. There are no automated tests. * Creating, editing and deleting page. * Creating, editing and deleting post. * Localization. * Submitting showcase idea. * Submitting dataset request. # Continuous integration Github Actions run unit tests using pytest, these produce coverage reports in , unit tests are mainly run on only sixodp ckan extension. # Installing updates Code and other updates can installed to instance via running following command on shell prompt on the instance itself: ``` sixodp-update ``` The script will fetch newest updates from github master branch and run ansible to install them. If there are no updates in github or the script has been previosly errored for some reason, ansible can be executed with following commands manually: ``` sudo su cd /root/sixodp/ansible // this directory might be different depensing on the instance ansible-playbook -i inventories/(instance_id) deploy-transitional.yml ``` Ansible will update OS packages and WordPress, updating CKAN will require manual work.