Setting up your IDE¶

This section outlines the process for configuring your IDE (Integrated Development Environment) for development.

🚩 Make sure you have gone through the Cloning Section before following these notes.

VS Code Setup¶

Open the project in VSCode (1️⃣, 2️⃣) by navigating the the place on your file system where you checked out the code in the pre-requisites step above (3️⃣).

Accept the 'trust authors' prompt

Copying the .env¶

Copy the template.env to .env Edit the .env file and change the

DJANGO_SETTINGS_MODULE=core.settings.prod

to

DJANGO_SETTINGS_MODULE=core.settings.dev

Override Docker Configs¶

We are going to copy the docker overrides template to a local file that will not be under version control.

Rename the file to docker-compose.override.yml

Initially you will not need to change anything in this file, though you may want to take a look through the various configurations provided here if you want to tweak your local setup.

Now that you have your IDE set up, we can move on to building the project.

Using pycharm¶

This section is for using pycharm.

Requirements:

Pycharm
Finished Setting up the project

Setup interpreter¶

Go to file -> setting -> Project -> Project Interpreter -> click cog -> add
Go to ssh interpreter -> Fill the form like below
Click next and fill docker as password
Click next and change interpreter like below and click finish
After finish, it will show all package like below.
In current page, click path mappings, click + button and put local path to where the project (django-project folder) and remote path is like below. and click oK.

Now the interpreter is done. When we restart the machine, we need to do make up to run the project.

Setup run configuration¶

After the interpreter is done, we need configuration to run the project in development mode.

Click "Add configuration" like in the cursor in the image below. (top-right)
There will be a popup, and click +, then click django server like below
It will show the form and fill like below.
Don't click the OK yet, but click Environment Variables and add environments like below 9by clicking + button).
After that, click OK.
Now we need to run the server by clicking go button in below image.
When we click the go button, pycharm will run a process until like image below.
Now it is done. We can access the development server in http://localhost:2000/

This development mode is DEBUG mode, and also whenever we change the code, the site will also change in runtime.

For more information how to set up on pycharm, please visit Using a Docker Compose-Based Python Interpreter in PyCharm

Quick Installation Guide¶

This project is a ckan extension, it can be installed standalone. To deploy this project we use docker, so you need to have docker running on the host.

NOTE: Docker is needed to continue with the quick installation guide. Ensure you have docker installed on your system before continuing.

Build docker images

cd SAEOSS-Portal/docker
./build.sh

Start up project

./compose.py --compose-file docker-compose.yml --compose-file docker-compose.dev.yml up

Run down the project

./compose.py --compose-file docker-compose.yml --compose-file docker-compose.dev.yml down

After starting up, the project is available on your local host at http://localhost:5000

Operations¶

Rebuild solr index¶

# check if there are any datasets that are not indexed
ckan search-index check

# re-index
docker exec -it saeoss_ckan_web_1 poetry run ckan search-index rebuild

Operate harvesters¶

You may use the various ckan harvester <command> commands to operate existing harvesters.

Create a job:

docker exec -t saeoss-ckan_harvesting-runner poetry run ckan harvester job <source-id>

Send notifications by email¶

This needs to be run periodically (once per hour is likely enough).

docker exec -it saeoss_ckan_web_1 ckan saeoss send-email-notifications

Additionally, in order for notifications to work, there is some configuration:

The CKAN settings must have ckan.activity_streams_email_notifications = true
The CKAN settings must have the relevant email configuration (likely being passed as environment variables)
Each user must manually choose to receive notification e-mails - This is done in the user's profile
Each user must manually follow those entities (datasets, organizations, etc) that it finds interesting enough in order to be notified of changes via email

Refresh pycsw materialized view¶

This needs to be run periodically (once per hour is likely enough).

docker exec -t ckan dalrrd-emc-dcpr pycsw refresh-materialized-view

Create sysadmin user¶

After having initialized the database you can now create the first CKAN sysadmin user.

docker exec -ti saeoss_ckan_web_1 poetry run ckan sysadmin add admin

Answer the prompts in order to provide the details for this new user. After its successful creation you can login to the CKAN site with the admin user.

Initialize CKAN database¶

The first time you launch it you will need to set up the ckan database (since the ckan image's entrypoint explicitly does not take care of this, as mentioned above). Run the following command:

docker exec -ti saeoss_ckan_web_1 poetry run ckan db init

Afterwards, proceed to run any migrations required by the ckanext-dlarrd-emc-dcpr extension

docker exec -ti saeoss_ckan_web_1 poetry run ckan db upgrade --plugin dalrrd_emc_dcpr

Now you should be able to go to http://localhost:5000 and see the ckan landing page. If not, you may need to reload the ckan web app after performing the DB initialization step. This can done by sending the HUP signal to the gunicorn application server (which is running our ckan flask app):

docker exec -ti saeoss_ckan_web_1 bash -c 'kill -HUP 1'

Generate pycsw DB view¶

In order to be able to serve the system's datasets through various OGC standards, create a DB materialized view in order to integrate with pycsw:

docker exec -ti saeoss_ckan_web_1 poetry run ckan dalrrd-emc-dcpr pycsw create-materialized-view

Ingest a collection of metadata from a given directory (currently CBERS xml files)¶

create datasets (metadata records) from files stored in a directory

docker exec -ti saeoss_ckan_web_1 poetry run ckan saeoss ingest cbers --source-path <path> --user <username>

Perform STAC Fetch and create datasets from STAC endpoint¶

create datasets (metadata records) from stac endpoint

docker exec -ti saeoss_ckan_web_1 poetry run ckan saeoss stac create-stac-dataset --url <url> --user <username> --max <max_number_of_records>

Production¶

Section for juanique and zakki to complete