Application Data Preparation¶

This guide covers how to prepare the data files that Decision Theatre consumes.

Vector Map Tiles (MBTiles)¶

Source Data¶

The map data originates from a GeoPackage file (UoW_layers.gpkg) containing:

Conversion Pipeline¶

The script resources/mbtiles/gpkg_to_mbtiles.sh automates the full conversion:

cd resources/mbtiles
./gpkg_to_mbtiles.sh UoW_layers.gpkg

The script stages output in resources/mbtiles/ during processing, then moves the final africa.mbtiles to data/mbtiles/ on completion.

Pipeline stages:

1. Layer discovery -- queries gpkg_contents for all feature layers
2. Geometry validation -- checks for NULL geometries; optionally repairs with ogr2ogr -makevalid
3. GeoJSONSeq export -- converts each layer to newline-delimited GeoJSON using ogr2ogr
4. Per-layer tile generation -- runs tippecanoe for each layer with configured zoom ranges
5. Merge -- combines all per-layer MBTiles into a single file using tile-join

Zoom Configuration¶

Each layer has configured minimum and maximum zoom levels:

Layers not in this table default to zoom 6--15.

Required Tools¶

• ogr2ogr (GDAL)
• tippecanoe
• sqlite3

All are available in the Nix dev shell (nix develop).

Map Style¶

The MapBox GL Style JSON at data/mbtiles/style.json defines how each layer is rendered (colours, line widths, label placement). Edit this file to change the map's visual appearance.

Fetching Data from Google Drive¶

The script scripts/fetch-data.sh downloads all files from a shared Google Drive folder (including subfolders) directly into the data/ directory, preserving the folder structure. Run this before dt geopackage to pull the latest source files in one step.

Installation: rclone¶

The script requires rclone, a command-line tool for cloud storage. Install it once on your machine:

sudo curl https://rclone.org/install.sh | sudo bash

brew install rclone

One-time Google Drive configuration¶

rclone needs a named remote that points to your Google Drive account. Create one called gdrive by running:

rclone config

Follow the interactive prompts:

1. Press n for New remote.
2. Name it gdrive.
3. Choose Google Drive 24 as the storage type.
4. Leave the client ID and secret blank (uses rclone's defaults).
5. Choose scope drive.readonly 2 if you only need to download, or drive for full access.
6. Follow the browser OAuth flow to authorise rclone with your Google account.
7. Accept the default for all remaining options and confirm.

Verify the remote works:

rclone lsd gdrive:

Finding the folder ID¶

Open the Google Drive folder in your browser. The folder ID is the last segment of the URL:

https://drive.google.com/drive/folders/1ABCdef_ghiJKLmnopQRSTuvwXYZ
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                        this is the folder ID

The folder must be shared with the Google account you authenticated with rclone, or set to Anyone with the link.

Usage¶

# Using Make (recommended)
dt fetch-data FOLDER=<folder-id-or-url>

# Or directly
./scripts/fetch-data.sh <folder-id-or-url> [data-dir]

Both a bare folder ID and a full URL are accepted:

# Bare folder ID
dt fetch-data FOLDER=1ABCdef_ghiJKLmnopQRSTuvwXYZ

# Full shareable URL
dt fetch-data FOLDER="https://drive.google.com/drive/folders/1ABCdef_ghiJKLmnopQRSTuvwXYZ"

# Custom destination directory
./scripts/fetch-data.sh 1ABCdef_ghiJKLmnopQRSTuvwXYZ /path/to/data

What the script does¶

1. Extracts the folder ID from the argument (whether a bare ID or URL).
2. Checks that rclone is installed and the gdrive remote exists.
3. Copies all files from the Drive folder and any subfolders into data/ using rclone copy, preserving the subfolder structure.
4. Skips files that are already up-to-date (same size and modification time).
5. Retries automatically on transient network errors.
6. Prints a summary of all files in data/ with their sizes when complete.

Using a different remote name¶

If your rclone Google Drive remote is not called gdrive, set the RCLONE_REMOTE environment variable:

RCLONE_REMOTE=my-drive ./scripts/fetch-data.sh 1ABCdef_ghiJKLmnopQRSTuvwXYZ

Full workflow¶

# 1. Download the CSV source files from Google Drive
dt fetch-data FOLDER=1ABCdef_ghiJKLmnopQRSTuvwXYZ

# 2. Build the GeoPackage datapack
dt geopackage

# 3. Launch the application
dt app

Scenario Data (GeoPackage Datapack)¶

The application uses a GeoPackage file (datapack.gpkg) containing catchment geometries and scenario data. This is built from raw input files using the scripts/build-geopackage.sh script.

Input Files¶

Place the following files in the data/ directory:

Geometry¶

Scenario CSVs¶

All scenario CSVs must have a catchID column that cross-references HYBAS_ID in catchments.gpkg. All other columns are per-catchment indicator values stored as REAL in the database (NA becomes NULL).

Metadata¶

Ecological Lookup Tables¶

These files support the cascading recalculation workflow triggered when a user adjusts a target indicator. All three are optional — if absent the application falls back to proportional scaling.

See the Datapack Format page for the full column specifications of each lookup file.

Building the Datapack¶

# Using Make (recommended)
dt geopackage

# Or directly
./scripts/build-geopackage.sh ./data

This script performs the following steps:

1. Base setup — Copies catchments.gpkg as the base for the output file
2. CSV import — Imports scenario CSVs as raw tables using ogr2ogr
3. Type conversion — Converts data columns to REAL type, converting NA strings to NULL
4. Column normalization — Normalizes column names across tables (replaces dashes, spaces with dots)
5. Indexing — Creates integer indexes on catchment IDs for fast joins
6. GeoJSON precomputation — Converts geometries to GeoJSON for fast API serving
7. Domain min/max — Computes global min/max for each attribute across both scenarios

Output GeoPackage Schema¶

The output datapack.gpkg contains these tables:

Scenario Tables Schema¶

Both scenario_current and scenario_reference tables have:

• catchment_id (TEXT) — The catchment identifier (normalized from catchID)
• catchment_id_int (INTEGER) — Integer version for indexed joins
• All attribute columns as REAL type (NULL for missing/NA values)

Domain Tables Schema¶

The domain_minima and domain_maxima tables each contain one row with:

• All attribute columns from the scenario tables
• Each column contains the global min (or max) value computed across both scenarios
• These are used for consistent color scaling across scenario comparisons

Column Normalization¶

The build script normalizes column names to ensure consistency between tables:

• catchID → catchment_id
• Dashes (-), spaces, apostrophes → dots (.)
• Multiple consecutive dots → single dot
• Duplicate ID columns (e.g., sp_current.catchID) are dropped

Required Tools¶

• ogr2ogr (GDAL)
• sqlite3
• python3

All are available in the Nix dev shell (nix develop).