Skip to content

Commit

Permalink
Merge pull request #4786 from thoughtful-explorer/doc-edits
Browse files Browse the repository at this point in the history
Formatting, spelling, and minor editorial corrections
  • Loading branch information
pnorman authored Mar 9, 2023
2 parents 6591d7b + 64ea80c commit dae7527
Show file tree
Hide file tree
Showing 2 changed files with 36 additions and 30 deletions.
46 changes: 26 additions & 20 deletions DOCKER.md
Original file line number Diff line number Diff line change
@@ -1,49 +1,55 @@
# Running OpenStreetMap Carto with Docker

[Docker](https://www.docker.com/) is a virtualized environment running a [_Docker demon_](https://docs.docker.com/get-started/overview/), in which you can run software without altering your host system permanently. The software components run in _containers_ that are easy to setup and tear down individually. The Docker demon can use operating-system-level virtualization (Linux, Windows) or a virtual machine (macOS, Windows).
[Docker](https://www.docker.com/) is a virtualized environment running a [_Docker daemon_](https://docs.docker.com/get-started/overview/), in which you can run software without altering your host system permanently. The software components run in _containers_ that are easy to setup and tear down individually. The Docker demon can use operating-system-level virtualization (Linux, Windows) or a virtual machine (macOS, Windows).

This allows to set up a development environment for OpenStreetMap Carto easily. Specifically, this environment consists of a
This enables easily setting up a development environment for OpenStreetMap Carto. Specifically, this environment consists of a
PostgreSQL database to store the OpenStreetMap data and [Kosmtik](https://github.com/kosmtik/kosmtik) for previewing the style.

## Prerequisites

Docker is available for Linux, macOS and Windows. [Install](https://www.docker.com/products/docker-desktop/) the software packaged for your host system in order
to be able to run Docker containers. You also need Docker Compose, which should be available once you installed
Docker itself. Otherwise you need to [install Docker Compose manually](https://docs.docker.com/compose/install/).
* Docker is available for Linux, macOS and Windows. [Install](https://www.docker.com/products/docker-desktop/) the software packaged for your host system in order
to be able to run Docker containers. You also need Docker Compose, which should be available once you install
Docker itself. Otherwise, you need to [install Docker Compose manually](https://docs.docker.com/compose/install/).

You need sufficient disk space of _several Gigabytes_. Docker creates a disk image for its virtual machine that holds the virtualised operating system and the containers. The format (Docker.raw, Docker.qcow2, \*.vhdx, etc.) depends on the host system. It can be a sparse file allocating large amounts of disk space, but still the physical size starts with 2-3 GB for the virtual OS and grows to 6-7 GB when filled with the containers needed for the database, Kosmtik, and a small OSM region. Further 1-2 GB are needed for shape files in the openstreetmap-carto/data repository.
* You need sufficient disk space of _several Gigabytes_.
* Docker creates a disk image for its virtual machine that holds the virtualised operating system and the containers.
* The format (Docker.raw, Docker.qcow2, \*.vhdx, etc.) depends on the host system. It can be a sparse file allocating large amounts of disk space, but still the physical size starts with 2-3 GB for the virtual OS and grows to 6-7 GB when filled with the containers needed for the database, Kosmtik, and a chosen small region of OSM data.
* An additional 1-2 GB are needed for shapefiles to be downloaded and stored in the openstreetmap-carto/data repository.

## Quick start

If you are eager to get started here is an overview over the necessary steps.
Read on below to get the details.
If you are eager to get started, here is an overview over the necessary steps:

* `git clone https://github.com/gravitystorm/openstreetmap-carto.git` to clone openstreetmap-carto repository into a directory on your host system
* download OpenStreetMap data in osm.pbf format to a file `data.osm.pbf` and place it within the openstreetmap-carto directory (for example some small area from [Geofabrik](https://download.geofabrik.de/))
* If necessary, `sudo service postgresql stop` to make sure you don't have currently running a native PostgreSQL server which would conflict with Docker's PostgreSQL server.
* `docker-compose up import` to import the data (only necessary the first time or when you change the data file). Additionally you can set import options through [environment variables](#Importing-data). More on that [later](#Hands-on-approach)
* Download OpenStreetMap data in osm.pbf format to a file `data.osm.pbf` and place it within the openstreetmap-carto directory (for example some small area from [Geofabrik](https://download.geofabrik.de/))
* If necessary, `sudo service postgresql stop` to make sure you don't have a currently-running native PostgreSQL server which would conflict with Docker's PostgreSQL server.
* `docker-compose up import` to import the data (only necessary the first time or when you change the data file). Additionally, you can set import options through [environment variables](#Importing-data). More on that [later](#Hands-on-approach)
* `docker-compose up kosmtik` to run the style preview application
* browse to [http://127.0.0.1:6789](http://127.0.0.1:6789) to view the output of Kosmtik
* Browse to [http://127.0.0.1:6789](http://127.0.0.1:6789) to view the output of Kosmtik
* Ctrl+C to stop the style preview application
* `docker-compose stop db` to stop the database container

## Repositories

Instructions above will clone main OpenStreetMap Carto repository. To test your own changes you should [fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) gravitystorm/openstreetmap-carto repository and [clone your fork](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository).
The instructions above will clone the main OpenStreetMap Carto repository. To test your own changes, you should [fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) gravitystorm/openstreetmap-carto repository and [clone your fork](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository).

This OpenStreetMap Carto repository needs to be a directory that is shared between your host system and the Docker virtual machine. Home directories are shared by default; if your repository is in another place you need to add this to the Docker sharing list (e.g. macOS: Docker Preferences > File Sharing; Windows: Docker Settings > Shared Drives).

## Importing data

OpenStreetMap Carto needs a database populated with rendering data to work. You first need a data file to import.
It's probably easiest to grab an PBF of OSM data from [Geofabrik](https://download.geofabrik.de/).
For testing purposes, it's probably easiest to grab a PBF export of OSM data from [Geofabrik](https://download.geofabrik.de
Once you have that file put it into the openstreetmap-carto directory and run `docker-compose up import` in the openstreetmap-carto directory.
This starts the PostgreSQL container (downloads it if it not exists) and starts a container that runs [osm2pgsql](https://github.com/openstreetmap/osm2pgsql) to import the data. The container is built the first time you run that command if it not exists.
At startup of the container the script `scripts/docker-startup.sh` is invoked which prepares the database and itself starts osm2pgsql for importing the data. Then the `scripts/get-external-data.py` is called to download and import needed shapefiles.

### Supplying command line options as environment variables

**osm2pgsql** has a few [command line options](https://manpages.debian.org/testing/osm2pgsql/osm2pgsql.1.en.html) and the import by default uses a RAM cache of 512 MB, 1 worker and expects the import file to be named `data.osm.pbf`. If you want to customize any of these parameters you have to set the environment variables `OSM2PGSQL_CACHE` (e.g. `export OSM2PGSQL_CACHE=1024` on Linux to set the cache to 1 GB) for the RAM cache (the value depends on the amount of RAM you have available, the more you can use here the faster the import may be), `OSM2PGSQL_NUMPROC` for the number of workers (this depends on the number of processors you have and whether your harddisk is fast enough e.g. is a SSD), or `OSM2PGSQL_DATAFILE` if your file has a different name.
**osm2pgsql** has a few [command line options](https://manpages.debian.org/testing/osm2pgsql/osm2pgsql.1.en.html) and the import by default uses a RAM cache of 512 MB, 1 worker, **and expects the import file to be named `data.osm.pbf`**.
If you want to customize any of these parameters, you have to set any number of the following environment variables:
* `OSM2PGSQL_CACHE` (e.g. `export OSM2PGSQL_CACHE=1024` on Linux to set the cache to 1 GB) for the RAM cache (the value depends on the amount of RAM you have available - the more you can use here the faster the import may be)
* `OSM2PGSQL_NUMPROC` for the number of workers (this depends on the number of processors you have and whether your hard disk is fast enough e.g. is a SSD)
* `OSM2PGSQL_DATAFILE` if your file has a different name

You can also [tune the **PostgreSQL**](https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server) during the import phases, with `PG_WORK_MEM` (default to 16MB) and `PG_MAINTENANCE_WORK_MEM` (default to 256MB), which will eventually write `work_mem` and `maintenance_work_mem` to the `postgresql.auto.conf` once, making them applied each time the database started. Note that unlike osm2pgsql variables, once thay are set, you can only change them by running `ALTER SYSTEM` on your own, changing `postgresql.auto.conf` or remove the database volume by `docker-compose down -v && docker-compose rm -v` and import again.

Expand Down Expand Up @@ -76,13 +82,13 @@ Depending on your machine and the size of the extract the import can take a whil

## Test rendering

After you have the necessary data available you can start Kosmtik to produce a test rendering. For that you run `docker-compose up kosmtik` in the openstreetmap-carto directory. This starts a container with Kosmtik and also starts the PostgreSQL database container if it is not already running. The Kosmtik container is built the first time you run that command if it not exists.
At startup of the container the script `scripts/docker-startup.sh` is invoked which downloads necessary shapefiles with `scripts/get-external-data.py` (if they are not already present). It afterwards runs Kosmtik. If you have to customize anything, you can do so in the script. The Kosmtik config file can be found in `.kosmtik-config.yml`.
If you want to have a [local configuration](https://github.com/kosmtik/kosmtik#local-config) for our `project.mml` you can place a `localconfig.js` or `localconfig.json` file into the openstreetmap-carto directory.
After you have the necessary data available you can start Kosmtik to produce a test rendering. To do that, run `docker-compose up kosmtik` in the openstreetmap-carto directory. This starts a container with Kosmtik and also starts the PostgreSQL database container if it is not already running. The Kosmtik container is built the first time you run the command if it does not yet exist.
When the container starts, the script `scripts/docker-startup.sh` is invoked, which downloads supplemental shapefiles specified in `scripts/get-external-data.py` (if they are not already present). Afterward, it runs the Kosmtik server. If you have to customize anything, you can do so in the script. The Kosmtik config file can be found in `.kosmtik-config.yml`.
If you want to have a [local configuration](https://github.com/kosmtik/kosmtik#local-config) for your `project.mml`, you can place a `localconfig.js` or `localconfig.json` file into the openstreetmap-carto directory.

The shapefile data that is downloaded is owned by the user with UID 1000. If you have another default user id on your system, consider changing the line `USER 1000` in the file `Dockerfile`.

After startup is complete you can browse to [http://127.0.0.1:6789](http://127.0.0.1:6789) to view the output of Kosmtik. By pressing Ctrl+C on the command line you can stop the container. The PostgreSQL database container is still running then (you can check with `docker ps`). If you want to stop the database container as well you can do so by running `docker-compose stop db` in the openstreetmap-carto directory.
After startup is complete you can browse to [http://127.0.0.1:6789](http://127.0.0.1:6789) to use the Kosmtik web tool. You can stop the container by pressing Ctrl+C on the command line used to start the container. If this happens, the PostgreSQL database container is still running (you can check with `docker ps`). If you want to stop the database container as well, you can do so by running `docker-compose stop db` in the openstreetmap-carto directory.

## Troubleshooting

Expand All @@ -92,7 +98,7 @@ Docker copies log files from the virtual machine into the host system, their [lo

While installing software in the containers and populating the database, the disk image of the virtual machine grows in size, by Docker allocating more clusters. When the disk on the host system is full (only a few MB remaining), Docker can appear stuck. Watch the system log files of your host system for failed allocations.

Docker stores its disk image by default in the home directories of the user. If you don't have enough space here, you can move it elsewhere. (E.g. macOS: Docker > Preferences > Disk).
Docker stores its disk image by default in the home directory of the user. If you don't have enough space here, you can move the images elsewhere. (E.g. macOS: Docker > Preferences > Disk).

## Style Debugging

Expand Down
20 changes: 10 additions & 10 deletions INSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Enable PostGIS and hstore extensions with
psql -d gis -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'
```

then grab some OSM data. It's probably easiest to grab an PBF of OSM data from [Geofabrik](https://download.geofabrik.de/). Once you've done that, import with osm2pgsql:
Then, grab some OSM data; It's probably easiest to grab an PBF of OSM data from [Geofabrik](https://download.geofabrik.de/). Once you've done that, import with osm2pgsql:

```sh
osm2pgsql -G --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua -d gis ~/path/to/data.osm.pbf
Expand Down Expand Up @@ -50,16 +50,16 @@ scripts/indexes.py -0 | xargs -0 -P0 -I{} psql -d gis -c "{}"
## Scripted download
Some features are rendered using preprocessed shapefiles.

To download them and import them into the database you can run the following script
To download them and import them into the database you can run the following script:

```sh
scripts/get-external-data.py
```

The script downloads shapefiles, loads them into the database and sets up the tables for rendering. Additional script option documentation can be seen with `scripts/get-external-data.py --help`.
The script downloads shapefiles, loads them into the database, and sets up the tables for rendering. Additional script options and corresponding documentation is available by invoking `scripts/get-external-data.py --help`.

## Fonts
The stylesheet uses Noto, an openly licensed font family from Google with support for multiple scripts. The stylesheet uses Noto's "Sans" style where available. If not available, this stylesheet uses another appropriate style of the Noto family. The "UI" version is used where available, with its vertical metrics which fit better with Latin text.
The stylesheet uses Noto, an openly licensed font family from Google with support for multiple scripts. The stylesheet uses Noto's "Sans" style where available. If not available, this stylesheet uses another appropriate style from the Noto family. The "UI" version is used where available, as its vertical metrics fit better with Latin text.

Hanazono is used a fallback for seldom used CJK characters that are not covered by Noto.

Expand All @@ -74,11 +74,11 @@ scripts/get-fonts.sh
## Dependencies

For development, a style design studio is needed.
* [Kosmtik](https://github.com/kosmtik/kosmtik) - Kosmtik can be launched with `node index.js serve path/to/openstreetmap-carto/project.mml` The 0.0.17 release of Kosmtik is not enough because we need up-to-date CartoCSS and Mapnik versions. To install the current master branch of Kosmtik, you can clone the Kosmtik repository and execute `npm install` within it.
* [Kosmtik](https://github.com/kosmtik/kosmtik) - Kosmtik can be launched with `node index.js serve path/to/openstreetmap-carto/project.mml`
* The 0.0.17 release of Kosmtik is not enough because we need up-to-date CartoCSS and Mapnik versions. To install the current master branch of Kosmtik, you can clone the Kosmtik repository and execute `npm install` within it.
* [TileMill](https://tilemill-project.github.io/tilemill/) is not officially supported, but you may be able to use a recent TileMill version by copying or symlinking the project directly into your Mapbox/project directory.

[TileMill](https://tilemill-project.github.io/tilemill/) is not officially supported, but you may be able to use a recent TileMill version by copying or symlinking the project directly into your Mapbox/project directory.

To display any map a database containing OpenStreetMap data and some utilities are required
To display *any* map, a database containing OpenStreetMap data and some utilities are required:

* [PostgreSQL](https://www.postgresql.org/)
* [PostGIS](https://postgis.net/)
Expand All @@ -91,7 +91,7 @@ To display any map a database containing OpenStreetMap data and some utilities a
Some colours, SVGs and other files are generated with helper scripts. Not all users will need these dependencies

* Python and Ruby to run helper scripts
* [Color Math](https://github.com/gtaylor/python-colormath) and [numpy](https://numpy.org/) if running generate_road_colors.py helper script (may be obtained with `pip install colormath numpy`)
* [Color Math](https://github.com/gtaylor/python-colormath) and [numpy](https://numpy.org/) (if running) feature a `generate_road_colors.py` helper script, which may be obtained with `pip install colormath numpy`)

### Additional deployment dependencies

Expand All @@ -100,4 +100,4 @@ For deployment, CartoCSS and Mapnik are required.
* [CartoCSS](https://github.com/mapbox/carto) >= 1.2.0 *(we're using YAML)*
* [Mapnik](https://github.com/mapnik/mapnik/wiki/Mapnik-Installation) >= 3.0.22

With CartoCSS you compile these sources into a Mapnik compatible XML file. When running CartoCSS, specify the Mapnik API version you are using (at least 3.0.22: `carto -a "3.0.22"`).
With CartoCSS, these sources are compiled into a Mapnik compatible XML file. When running CartoCSS, specify the Mapnik API version you are using (at least 3.0.22: `carto -a "3.0.22"`).

0 comments on commit dae7527

Please sign in to comment.