English فارسی Suomi
Français Nederlands Translate

OpenStreetMap

Providing maps for your web site

There are many different options for providing maps on your own web site. The most basic and easiest for you to use is a JavaScript library called OpenLayers that provides a draggable map that you can embed on your own webpage. This is similar to the earlier section in this book describing the "Export" tab on http://osm.org/. In this chapter we'll go farther by describing how to set up your own server with a customized map style based on OpenStreetMap data. You might want to do this if you are interested in highlighting a particular kind of OpenStreetMap data like bicycle or foot paths or if you are dissatisfied with the elements present in the other available map styles.

System requirements

Serving your own maps is a fairly intensive task. Depending on the size of the area you're interested in serving and the traffic you expect the system requirements will vary. In general, requirements will range from 10-20GB of storage, 4GB of memory, and a modern dual-core processor for a city-sized region to 300GB+ of fast storage, 24GB of memory, and a quad-core processor for the entire planet.

What are tiles?

For our purposes, a web map is made up of several adjacent square images, each 256 pixels square, that cover a particular geographic region of the Earth. The OpenLayers library we are using in this example will place these tiles in a grid to form a draggable view of anywhere on the Earth.

Using OpenLayers

OpenLayers is a JavaScript library for displaying maps on your web site. It is very powerful and can be used to add advanced features to your map like buttons, overlays, popup windows and lines. In this example we'll use it simply to show the map that you render, but to learn more about extending it to include these features, please visit http://openlayers.org/.

The toolchain

We use a series of tools for generating and serving map tiles:

Apache provides the front end server that handles requests from your web browser and passes the request to mod_tile. The Apache web server can also be used to serve static web content like the HTML, JavaScript, or CSS for your map webpage.

Once Apache handles the request from the web user, it passes the request to mod_tile to deal with. Mod_tile checks if the tile has already been created and is ready for use or whether it needs to be updated due to not being in the cache already. If it is already available and doesn't need to be rendered, then it immediately sends the tile back to the client. If it does need to be rendered, then it will add it to a "render request" queue, and when it gets to the top of the queue, a tile renderer will render it and send the tile back to the client.

We use a tool called Mapnik to render tiles. It pulls requests from the work queue as fast as possible, extracts data from various data sources according to the style information, and renders the tile. This tile is passed back to the client and moves on to the next item in the queue.

For rendering, OpenStreetMap data is stored in a PostgreSQL database created by a tool called osm2pgsql. These two pieces work together to allow efficient access to the OpenStreetMap geographic data. It is possible to keep the data in the PostgreSQL database up to date using a stream of diff files produced every 60 seconds on the main OpenStreetMap server.

Setting up your own tile server

Setting up your own OpenStreetMap server is a lengthy task and best described on Richard Weait's website here: http://weait.com/content/build-your-own-openstreetmap-server.

Software Configuration

Start with an installation of Ubuntu (10.04) Lucid Lynx Server for your hardware architecture. Add the LAMP server and SSH server during installation. Once installed, this can be run as a headless server with neither monitor nor keyboard. So ssh to the new box, and let's get started. Since there may be several updates for Lucid Lynx, update the packages on your system:

sudo apt-get update
sudo apt-get upgrade

Fetch some essential system tools for this tile server installation process:

sudo apt-get install subversion autoconf screen munin-node munin htop

Organize the file system to make the installation process a bit easier:

cd ~
mkdir src bin planet

Get the latest OpenStreetMap data 

Retrieve a piece of OpenStreetMap data from http://planet.openstreetmap.org/. Since the whole planet is at least 18GB when compressed, there are links to smaller country or state sized extracts on that page. Since it is smaller and faster to interact with, the PBF file format is preferable when available. In this case we'll download the entire planet file by issuing the following command:

cd planet
wget http://planet.openstreetmap.org/planet-latest.osm.bz2

Install and configure PostgreSQL database 

Next, we need to install the PostgreSQL database system. Start by using apt-get to install the database and required extensions:

sudo apt-get install postgresql-8.4-postgis postgresql-contrib-8.4 postgresql-server-dev-8.4 build-essential libxml2-dev libtool libgeos-dev libpq-dev libbz2-dev proj

The default configuration for PostgreSQL 8.4 needs to be tuned for the amount of data you are about to add to it. Edit the file /etc/postgresql/8.4/main/postgresql.conf and make the following changes:

shared_buffers = 128MB
checkpoint_segments = 20
maintenance_work_mem = 256MB
autovacuum = off

These changes require a kernel configuration change. Perform that change and save it so that it is applied next time your computer is rebooted:

sudo sysctl -w kernel.shmmax=268435456
sudo sysctl -p /etc/sysctl.conf

Restart PostgreSQL to apply these new changes:

sudo /etc/init.d/postgresql-8.4 restart

It should restart without any errors and report back with this message:

 * Restarting PostgreSQL 8.4 database server
   ...done.

Create a database called gis. Some of our future tools presume that you will use this database name. Substitute your username for username in two places below. This should be the username that will render maps with Mapnik.

sudo -u postgres -i
createuser username # answer yes for superuser
createdb -E UTF8 -O username gis
createlang plpgsql gis
exit

Set up PostGIS on the postresql database:

psql -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql -d gis

This should respond with many lines ending with:

...
CREATE FUNCTION
COMMIT
...
DROP FUNCTION

Give your newly-created user permission to access some of the PostGIS extensions' data. Make sure you replace username with your user's name:

echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis

Load OpenStreetMap data into the database 

We need to set up the tool that converts OSM data into the PostgreSQL data format. Start by checking out the latest source code and compiling it:

cd ~/bin
svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
cd osm2pgsql
./autogen.sh
./configure
make

Once compiled, we need to set the spatial reference on the newly-created database:

psql -f ~/bin/osm2pgsql/900913.sql -d gis

With the conversion tool compiled and the database prepared, the following command will insert the OpenStreetMap data you downloaded earlier into the database. This step is very disk I/O intensive and will take roughly 30 hours depending on the speed of the computer performing the conversion and the size of the data you retrieved from the server.

cd ~/bin/osm2pgsql
./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/planet-latest.osm.bz2

Let's have a look at your data import as it progresses. The first part of the osm2pgsql output looks scary, but is normal:

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE:  table "planet_osm_point" does not exist, skipping
NOTICE:  table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE:  table "planet_osm_line" does not exist, skipping
NOTICE:  table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE:  table "planet_osm_polygon" does not exist, skipping
NOTICE:  table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE:  table "planet_osm_roads" does not exist, skipping
NOTICE:  table "planet_osm_roads_tmp" does not exist, skipping
Mid: pgsql, scale=100, cache=4096MB, maxblocks=524289*8192
Setting up table: planet_osm_nodes
NOTICE:  table "planet_osm_nodes" does not exist, skipping
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes"
Setting up table: planet_osm_ways
NOTICE:  table "planet_osm_ways" does not exist, skipping
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways"
Setting up table: planet_osm_rels
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"

Don't be concerned by the NOTICE: entries above. All normal. Next, osm2pgsql will start reading the compressed planet file.

Reading in file: /home/user/planet/planet-latest.osm.bz2

As osm2pgsql reads the planet file it will give progress reports. The line below will refresh every few seconds and update the numbers in brackets. This part of the import takes a long time. Depending on your server, it will take between hours and days.

Processing: Node(10140k) Way(0k) Relation(0k)

As the import proceeds, the Node number will update a couple of times per second until complete, then the Way number will update roughly every second or two. Finally, the Relation number will update at roughly once per minute. As long as you can see these numbers advancing the import process is still operating normally for your server. Do not interrupt the import process unless you have decided to start over again from the beginning.

Processing: Node(593072k) Way(45376k) Relation(87k)
Exception caught processing way id=110802
Exception caught processing way id=110803
Processing: Node(593072k) Way(45376k) Relation(474k)

The exceptions shown above are due to minor errors in the planet file. The planet import is still proceeding normally.

The next stage of the osm2pgsql planet import process also will take between hours and days, depending on your hardware. It begins like this:

Node stats: total(593072533), max(696096737)
Way stats: total(45376969), max(55410575)
Relation stats: total(484528), max(555276)

Going over pending ways
processing way (752k)

The processing way number should update approximately each second.

Going over pending relations

node cache: stored: 515463899(86.91%), storage efficiency: 96.01%, hit rate: 85.97%
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_polygon
Committing transaction for planet_osm_point
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
Stopped table: planet_osm_nodes
Building index on table: planet_osm_ways
Stopped table: planet_osm_rels
Completed planet_osm_point
Completed planet_osm_roads
Completed planet_osm_polygon
Completed planet_osm_line
Stopped table: planet_osm_ways

At this point the import is complete and successful.

Install Mapnik library 

Next, we need to install the Mapnik library. Mapnik is used to render the OpenStreetMap data into the tiles used for an OpenLayers web map. To start, we'll install the dependencies for the Mapnik library followed by downloading and compiling the source:

sudo apt-get install libltdl3-dev libpng12-dev libtiff4-dev libicu-dev libboost-python1.40-dev python-cairo-dev python-nose libboost1.40-dev libboost-filesystem1.40-dev libboost-iostreams1.40-dev libboost-regex1.40-dev libboost-thread1.40-dev libboost-program-options1.40-dev libboost-python1.40-dev libfreetype6-dev libcairo2-dev libcairomm-1.0-dev libgeotiff-dev libtiff4 libtiff4-dev libtiffxx0c2 libsigc++-dev libsigc++0c2 libsigx-2.0-2 libsigx-2.0-dev libgdal1-dev python-gdal imagemagick ttf-dejavu

Build the Mapnik library from source:

cd ~/src
svn co http://svn.mapnik.org/tags/release-0.7.1/ mapnik
cd mapnik
python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/truetype/
python scons/scons.py
sudo python scons/scons.py install
sudo ldconfig

Verify that Mapnik has been installed correctly:

python
>>> import mapnik
>>> 

If python replies with the second chevron prompt >>> and without errors, then Mapnik library was found by Python. Congratulations!

Install Mapnik tools 

Next, we need to install the OpenStreetMap Mapnik tools, which include the default style file and tools to help Mapnik render OpenStreetMap data:

cd ~/bin
svn co http://svn.openstreetmap.org/applications/rendering/mapnik

Mapnik uses prepared files to generate coastlines and ocean areas for small scale maps since it is faster than reading the entire database for this information.

cd ~/bin/mapnik
mkdir world_boundaries
wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz
tar xvzf world_boundaries-spherical.tgz
wget http://tile.openstreetmap.org/processed_p.tar.bz2
tar xvjf processed_p.tar.bz2 -C world_boundaries
wget http://tile.openstreetmap.org/shoreline_300.tar.bz2
tar xjf shoreline_300.tar.bz2 -C world_boundaries
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/10m-populated-places.zip
unzip 10m-populated-places.zip -d world_boundaries
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/110m-admin-0-boundary-lines.zip
unzip 110m-admin-0-boundary-lines.zip -d world_boundaries

The database is loaded and the tools are installed. Let's test everything together. Remember to replace username with your username.

cd ~/bin/mapnik
./generate_xml.py --dbname gis --user username --accept-none
./generate_image.py

View image.png to confirm that you have rendered a map of England. Congratulations!

Install and configure Apache web server and mod_tile

Now that we've created an image using your map data, we need to set up the tile serving software to handle tile requests using Apache. To do this, we'll install the Apache web server and use the mod_tile extension to handle the tile rendering system.

Begin by installing tools necessary to build mod_tile:

sudo aptitude install apache2 apache2-threaded-dev apache2-mpm-prefork apache2-utils libagg-dev

Compile the mod_tile source code:

cd ~/src
svn co http://svn.openstreetmap.org/applications/utils/mod_tile
cd mod_tile
make
sudo make install

Change the the mod_tile settings by editing the /etc/renderd.conf and change the following lines like so (remember to change username to your user's name):

plugins_dir=/usr/local/lib/mapnik/input
font_dir=/usr/lib/mapnik/fonts
XML=/home/username/bin/mapnik/osm.xml
HOST=localhost

Create the files required for the mod_tile system to run (remember to change username to your user's name):

sudo mkdir /var/run/renderd
sudo chown username /var/run/renderd

Next, we need to tell the Apache web server about our new mod_tile installation by creating the file /etc/apache2/conf.d/mod_tile and adding one line:

LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so

Also, Apache's default website configuration file needs to be modified to include mod_tile settings. Modify the file /etc/apache2/sites-available/default to include the following lines immediately after the admin e-mail address line:

LoadTileConfigFile /etc/renderd.conf
ModTileRenderdSocketName /tmp/osm-renderd
# Timeout before giving up for a tile to be rendered
ModTileRequestTimeout 0
# Timeout before giving up for a tile to be rendered that is otherwise missing
ModTileMissingRequestTimeout 30

You'll then need to start the mod_tile rendering application. Note that it will not output anything upon successful start.

cd ~/src/mod_tile
./renderd

Congratulations! You now have a server capable of generating tiles using your own copy of the OpenStreetMap data. Visit http://localhost/osm_tiles2/0/0/0.png to verify that you have a working map tile generated.


EDIT