osm2pgsql Tuning for Modest Hardware

Fourteen Hours. You Don’t Have Fourteen Hours.

You’re importing the USA extract into Postgres. osm2pgsql is chugging along. You checked the logs twenty minutes ago — it’s still indexing nodes. The ETA some optimistic blog post promised you (“a few hours on reasonable hardware!”) has quietly become a silent lie. The import has been running for fourteen hours. You have a 16 GB RAM box, 8 cores, an NVMe drive, and a mounting suspicion that you need a 64 GB Threadripper to get this done in your lifetime.

You don’t. You need correct flags, a Postgres config that isn’t tuned for serving five concurrent web users, and about forty minutes of reading. The defaults osm2pgsql ships with are conservative enough to run on hardware from 2014 without blowing up. That’s not a criticism — it’s sensible. But it means if you just run osm2pgsql north-america-latest.osm.pbf and walk away, you’re leaving most of your hardware on the table.

Here’s what actually matters.

Full example: Tuned config and import scripts at github.com/KingPin/sumguy-examples/tree/main/self-hosting/osm2pgsql-tuning-modest-hardware

The Default-Settings Tax

osm2pgsql without flags will use legacy output mode, modest RAM, a single-digit cache, and whatever Postgres’s out-of-the-box config permits. On a fresh Ubuntu server with default Postgres, those defaults look roughly like this under the hood:

• shared_buffers = 128 MB (PostgreSQL default, written for a machine that shares RAM with other apps)
• --cache = 800 MB (osm2pgsql default)
• --number-processes = 1
• Output mode: pgsql (the legacy mode)
• Writes go through fsync on every transaction

The result is an import that crawls because it’s thrashing disk instead of using RAM, serializing work that could be parallel, and calling fsync() after writing a batch of a few hundred nodes. Change all of these, and the same machine with the same disk goes from a fourteen-hour USA import to something around four to five hours. Real numbers at the end of this article.

Flex Output vs Legacy Pgsql Mode

This is the most impactful architectural decision you’ll make before you even type a flag.

osm2pgsql has two output modes. The legacy pgsql mode has been around forever. It generates a fixed set of tables (planet_osm_point, planet_osm_line, planet_osm_polygon, planet_osm_roads) with a hardcoded column schema. It works, it’s familiar, and it’s the reason most tutorials from 2019 still run without modification.

The problem: legacy pgsql mode creates columns for every tag it knows about, most of which will be NULL for most rows. You’re paying index creation cost and disk space for columns nobody asked for.

Flex mode — stable since osm2pgsql 1.5 — lets you define exactly the tables and columns you want via a Lua script. You only import what you need. If you’re building a routing database and you don’t care about amenity tags, you just don’t create that column. The resulting tables are smaller, the indexes are faster to build, and queries run faster because the data is tighter.

A minimal flex config for roads only:

local tables = {}

tables.roads = osm2pgsql.define_way_table('roads', {
    { column = 'name',     type = 'text' },
    { column = 'highway',  type = 'text' },
    { column = 'surface',  type = 'text' },
    { column = 'maxspeed', type = 'text' },
    { column = 'geom',     type = 'linestring', projection = 4326 },
})

function osm2pgsql.process_way(object)
    if object.tags.highway then
        tables.roads:insert({
            name     = object.tags.name,
            highway  = object.tags.highway,
            surface  = object.tags.surface,
            maxspeed = object.tags.maxspeed,
            geom     = object:as_linestring(),
        })
    end
end

Pass it with --output=flex --style=roads.lua. You’ll be shocked how much faster a targeted import is compared to ingesting every tag in the file.

For general-purpose use cases (geocoding, tile serving), the osm2pgsql project ships community Lua scripts under flex-config/. Start there if you don’t want to write your own.

--slim vs In-Memory: Know Your Trade-Off

osm2pgsql has two main operating modes for the node/way/relation store:

In-memory mode (no --slim): all the intermediate OSM data lives in RAM during import. Blazing fast when it fits. When it doesn’t fit, the process OOMs and dies, or thrashes swap until you regret owning a computer.

Slim mode (--slim): the intermediate store goes to Postgres tables on disk. Slower by nature — you’re doing round-trips to disk for lookups — but it won’t blow up your RAM budget.

The rule of thumb I use: if the uncompressed import file is larger than 60% of your available RAM, use --slim. For reference:

• Full planet PBF is ~80 GB compressed, ~1.5 TB uncompressed equivalent
• USA extract is ~12 GB compressed; intermediate data needs ~40–60 GB RAM without slim
• Germany alone is ~4 GB compressed; manageable without slim on a 32 GB machine

On a 16 GB box importing anything bigger than a small European country, slim mode is not optional — it’s just reality.

The trick is that slim mode with a well-tuned Postgres is not that much slower than in-memory mode on average hardware. The disk is the bottleneck either way if you’re already pushing RAM limits. And slim mode gives you an additional bonus: if the import crashes halfway, you can resume it with --slim --drop removed and the middle-tables intact. In-memory mode is start-over-from-zero territory on failure.

--cache and --number-processes: More Isn’t Always Better

--cache controls how many nodes osm2pgsql caches in RAM while resolving way and relation geometries. The default (800 MB) is conservative. On a 16 GB machine doing a USA import, you can safely push this to 4000–6000 MB — but there’s a ceiling.

The cache is used during the way-processing pass. If you blow your available RAM with a too-large cache value, the OS starts swapping, and you’ve turned a speed optimization into a performance disaster. Practical guide:

• 8 GB RAM: --cache 2000
• 16 GB RAM: --cache 4000
• 32 GB RAM: --cache 8000

Leave at least 4 GB for the OS, Postgres shared buffers, and osm2pgsql’s own working memory overhead.

--number-processes controls how many parallel worker processes handle the geometry creation phase. On an 8-core machine, --number-processes 4 is a solid default. Going to 8 is not twice as fast — you hit IO contention before CPU saturation on most import workloads. Generally, cores / 2 is a good starting point. Run it higher and watch iostat — if disk utilization is pegged at 100% and processes are waiting on IO, adding more processes isn’t helping.

The full command for a 16 GB / 8-core box:

osm2pgsql \
  --output=flex \
  --style=/path/to/your-style.lua \
  --slim \
  --drop \
  --cache=4000 \
  --number-processes=4 \
  --host=localhost \
  --database=osm \
  --username=osm \
  north-america-latest.osm.pbf

--drop tells osm2pgsql to delete the slim mode intermediate tables after the import finishes. Skip it if you want to run updates against the same database later with --append. Keep it if this is a one-time bulk import — reclaim the disk.

Postgres Tuning for Bulk Imports

This is where most people leave the most time on the table. Default Postgres is tuned to be a safe, concurrent OLTP database. An osm2pgsql import is a bulk-load job — it’s basically the opposite workload. Tuning accordingly is fair game.

A postgresql.conf tuned for import time (apply before running osm2pgsql, revert after):

# RAM allocation
shared_buffers = 4GB               # ~25% of system RAM
maintenance_work_mem = 2GB         # for index builds; higher = faster VACUUM, CREATE INDEX
work_mem = 256MB                   # per sort/hash operation

# Safety off for speed — REVERT THESE AFTER IMPORT
synchronous_commit = off           # don't wait for WAL flush on every commit
wal_level = minimal                # less WAL overhead, disables replication slots/archiving
fsync = off                        # dangerous in production; fine for a one-shot import
full_page_writes = off             # pairs with fsync=off

# Checkpoint tuning (reduce frequency, increase size)
checkpoint_completion_target = 0.9
checkpoint_timeout = 60min
max_wal_size = 8GB

# Autovacuum off during import — it'll fight you
autovacuum = off

The fsync = off line will cause database corruption if the machine powers off mid-import. You accept that trade-off because: (a) you can re-import from the PBF, and (b) a crash during an 8-hour import is annoying but not catastrophic. Never leave this in place for production use.

After the import completes:

# Revert the dangerous settings
psql -U postgres -c "ALTER SYSTEM RESET fsync;"
psql -U postgres -c "ALTER SYSTEM RESET synchronous_commit;"
psql -U postgres -c "ALTER SYSTEM RESET wal_level;"
psql -U postgres -c "ALTER SYSTEM RESET full_page_writes;"
psql -U postgres -c "SELECT pg_reload_conf();"

# Run ANALYZE so the query planner has fresh stats
psql -U osm -d osm -c "ANALYZE;"

You can apply these settings without restarting Postgres using ALTER SYSTEM + SELECT pg_reload_conf() for most of them. wal_level requires a full restart — plan accordingly.

Real Numbers: 16 GB / 8-Core Box, USA Extract

The hardware: an old workstation. 16 GB DDR4, 8-core Intel i7-8700K, 1 TB NVMe (Samsung 970 Evo). USA extract from Geofabrik (~12 GB compressed PBF). Flex output with a roads + buildings + places schema.

The full optimization stack cuts import time by roughly 70%. The single biggest win is the Postgres tuning — fsync=off and synchronous_commit=off alone account for 30–40% of the improvement depending on your disk subsystem. The move from legacy pgsql to flex mode is the second biggest, especially as you increase the size of the dataset because you’re just writing less data.

What scales linearly: smaller extracts. A single US state (California PBF is ~2 GB compressed) takes about 35 minutes with the tuned setup versus three-plus hours without tuning. The percentage improvement is consistent — the tuning unlocks your hardware regardless of dataset size.

Why Is It Stuck on Indexes

You’ve applied all the tuning, you’ve been watching the logs, and then it hits the indexing phase and appears to stop. osm2pgsql logs something like Creating index on roads... and then — nothing. For 45 minutes.

This is normal, and it’s the most frequently misunderstood part of a long import.

Index creation in Postgres is largely single-threaded for most index types (B-tree, GiST). The --number-processes flag parallelizes the geometry processing pass and the data-loading pass, but when osm2pgsql calls CREATE INDEX, that’s Postgres doing one index at a time per table, using a single backend process. There’s a max_parallel_maintenance_workers setting in Postgres 11+ that lets B-tree index builds use parallel workers, but GiST (used for spatial indexes) doesn’t benefit from it.

On the USA dataset, GiST spatial index creation on a large table takes 20–60 minutes per index, running at 100% of a single core. It’s not stuck. It’s working. Watch it with:

# In a separate terminal while import runs
watch -n5 "psql -U osm -d osm -c \
  \"SELECT phase, blocks_done, blocks_total, \
    round(blocks_done::numeric/nullif(blocks_total,0)*100, 1) AS pct \
    FROM pg_stat_progress_create_index;\""

That view shows you exactly which index is building, how many blocks it’s processed, and the percentage complete. Once you see it moving — even slowly — you know it’s not hung.

The mitigation: maintenance_work_mem set high (2 GB or more) lets Postgres use more RAM for index sort passes, which speeds up B-tree builds meaningfully. GiST improvement from higher maintenance_work_mem is more modest but still real.

After the Import: Don’t Forget ANALYZE

Fresh imports skip most ANALYZE calls during the process to avoid stalling on stats collection. After the import and index creation complete, run a full analyze pass before you hit the database with queries:

psql -U osm -d osm -c "ANALYZE VERBOSE;"

Without fresh stats, the query planner will make bad decisions. Spatial queries on geometry columns especially need up-to-date stats or you’ll get sequential scans on tables that have perfectly good indexes.

The ANALYZE on a USA-sized database takes 5–10 minutes. It’s not optional if you want decent query performance.

The Short Version

If you’re currently running a bare osm2pgsql command against default Postgres and wondering why it’s slow, the checklist is:

1. Switch to flex output with a schema that imports only what you need
2. Use --slim --drop for anything you can’t fit comfortably in RAM × 0.6
3. Set --cache to roughly 25% of your available RAM
4. Set --number-processes to cores / 2, watch IO before going higher
5. Apply the Postgres import tuning block (fsync=off, synchronous_commit=off, maintenance_work_mem=2GB)
6. Accept that index creation is single-threaded and not a hang — watch pg_stat_progress_create_index
7. Revert the dangerous Postgres settings and run ANALYZE when it’s done

The hardware you have is probably fine. The defaults just weren’t written for it.

Related posts

• Nominatim: Self-Hosted Geocoding — put that imported database to work
• PostGIS for Self-Hosted Mapping — spatial database fundamentals
• Nominatim vs Photon vs Pelias — when each geocoder wins

Related Reading

• Full Self-Hosted Maps Stack
• PostGIS for Self-Hosted Mapping
• Wiki.js for your documentation in docker
• BRouter: Cycling Routes That Don’t Suck
• Nominatim Diff Replication