I'm involved in many of these projects (helped get MapLibre started, wrote a library supporting PMTiles, have used OpenMapTiles for years), and to see open-source, open-data win in the end is really satisfying.
If anyone has any questions about how you could do this for yourself, I'm happy to answer! (I have a company offering SaaS maps, but we know SaaS isn't for everyone, and are happy to point people the right direction when self-hosting is what you want.)
Thanks for all your work on MapLibre, along with all the other contributors! I use MapLibre extensively in an open source and non-profit transit data aggregator and everyone keeps being blown away by how smoothly the map runs, even with tens of thousands of points.
We burn through MapTiler free tiles on a regular basis and have been looking to migrate to self-hosting, but somehow PMTiles flew under my radar and every other approach seemed to either still rely on MapTiler for data (where you have to chose between outdated or expensive) or just be a huge pain to set up. Hopefully this finally solves our map issues so we can actually start promoting the project without fear of the map disappearing in the first week every month.
My company may be able to help save you the trouble of self-hosting. We do have a free tier (for non-commercial / test usage), and our pricing structure allows you to save quite a bit compared to MapTiler.
Stadia Maps, right? Your pricing is better than MapTiler, but both have the same issues for us - there's a significant jump from free to paid (no pay-as-you-go option) and server-side caching is prohibited. Both of these are completely sensible business decisions, but don't make sense for us specifically.
We have plenty of server resources from both our local partners and various cloud providers that have non-profit grants, but basically no regular budget to spend on SaaS, so self-hosting everything is the only thing that makes sense.
If you have access to SQLite in your desktop application then there isn't any big wins from using PMTiles over MBTiles; PMTiles does have some special design around tile-deduplication but this is possible in MBTiles too by using table views.
There isn't a tool to do that right now. It could be a fit in either https://github.com/protomaps/go-pmtiles or https://github.com/protomaps/PMTiles/tree/main/python - the Go program is faster and more production ready at this point. I imagine if folders are working for you the quantity of tiles doesn't number into the millions, so the Python program might be sufficient.
> But for most of our use cases, we don’t need the latest and greatest.
This is the crux of this. Let’s not pretend that much of this is anything other than “open source clones of Mapbox”.
PMTiles is genuinely innovative and I’m delighted to see it getting traction.
Maplibre GL (which I use and like very much) is a fork of the last open source version of Mapbox GL, with not many code changes between now and then.
Maputnik is an open source reimplementation of Mapbox Studio. Mapbox went on to hire its original dev.
OpenMapTiles started life as basically a how-to/makefile pulling together the disparate, poorly documented parts of Mapbox’s open source tile generation code. It’s now mostly known for a reasonable, not particularly exceptional vector tile schema.
It’s great that people are now shipping production vector tiles without paying the Mapbox dollar. But, PMTiles aside, this is still Mapbox’s world. The next challenge is to evolve the tech stack to something beyond what Mapbox worked up five/ten years ago.
Isn't this a bit like saying we're still in Google's world because most maps still use Web Mercator? :)
Good tech builds on what came before. Mapbox did a lot of ground-breaking work in building tooling around OSM, but so have many others. The fact that they named it Mapbox Vector Tiles is genius in hindsight, because even though we may use tons of tooling they didn't create to build and render them, their name is still there.
> The next challenge is to evolve the tech stack to something beyond what Mapbox worked up five/ten years ago.
Agreed, and I think we've seen a lot of iterative work in the open since then. The next challenge is likely building a OSS stack to do proper 3D: open data (including OSM) to pixels, and that work is already beginning across a lot of organizations: https://github.com/nyurik/future-mvt/discussions, Overture Maps, MapLibre, etc.
I get the parent comment. There are good alteratives to the mapbox ecosystem though they arent always used because people see mapbox as default
DeckGL for instance is a very innovative js library for front end (really developed by Uber I believe), with better 3d and movement, better with dealing data at scale, etc.
Prettymaps is a pure python approach to rendering OSM data.
It's weird, 10 years ago I implemented using Google Maps API a website with dynamic maps, distance matrix, geolocation, map overlays and address autocomplete completely free also at high traffic. It was also easy to use.
Five years ago Google started to charge an outrageous amount > $1000/month and we could not continue so we switched to something that cost $100/month. And that was for only the maps. We had to scrap the distance matrix and address autocomplete.
Today it's still the same status, and if you want something cheaper then you have to work on putting pieces together like the Washington Post does.
This site made a little bit of money, but not that much. But then 5 years ago google raised their prices so much that we were suddenly in the red. We had to drop the places API (replaced with Yelp) and the autocomplete API (we don't even use that anymore). Recently we've been moving to Apple Maps API which has a pretty generous free tier. Stadia Maps also looks interesting.
Hey guys not affiliated with Stadia but I can vouch for them, I use it in one of my products for a few years now and never heard a complaint, only nice things.
You can't go wrong with them.
"OpenStreetMap is the largest open geographic database in the world, the data infrastructure for multitudes of mapping projects around the globe. Your donation to the OpenStreetMap Foundation will cover our core operational expenses in supporting the OpenStreetMap project: hardware costs, legal fees, administrative assistant and other expenses of our working groups and administration.
We currently run extremely lean for an operation for a project the size and importance of OpenStreetMap. The OpenStreetMap Foundation relies on revenue from individual and corporate membership dues, profits generated by the annual State of the Map conferences, and donations. We must keep our income sources diversified, as these vary from year to year, but our modest needs stay the same. For this, we need your support."
[founder of felt, maintainer of some of the libraries mentioned]
this is amazing to see! we are happy to see our OS contributions like tippecanoe are being used by people all over the place, and i personally really love to work with journalists.
The Post has been doing great data journalism lately. The maps from the article were really well done. Yesterday they ran this story that has all its graphics done in D3 and provides links to notebooks at ObservableHQ with evidence supporting the arguments made and conclusions drawn in the article. https://www.washingtonpost.com/climate-environment/interacti...
Currently using Mapbox in an open-source project for their static map API and I wholeheartedly recommend against it. Everything but the simplest of polylines look jagged since they refuse to switch to POST requests with a json body and instead rely on you fitting in as much detail into URL query parameter length constraints.
Going to take a look at replacing them with PMTiles and my own polyline rendering.
I've switched over a fairly large transit client to PMTiles hosted on CloudFlare for their dynamic map and am switching another large transit client to using https://github.com/maptiler/tileserver-gl with MBTiles and the same style.json to generate static maps. It's all gone swimmingly and we've cut costs by about $1,000 a month.
There's actually a reason why most of these APIs support POST—most use-cases are hot-linking, where you don't control how the client makes the request. (Often using an <img>.)
Here's one alternative, if you don't want to build from scratch: https://docs.stadiamaps.com/static-maps/ We don't have much in the way of query string limits, so have at it!
You always have the option of creating a map style and using Mapbox's Raster Tile API - although that does mean you probably can't dynamically adjust your raster images in time for a request to be served.
How are you using the Static Maps API currently where you would be able to send a POST request and display the image back to the user? Most use-cases for the Static Maps API involve the HTML <img> tag or something similar, which all send GET requests.
Rather curious about the URL query parameter length constraint. Excluding IE11/Edge, don't most major browsers have at least 64k query lengths these days? Could you just serve the low quality jaggy poly to few percent still on IE11/Edge and a high detail one to the rest?
Titiler works pretty well for raster data in my experience. It is a step up from using an out of the box solution and involves a bit more development work but pretty easily allows customization. Using titiler on lambda with COGs stored on S3 we could get about 200ms load of a retina resolution map.
We wanted to do things even faster so we started migrating to ECS which when using a network optimized instance we could render a full page map in sub 100ms.
The author states that having all the data in a single file for PMTiles is advantageous, but it is a tradeoff. First of all you cannot modify parts of a S3 file so if you want to modify/update your dataset you need to re-process and re-upload the entire dataset. This can be a real pain if you are working with many TB datasets and you just want to update a subset of the data. Also, I thought that there is a global throughput limit on S3 files so wouldn’t putting all data in a single S3 file potentially hit bottlenecks?
Yes, PMTiles is a tradeoff that isn't appropriate for transactional use cases. SQLite is pretty good for that already.
There is a throughput limit on S3 files of approximately 5500 GETs/sec per key. Bare archives on S3 is an appropriate choice for small-scale, zero maintenance deployments. If your application demands any thing close to that level of throughput, you're probably either:
* Serving individual tiles over the internet: you should use the CDN integration http://protomaps.com/docs/cdn ; most tile requests will be cached and only misses will interact with the S3 bottleneck.
* Bulk accessing a spatial subset of tiles: You shouldn't be requesting HTTP GETs for single tiles, but instead entire subsets of tiles with a single Range request made possible by the internal Hilbert curve ordering. This is still WIP here: https://github.com/protomaps/go-pmtiles/issues/31
> Using titiler on lambda with COGs stored on S3 we could get about 200ms load of a retina resolution map.
Since Lambda presumably has a separate GDAL RAM cache for each request, I'm guessing this wasn't able to take advantage of any of GDAL's header or block caching? It's pretty impressive you were able to get latencies that low with that approach.
The big performance gains of GDAL in this configuration is HTTP2 multiplexing to S3 and TCP keepalives. When a tile request is sent to the lambda tiler it should already have a TCP/TLS connection open to S3 which can then be used to multiplex requests.
Looks great. Been using Mapbox as my go-to for small projects since it's much smoother than Leaflet and has better DX in my opinion. This seems to capture all of that with an even better DX so definitely will be trying it out.
If all of the tiles are coming from a single file (via PMTiles), how are they cached by the client? Do clients cache byte ranges, or do you configure things to give the appearance of serving separate files?
The renderer used in the linked article (MapLibre) has an internal tile cache used when you return to a previously loaded area of the map.
For the non-tiled parts of the PMTiles file, like metadata and the directories that store Z/X/Y to offset information, those are LRU cached by the PMTiles implementation. The most widely deployed right now is for TypeScript/Node. The "client" can either be web browser or inside a runtime like Lambda/Cloudflare Workers. On those the cache will persist across invocations using the ephemeral memory of the serverless function.
A level deeper, browsers vary in how they cache Range requests; IIRC Firefox does not cache while others will treat Range as if it was any other HTTP header.
You typically use byte-range caching in libraries—either using an edge worker (a la Cloudflare) to create separate GET requests from the client's perspective or directly in the client.
We use maplibre and map tiler for the map tiles in the startup that I run. We switched to that from leaflet and I've been pretty happy with it. I might invest some time at some point in self hosting tiles. But honestly, map tiler is good value for money for now.
I actually used to work with Nokia and Nokia Maps (in Berlin), which is now Here Maps. So, I know a thing or two about this market and still know some people working with various map companies, including at mapbox.
Mapbox is increasingly an alternative to Here, Tom Tom and a few others that offer a wide range of services. The most simple one of which is showing maps. Think of indoor maps, traffic services, routing, geo coding, address search (surprisingly hard problem with open data), etc. Customers for these services tend to spend a lot of money on getting all sorts of bespoke solutions. Maps are a commodity at this point and there's very little money in providing just some basic maps. It's all the other stuff that's built on top of that that is a lot harder and more valuable.
The recently announced overture collaboration could actually shake things up a little in this space. Openstreetmap is nice but also has some pretty significant limitations. Basically what the above mentioned companies do is creating proprietary services and data (with and without open data typically) that they then sell access to for a lot of money. The big companies behind overture are each big customers for this stuff. And now they are pooling resources to create a cheaper, more open alternative to all that.
Although I used to exclusively use it, Canvas-based maps just run so much smoother and support things like rotation that are really helpful for changing perspective.
This discussion is great to see! We’ve been talking about this article at work, the only thing I’d point out that hasn’t been said is that there is some complexity to designing with and managing the other renderer assets besides tiles (i.e. styles, markers, textures, icons, fonts). Most of this complexity is well documented if you know where to look, but will trip you up at least the first time you set up a serverless mapping stack, so buyer beware.
The trickiest of these assets is the stylesheet, necessary for the renderer to know how to draw the data in tiles. The article mentions Maputnik, which works great for quick prototyping and the production deadlines of newsrooms, but limiting once your map, application, and or tiles get to a certain level of complexity.
Do you know of any good alternatives to geoserver if i have a PostGIS db with some feature gis data and i need to expose it to other people through a Web Feature Service?
If you're just looking for a WFS (geojson/data) and not WMS/raster output, you might look at PyGeoAPI. (https://pygeoapi.io/) I haven't used it, but have looked at it a bit for a potential project to export geo data as an API.
Geoserver does fill a pretty big hole in the capabilities space -- it's pretty easy to get going with a bunch of layers and style them, but ultimately they're implementing a RDBMS in xml files, and it's a big, complicated, java system that's been one of the more troublesome portions of the stack (IME).
They used to be low-cost _and_ open source. Now they're source available and not interested in anything but big clients. I was using Mapbox for all my clients before their unopen sourcing, and I tried reaching out three times to ask about options for one of my clients. It wouldn't have been a huge amount of money, probably a few hundred per month, but they never, ever reply. That client wasn't even using any of their server resources, they are 100% self hosted, it's just that Mapbox now charges even for using their client. Anyways, I was trying to give them money, and they wouldn't even acknowledge I exist. I switched that client over to Maplibre and never looked back.
The other thing that annoys me is that they've more or less abandoned all their open source repos without even as much of a notice in the README. All these amazing tools sitting there with open pull requests and no one responding for years. If they just did a clean break, put a notice on everything they weren't going to touch again, it would have been a lot better/easier for the community to fork and move on with their lives.
The hosting and bandwidth costs are huge, and most of your customers (by raw count and nominal usage) are hobbyists or free products unwilling to pay even a few cents per user.
VC funding? I think they were funded mostly by grants and their own revenues for the first few years. Then they started taking VC money (over $300M to date).
They were once Development Seed [0], a company that specialized in making geo spatial visualizations for the UN, NGOs and nonprofits. Development Seed was a large contributor of code and modules to Drupal 6 which they left. In the beginning, their open source mapping and geo spatial visualizations projects were mostly funded by grants from the Knight Foundation which funded news related projects. MapBox came from an in-house tool they developed to create and manage map overlay tiles. (I could be making this all up.)
And since the state of the open source map rendering art is consisting almost exclusively of pre-licence-change forks of mapbox code these days, they kind of still are.
I wonder if Strava's recent acquisition of fatmap.com eventually boils down to them cutting ties with mapbox via acqui-hiring some mb-libre excellence. I could imagine Strava, due to the extreme map-centric nature of their product, bleeding quite considerable money to mapbox.
MapLibre has made really good strides (especially on the web) since the fork—3d-terrain support, more tile sources, other DX improvements (e.g., conversion to TypeScript).
I’m interpreting the comment above to mean Strava was/is a user of Mapbox and bought Fatmap in an effort to wean themselves off of Mapbox and use “inhouse” (Fatmap) mapping tech.
I took a superficial glance at what the site is loading and iirc there were plenty of the usual suspects in there. And it would be silly to do it any other way, because their product is what they do with maps, not winning some NIH purity award.
1. Subsidize low costs with venture capital and take on a ton of customers at a loss
2. Lock those customers into your API somehow. Enough code is effectively lock in for a while.
3. Change your pricing to be profitable.
4. Hopefully get acquired before all your customers leave
I think you can do this quite easily.
Use docker compose to start an nginx and maptiler/tileserver-gl. Use an nginx config to send requests upstream to tileserver and cache.
Download a region file from maptiler, or make one.
That's it.
It's about 2h work, using certbot for certificates.
While it might be nice if there we pre-rendered tiles available for download, and openmaptiles.org constantly steers users to "buy now" pages, instead of offering a simple button to the self-service docs, they do in fact offer a significant amount of documentation on how to generate and serve your own tiles[1], including a page with fairly step-by-step instructions on how to create a tileset "from scratch" without sending them a cent[2].
That's why I actually support paid licensing for commercial use. I'm all for helping out people but not a fan of getting taken advantage off - you make money off my or someones else's work then pay up.
Check this out for backend:
https://postgis.net/docs/ST_AsMVT.html
if you can run ogr2ogr to import your data into PostGIS, and serve the binary data returned by a query, you're all set. You literally only need PostGIS for a dynamic vector tile service.
Then mapboxgl or libremapjs on the frontend, or even OpenLayers.
I currently use a .mbtiles file and plug it into tileserver-gl, but i guess in theory you could just import OSM to PostGIS and then add the PostGIS as a source in tileserver-gl? That would be amazing
I don't want to sound annoying, but it's a bit hard to generate raster tiles. I mean openstreetmap data is open and free, but it doesn't seem they distribute any software to build the raster tiles they host.
I don't know what they use, but it doesn't seem there is any software that is simple enough to use.
I just want to configure a subregion, call generate-tiles.py planet.xml or whatever, and be done with it.
I like open source, but there will always be open source software that is needlessly complicated because, well, it lets companies like mapbox and others to generate income for their expertise.
That's not the best open source spirit, in my view.
We used https://github.com/developmentseed/titiler (and rio-tiler, the underlying library) at my last company which does dynamic tiling based on some input raster. It's an awesome project and there's nothing else quite like it in the geospatial space.
I discovered it while looking at raster solutions, and it is pretty great! The raster stack is pretty bonkers. TiTiler uses at least three fundamental layers: GDAL, RasterIO, and rio-tiler. Each contribute a fairly significant improvement (for the use-case) over the underlying one.
There's a lot to like about GDAL+Rasterio, although I've found having all HTTP requests go through GDAL's C API does result in some limitations on concurrency and multithreading. GDAL's configuration being based entirely on env vars also has its downsides: https://github.com/developmentseed/titiler/issues/186
Super easy way to generate a MBTiles, which you can then serve directly, or further convert to PMTiles, which can be used to host vector tiles for client-side rendering using MapLibre (or other renderers).
Raster tiles are a lot harder because you have to generate them on the server, and that's a lot more resource intensive.
The requirements are considerable, though, given the workload:
> Serving your own maps is a fairly intensive task. Depending on the size of the area you’re interested in serving and the traffic you expect the system requirements will vary. In general, requirements will range from 10-20GB of storage, 4GB of memory, and a modern dual-core processor for a city-sized region to 1TB of fast storage, 24GB of memory, and a quad-core processor for the entire planet.
Personally, for lower end mobile devices raster tiles still seem to perform better, when compared with at least some of the current vector tile implementations. I actually recorded a short comparison on my phone a while back: https://www.youtube.com/watch?v=A-IRtBGO9rM
Open-source geospatial tools in general have developed rapidly over the last couple of decades. Mapbox and friends have laid groundwork, both good and "needlessly complicated." But there are just too many people who care about this to think it'll always be hard.
I assure you, the OSS (and in this case, OSM) community will steadily build out better, faster software that includes more and more of the features a modern map needs. It's already looking like companies like WashPo are seeing the potential here.
https://github.com/maptiler/tileserver-gl makes it really easy. I have a pretty simple pipeline set up to download an OSM extract, convert it to MBTiles, host it in CloudFlare, and render raster tiles with this software.
I'm involved in many of these projects (helped get MapLibre started, wrote a library supporting PMTiles, have used OpenMapTiles for years), and to see open-source, open-data win in the end is really satisfying.
If anyone has any questions about how you could do this for yourself, I'm happy to answer! (I have a company offering SaaS maps, but we know SaaS isn't for everyone, and are happy to point people the right direction when self-hosting is what you want.)