Super excited for your project. I have been using map tiles from Mapbox, MapTiler, OSM Buildings for my project and am coming to the point that I need to merge elevation data with building height data in order to cast realistic shadows across the earth. Unfortunately elevation data and building data come from two different tile sets from two different tile providers and it's a lot of download/processing overhead to merge the two data sources. I need to make my own tiles!
I spent part of today on the OSM wiki trying to wrap my head around Mapnik, how tiles are generated, how I am going to host this cheaply because my project is non-commercial, and then the HN gods smiled on me. Thank you for taking this on and I'm coming along for this journey!
I’ve had success using tippecanoe to generate mvt tiles from geojson sources, which in turn came from various sources (osm and otherwise) from QGIS. I didn’t look into this project too much but it appears to be bundling mvt tiles into a single file? Either way, straight up mvt tiles are easy to generate and can be easily hosted statically on any web server/s3 bucket/cdn.
I'd also recommend having a look at https://openmaptiles.org/. They have workflows and editors for working with MVT and Maplibre. You can combine a base map from maplibre with other sources using Openlayers [0]
If you need to create your own tiles, I'm not sure how this changes anything. I can't seem to find how one would go about generating one's own tiles for this platform in their documentation. Did you find something to that effect in it?
Author here. You should be able to first download OpenStreetMap tiles in PMTiles format (http://github.com/protomaps/pmtiles) and then write out a new augmented tileset in the same format. It's a use case I want to have first-class support for, but the tooling doesn't exist yet.
Right now it's best suited for mapping applications focused on some part of the world - neighborhood, city, country scale - especially ones that might avoid using a hosted service. Think visualization projects built for clients, newspaper site interactives, etc.
I think the author would explain best. Personally, I'm working on casting live earth shadows on a tiled map (link in HN profile) and lack an off-the-shelf tile set that contains all the data I need and in some cases the available tiles I use contain more data than I want. Also, the tiles cost $$$ for each tile load.
So long-term I am looking into generating and hosting my own map tiles.
Depending on your needs, tilemaker (https://github.com/systemed/tilemaker) may be the least painful way for generating tiles from OSM data with modifications to it. Or at least it seems to be the least painful way of doing that that I've found so far. The input data is manipulated by Lua code that you provide, and some reasonable default configuration for it exists. Your custom Lua code could pull additional information from other information sources should you wish to do so.
This is very nice! Happy to see a few of my libraries in the dependencies :)
Also check out https://github.com/kothic/kothic-js, an older but similar attempt at making a full-fledged map renderer using 2D Canvas. It has a pretty good label rendering in particular which you might find useful.
I am super excited about the parent tool, but unsure whether/when I can generate these bundles programmatically. I tried the web based tool and it works great, but what if I need to do it many, many times?
Can your tool do this easily? Is there a way to generate tiles (tiles.pmtiles) that are imported in the same way?
If I had stumbled on your project, I would not have been enthusiastic about it with just the writeup as-is. Seeing the parent project makes me see the possibilities: a completely independent mapping solution, free of any server/service tie in and additional costs. That is super exciting, and I would love to know if and how your project compares.
I'd like to understand what you mean by doing it many times - do you mean for different areas of the world? I haven't thought about API access to the bundles in detail yet, but you can contact me via email at brandon@protomaps.com if you have some idea of what you need.
Thank you! Really appreciate the commitment to simplicity among the libraries I'm using like potpack, rbush, and polylabel. I did take a look at Kothic and agree the labeling is great; I did feel like I needed to write yet another Canvas renderer to take advantage of async/await and also eventually expose runtime-extendable custom Symbolizers.
This is really interesting to me. I do a lot of mapping of natural surface single track trails (for hiking and mountain biking) along the way be sure to properly name things, add intersection markers, and create relations (routes and superroutes) to document the systems.
The biggest downside thus far is a good way of displaying the data. mtb.waymarkedtrails.org gets close, but I'd really like something that displays relations in the specified color, has sane intersection markers, etc.
I feel like there should be something out there which effectively lets me design a style sheet and then apply it to a map which I embed somewhere (maybe even in an app).
This... feels like it? Or if not this, can any of you recommend another system?
Ideally I'd love something I can self-host. I tried getting a copy of waymarkedtrails going and then modifying that, but it's a bit too specialized and I kept running into quirks getting the toolchain set up.
EDIT: Looks like this might not include the highway=path data that I need, much relations. Alas.
Have you used GaiaGPS at all? I use it along with ridewithgps for looking at trail conditions and mapping points of interest on my rides and hikes. You can import your own custom map layers as well.
The developers are fairly consistently adding new content as well, might not hurt to reach out.
Author of OP here. Is this information that you're interested in contributing, and is appropriate for, OpenStreetMap?
The end goal is to build a system that lets you customize the tileset to include any node, way, relation, or tag in OSM. This gets quite tricky when considering automated cartographic generalization (how such features will appear when zoomed out) but it's one of my next steps.
Really interesting project/product! It's really nice to see new ways of rendering OSM data, and alternative map services.
I am developing another way to run your own map server (without mapnik!) from a low-end computer (https://github.com/jamesrr39/ownmap-app in case you're interested), but a little bit different to you in that I'm currently focusing on rendering raster tiles. I'm really happy I saw this thread, however, I hadn't heard of the OSM Express, MBTiles or PMTiles formats before and was/am rolling my own format, so it's really interesting looking at some ideas from these.
Does anyone have a really good (maybe illustrated?) guide/short reference to the world of GIS? Every time I see GIS stuff mentioned the problem space itself, solution jargon, and various movements in the ecosystem (projects being unsupported, stewards/companies changing etc) always throws me for a loop.
It's certainly the case that I just haven't built enough mapping-related software to be comfortable but if there's a resource out there that's like "baby's first GIS application" I'd love to read through it and try to ram the concepts into my gray matter again for the day when I do work on one.
Thanks for starting this way back then, the amount of value created directly and indirectly is immeasurable.
That said, I'm very doubtful of the likelihood of me contributing something novel and useful to the GIS world -- I'm keep to learn from and stand on the shoulders of giants that got us here for at least the first bit of my foray into GIS before trying to contribute something fresh, RTFM and all that. To be fair it has taken 15 years to whittle down this particular wheel, starting from scratch feels almost disrespectful -- I'll leave that for younger explorers.
Author of the OP here. I don't have a GIS background in particular; I do have a decade of experience building web mapping applications, and here's some of the foundational software I use:
Thanks for the pointers -- Leaflet I know as the visualization layer that is basically not-mapbox-GL. The rest are new to me, and will crawl through them.
One of the nice out-of-the-box systems I've come across is Pelias[0], what are your opinions on it? It bundles and provides the API so it's one of the things I've thought to self-host if I'm going to. It looks like it doesn't do tiles so I'd need to find something for that?
> I do have a decade of experience building web mapping applications
> I don't have a GIS background in particular;
It's only on HN that you get statements like this, haha, what do you think 10 years of experience building mapping applications is? I building mapping applications is generally what people consider "GIS background" -- if you did GIS but never produced any maps... would the trees make a sound?
Yes, Pelias is a mature project backed by a great team at https://geocode.earth - definitely one of the best options if you need an autocompleting geocoder based on OSM+other data like http://whosonfirst.org
About OSM data extracts, I also run an extracts service at http://protomaps.com/extracts that provides on-demand data updated once a minute.
I don't really know what is throwing you for a loop. I have a Certificate in GIS from UC-Riverside, very respected at the time and some of my professors worked for ESRI by day and taught part-time for this certificate program.
Some basics:
Software-wise, GIS is a database hooked to mapping software so you can query the database and spit out maps as your answer. It's valuable because "A picture's worth a thousand words" and most data can be connected to a mappable bit of data (address, lat-long, etc).
A Geographic Information System has the following components:
Data is the most expensive part of that. Good data is hard to come by.
People with the right skills and talents are the most critical part.
You can start a GIS project by surveying your in-house software, hardware, data flows, etc and putting together a use case. You probably already have some of the pieces you need to create a proper Geographic Information System where the data you already track can be hooked into a system for making it query-able and mappable so you can up your decision-making game.
ESRI is probably still the world leader in GIS software and they put out a certain amount of free info.
If I want to build a free open-source web application with a map view that can be annotated, works on mobile etc and uses Open Street Map data, what are the best options right now?
Very interesting, I'm currently using Google My Maps as a dead-simple custom mapping tool. I'm using it to collect layers of information as I conduct a nationwide housing search. It would be great if there were a solution that was as simple to get started with as My Maps but also had the flexibility to easily add custom data layers programmatically.
> It would be great if there were a solution that was as simple to get started with as My Maps but also had the flexibility to easily add custom data layers programmatically.
I believe _with some trickery_ you can use Google Earth for that
Paying a guy in Kenya to scrape data for me. Swear I tried to pay every national real estate API out there and no one is interested in talking to you at less than 7k a month and a full on lead gen business.
Towns across the country that meet our criteria overlayed with climate change maps. The existing layers around satellite and topo views are also super helpful to get an idea of if the area has good tree cover and interesting landscapes.
When we were city and house shopping, I kept wanted to be able to query houses for things that the major providers don’t provide query terms for. Even proximity to parks was difficult to discern or searching by walkscore, crime rates, etc, was difficult. Each house has that data, and you can search for some of it visually, but as a developer, you just want to run queries...You seem like the sort of person to crack that code when you do choose a city. I’d be curious if you do (or have) come up with anything :)
I am, but you really need to be at business scale to get data access, for an individual doing a 1-off search the economics don't make sense and the incentives of data providers arn't aligned.
That said there is a company (nextburb.com) who's CEO I've been in regular contact with who is trying to dramatically scale what kind of searches people can do. I've been talking to them about our needs and use-cases and they've been quite responsive.
Originally, MBTiles was designed for mobile offline. Then, Mapbox started serving directly from the SQLite for web & other clients, creating the Node.js SQLite bindings in the process. Eventually, they had an unpacker for MBTiles uploads that would stick the tiles in CDN. So, really no more complex than old school static content hosting.
Author of OP here, I use the tile unpacker as well! I didn't point this out in the OP, but I found the ergonomics of uploading tiles to S3 etc get bad once you're over a couple tens of thousands of tiles, as each upload has some overhead. I've tested PMTiles on S3 successfully with hundreds of millions of tiles.
Deduplication is also important to save space on redundant ocean/earth tiles; MBTiles can deduplicate via views, PMTiles deduplicates by reference, but unpacked Z/X/Y tiles on S3 can't.
When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so that the upload could be faster. It sounds like that’s what you did for the the PMTiles format?
That's exactly what PMTiles is for, and there's nothing in it specific to maps or geographic coordinates, just a Z/X/Y pyramid structure. I chose to support Leaflet for now because it is very popular; there's other viewers like OpenSeadragon for which a decoder plugin would be possible.
Fantastic. The renderer I was using was indeed leaflet based, so this sounds drop-in easy.
I just read through https://github.com/protomaps/PMTiles
Nice work!
The only downside would be that these maps are often re-rendered periodically and only have a few tiles that actually get updated. In that case, it’s nice to have to update only a handful of files. But I think the benefits would far outweigh the costs. Upload one massive file repeatedly is still gonna be faster I would think. Thanks!
That was the impetus behind MBTiles as well. I was building out an iOS toolkit to render OSM-based maps on iPad, offline, and transferring 1000s of rasters (at the time) even over USB was way too slow and error-prone.
Oh wow - very cool. Thanks for this! I love SQLite, and think this is an excellent use case. (File systems, especially networked ones, don’t handle “lots of tiny files” well). I think using SQLite as the primary, writeable storage mechanism makes sense because I could ship that around easily to rendering nodes and incrementally update it easily. Then, I could export to a static PMTile for upload to S3 for serving purposes.
AFAIU MBTiles still need some application server to transform x/y/z into proper tiles to be served, which means provisioning a server, scaling it, maintaning it, and all the fun that comes with it; PMTiles is a simple blob that the frontend will download (only desired parts) and render directly in the browser.
You can use tippecanoe (https://github.com/mapbox/tippecanoe) with the --output-to-directory flag to output the individual .pbf tiles instead of a single MBTiles file. These tiles can then be hosted on something like s3 and your map-rendering client can query the necessary tiles, no server required.
So I fee like it actually missed the mark as to why it's so impactful.
When you get to global scale tiling with building accuracy, you literally end up with hundreds of millions of tiles.
The upload time / cost per upload STACKS. However, the tilesets combined usually sits between 100GB and 200GB.
ONE upload of ~150GB seems is faster and more cost efficient via S3.
For comparison, Cloud Optimized GeoTIFF (COG) is specifically designed to be readable with HTTP Range requests, so the single file can be placed on S3 and be immediately usable.
As others pointed out, the MVTs inside can be unpacked for static serving. But a SQLite database is not itself usable in this way.
@bdon Awesome! The geospatial industry is in need of simpler tooling for specific jobs such as generating tilesets.
Currently complex tools such as QGIS and GDAL do everything, but the learning curve is very steep.
An addon you might consider a feature to drop an image (and adjust it to fit) over the OSM data to generate a XYZ tilesets directly on the customer AWS S3 (or equivalent storage).
This might be a good way to monetise your offering... I know my customers would buy it.
Curiously, the editor does not render (well, it renders a black screen) in Firefox 78.9.0. I see this error in devtools:
Uncaught SyntaxError: expected expression, got '=' protomaps.min.js:1:52669
On props update af913d4d.0fa8ea22.js:1:7975 ReferenceError: protomaps is not defined
Yes, I'll look into this. Can you email me at brandon@protomaps.com or make an issue at http://github.com/protomaps/protomaps.js ? I may need to make a get a sourcemap-enabled build online as well.
I personally avoid blanket statements about a certain design being faster than relational databases; for this project as a whole I try to take a "systems programming" approach; I make things faster by doing less work, in this case because I am designing for one specific use case and no more.
For OSMX vs. PostGIS, this entails:
1. PostGIS is a client-server database, OSMX is embedded (in-process). PostGIS must be written defensively for cases like multiple writers; this isn't important for a system that occasionally has a single writer.
2. PostGIS indexes must be built for arbitrary query plans, OSMX only has two kinds of access patterns: by primary key and by S2 cell. S2 cell indexing is an approximation, but will always return more data than you asked for and never less. This is an acceptable downside.
3. PostGIS must treat geodata as OGC Simple Features like points, lines and polygons, this involves lossy conversion from the OSM data model, which is topological.
4. OSMX is built on LMDB, which is an outstanding and underrated piece of database tech (I think the author posts on HN). It's paired with a message format that does not require deserialization (CapnProto) so certain operations common in working with OSM data, such as accessing the coordinates for an OSM node, do not need to malloc for each node. OSM data can be many times larger than RAM; memory-mapping the entire database means you don't need to write your own caching layer; spatial access patterns exhibit high locality since cells are arranged on space-filling curves.
I have tinkered with PostGIS before, and it seemed great. Nonetheless, you make several good points about how a special purpose tool/format can be faster by avoiding certain work.
As for LMDB, I haven't used it, but I have heard about it previously. A full-text search server called MeiliSearch uses it internally.
---
Separately, I think your broader project is really cool. Simplifying the whole end to end process is incredibly useful. I have previously thought "I would love to host a regional map, but I have no idea how".
In the past I made a map of train lines in and around Prague, which required quite a complex stack, since the information I wanted was in relations. How does Protomaps deal with relations?
It can deal with relations more flexibly than PostGIS-based workflows, because the tiles are generated from a near lossless representation of OSM instead of having to convert to OGC Simple Features first.
Can you email me at brandon@protomaps.com with what specific tags and relations you are interested in? I want to support transit layers as part of basemaps and ideally this is an option when generating a PMTiles archive.
1) If your information belongs in OpenStreetMap, you can add it via an editor like the web editor at https://openstreetmap.org - this will also benefit all other OSM users. You can then "refresh" your Protomaps download to get a new map.
2) If there isn't many point and polygons, it may sense to add them as Leaflet layers, especially if you want them to be interactive
3) Other options are creating vector tiles of your own data and merging or displaying them in the renderer (https://github.com/mapbox/tippecanoe is a great tool to do this from GeoJSON) but I don't have much to support this yet.
I am working on making the set of information included in the tiles configurable via a JSON definition, kind of like Overpass Turbo if you've used that.
The reason I'm doing it this way is because data completeness and performance - the total size of the tiles - are direct tradeoffs. If I keep augmenting the dataset with tags people request, they'll become too big for everyone!
So I'm exploring this idea of "a la carte" tags in tile sets, but am still gathering requirements. Can you email me at brandon@protomaps.com with more details of the OSM tags you want, as well as the area you're interested in so we can work together on checking out the data availability there?
The typical way to build and distribute Mapbox vector tiles has been by packing them into a sqlite database with individual rows for each Z/X/Y quad tree coordinate. This is what tools like tippecanoe typically produce.
The problem with this is it still requires a running server process colocated with that sqlite db in order to service requests for individual tiles, like what you'll receive from a client-side mapping library.
This project has a lot of different parts, but one of them is a spec for serving this type of data at low latency without a server by combining a custom packing format with S3 range-get requests to read individual tiles direct from blob storage. So that is certainly interesting for this kind of use-case.
OP author here; one thing I didn't touch on is the way that most map services license their basemap data explicitly disallows "caching" which would be inclusive of hosting basemaps yourself on your own storage. So for the cases where it makes engineering or product sense to have your maps on S3, it goes against the dominant map business model.
I’ve used the utility tippecanoe with success to generate vector tiles from geojson sources and am hosting them statically. It’s been overall pretty easy. The only gotcha I can remember is I needed to pass the no compression flag since mapbox gl can’t read compressed tiles (and the file host will gzip everything anyways).
What issues have you run into with gzipped tiles? I store a lot of gzipped individual tiles for use in Mapbox GL maps on S3 with no problem - the only necessary change was making sure that S3 was emitting `content-encoding` headers.
At least for simple GeoJSON you can use tippecanoe (https://github.com/mapbox/tippecanoe) with --output-to-directory to get individual .pbf tiles. Most rendering clients will have some scheme where you can provide a root url and then the tiles need to be stored in some defined structure beneath that (e.g. root/z/x/y.pbf)
> "In about five minutes, you can select any area in the world and get a self-contained map that runs locally, offline or serverlessly on S3 - check out the Getting Started guide."
5 mins? It took me a night to build the map system using openstreetmap in the original way
I spent part of today on the OSM wiki trying to wrap my head around Mapnik, how tiles are generated, how I am going to host this cheaply because my project is non-commercial, and then the HN gods smiled on me. Thank you for taking this on and I'm coming along for this journey!