
Serving Dynamic Vector Tiles from PostGIS - liotier
https://info.crunchydata.com/blog/dynamic-vector-tiles-from-postgis
======
flippmoke
Author of Mapbox's Vector Tile specification here and also contributor to some
of the code that is used by PostGIS and I wanted to add some additional
clarity on some topics associated with Vector Tiles and dynamic serving of
them that seems to be a new trend.

The Vector Tiles specification was designed for map visualization but has
expanded into other uses as well, but in general the purpose is to be able to
quickly provide a complete subset of data for a specific area that is highly
cacheable. Most of this provided speed and cache-ability is specifically
gained by preprocessing all the data you will use in your map into tiles.

The general steps for turning raw data into Vector Tiles are:

1\. Determine a hierarchy of your data. For example if you are talking about
roads at some zoom levels you will want to see only highways or major roads
while at other zoom levels you will want all your data.

2\. For each tile at each zoom level; Select your data following your
hierarchy rules, simplify your data based on your zoom level (for example you
might need less points to display your road) and then clip your data to your
tile and encode it to your Vector Tile.

The problem is that doing these steps is often very complex and requires
thought about the cartography of your final resulting map, but it can also
drastically effect performance. If you are dynamically serving tiles from
PostGIS it is very hard to reduce large quantities of data quickly in some
cases. For example take a very detailed coastline of a large lake that is very
precise and you are wanting to serve this dynamically. If you are attempting
to serve this data on demand each time you need a tile you have to simplify
and clip a potentially massive polygon. While this might work for single
requests, if you increase in scale this quickly adds lots of load to a PostGIS
server. The only solution is to cache the resulting tiles for a longer period
to limit load on your database or to preprocess all your data before serving.

Preprocessing of all the tiles is already something other tiling tools such as
tippecanoe are really good at doing and comes with the benefit of helping you
determine a hierarchy for your data. Preprocessing might seem excessive when
it comes to making potentially millions of tiles, but in general it makes your
application faster because it is simply serving an already created tile.

Therefore, if your data does not very change quickly I would almost always
suggest using preprocessing over dynamic rendering of tiles. You might spend
more effort maintaining something than you expect if you start using PostGIS
to create tiles on demand over existing tiling tools.

~~~
hkchad
So I saw this post earlier to day and tried it on a dataset we have (fixed
boundaries w/ some properties that change 4x/hr). We use the value of the
properties for styling of the vector tiles. Currently the tiles are re-
rendered every 4hrs (even though the data is updated every 15 min) using
tippiecanoe, served by tileserver-gl and cached in cloudfront. So I wanted a
way to get new data to users faster. But as you have noted this dynamic
process crunchy posted IS SLOW, it takes about 3 minutes to paint the world on
my brand new macbook pro (about 3 seconds w/ pre-rendered). Given the country
boundaries do not change very often is there a way to change just the
properties that actually needed updated in the already rendered vector tiles?
Our pipeline takes about 45 min to run completely to regenerate the new tiles
with updated properties. Or is there a better way to present this data? We
started out w/ GeoJSON directly from the DB but the size of the files were
huge, the vector tiles are 30% the size of GeoJSON. We were in the MTS private
beta but they didn't have the 'update' process worked out yet so it was a full
refresh each time.

~~~
flippmoke
We will be releasing incremental updates to Tilesets API Beta (MTS) here
shortly, reach out to us again and we can talk about having you test it out!

------
jokoon
I wish I could learn to build my own tiles, vector or PNG. I don't really
understand where the data comes from, how is data gathered and assembled.

I'm also really curious about the choices involving the zoom level, how do you
decide to render things depending on the zoom level, when is data discarded,
to have good detail or better performance and lighter tiles. I would really be
willing to try build lighter maps so I can have my own mapping software on a
desktop machine.

The data sizes and hardware requirements involved are generally pretty big. It
could be interesting to see how much details one could achieve to make a
"portable" map browser when limiting the data size to 2GB, 5GB or 10GB.

I would really like to ask why, on some mapping software, you can't see names
of large enough cities/places/regions that are obviously there. It often makes
it difficult to browse maps properly.

~~~
bransonf
I’ll try my best to explain the process.

The data comes from places like the Census Bureau (roads, place names) and
then a lot of it has to be collected by the like of OpenStreetMap/Google/Other
Providers. (GIS Data is big business)

For Vector based approaches (See mapbox) these data are stored in special
built databases and usually simplified geometries are served to the browser.
The benefit is continuous zoom, but the pitfall is more server side
computation and hence cost.

Because of the cost/compute, raster tiles (PNG, jpg, any pixel format) have
been much more popular. These start the same, you collect all these data and
put them in a database. The difference is the added step of rendering tiles.
This one-off computation saves you work from then on. See maps.stamen.com for
an example of tiles made from OSM data.

And you’re right about place names sometimes not being apparent. This is a
trade off when using open data and auto generated tiles. With something like
MapBox’s vector tiles, you have individual decimal level control of things
like labels. And zoom level is another computational trade off. You start at 0
and define an arbitrary end. The higher the number, the computation/data
increases four fold each n. O(4^n)

And as far as why the size requirements are so big, geospatial data is big.
You have to record information on every point for vectors which depending on
quality can be a ton. And for rasters, we’re talking trillions of pixels
really. That’s why all of this is server side.

And lastly to your point about lightweight desktop software, tiles don’t
really have a place in the data process. They’re only really useful for the
visualization aspect. And frankly, I think we’re reaching the capacity of the
technique, we just might have some headroom in server efficiency.

~~~
snodnipper
+1

> And lastly to your point about lightweight desktop software, tiles don’t
> really have a place in the data process. They’re only really useful for the
> visualization aspect. And frankly, I think we’re reaching the capacity of
> the technique, we just might have some headroom in server efficiency.

Not totally sure what you mean on your last point...data can be feature
centric (e.g. stored by feature id) or area centric (stored by area location)
etc. Storing data by location is important far beyond visualisation and is
abstracted in databases such as PostGIS/Postgres (a branded data structure).
That said, I acknowledge that ArcGIS Pro, QGIS etc. have limited support for
tiled data but of course that is changing. Safe funded much (all?) of the OGR
MVT development afaik.

~~~
bransonf
Oh, I meant more about data analysis. Typically you don’t import raster tiles
unless we’re talking about imagery.

But as far as like roads and boundaries, you should always work with the raw
vectors.

~~~
kylebarron
This isn't _necessarily_ true, doing data analysis on vector tiles allow for
high parallelization. See TileReduce [0]

[0]: [https://github.com/mapbox/tile-reduce](https://github.com/mapbox/tile-
reduce)

~~~
bransonf
I meant specifically raster tiles, but this is indeed very cool. I absolutely
love what MapBox is doing.

------
crmrc114
I have a pal who does GIS work in the oil and gas industry- I think its crazy
how much influence ESRI has on that market. Would love to learn more about
interaction with map data like this.

For a non-gis person this was a fun read. So thanks for the post!

~~~
bransonf
I have the same sentiment on ESRI. It’s basically all I got taught in my
university courses, but it’s not what I ever want to use.

It’s crazy that a privately held company holds like a third of the market
share.

And personally, I don’t think their software is that good. I find their
documentation to be undesirable and their solutions to be strict.

Case in point, the geodatabase (gdb) standard is purposefully meant to
obfuscate the data within. No one has ever been able to explain to me why this
is, and the standard has been open sourced by now.

Not to mention, the number of times I’ve had ArcMap crash without any helpful
information as to why it crashed...

That said, ArcMap is the Excel of GIS. It captured market share (especially
government contracts) two or three decades ago and no one has disrupted the
desktop GIS platform. On the web front, however, I see companies like MapBox
far outpacing anything ESRI is capable of yet.

And to anyone looking to learn GIS: Post GIS, GDAL and any scripting language
will make you more powerful than most of the people I know within the field.

~~~
sleavey
> And to anyone looking to learn GIS: Post GIS, GDAL and any scripting
> language will make you more powerful than most of the people I know within
> the field.

Funny this article was posted today, because yesterday I was looking into
rendering a custom map for a ~100x100 km area from OpenStreetMap data for a
particular application. I've got basically no experience making maps but I've
dabbled with GDAL and Rasterio. I was thinking of using Mapnik with a dump of
(part of) the OpenStreetMap database into a local PostGIS instance. Ideally
the rendered tiles should be vector format. Do you think this approach seems
reasonable or am I missing a potentially simpler way?

~~~
bransonf
Are you trying to render a static map or make something interactive?

If this is static, Mapnik is a good call. It has some extra anti-aliasing
under the hood and it’s exceptionally fast.

~~~
sleavey
Static; thanks for the info. I would ideally like to dump a bunch of SVG tiles
for various zoom levels so I can store them in a static directory on my server
rather than serve them dynamically. I take it that Mapnik is capable of dumps
like this? And, I would like to use the Python bindings but they look
relatively badly documented. Would you suggest a newbie like me uses the C or
XML interfaces instead, if they are better documented?

~~~
bransonf
I’ve only used the Python library, and I think I survived. I only used it for
some visualization however.

And I’m still unclear. Are you trying to serve these tiles to another
application? Or are you trying to make a digital/print map?

And SVG is probably excessive for static tiles. It won’t have the size
reduction of a raster tile nor the benefit of a true vector solution.

~~~
sleavey
> And I’m still unclear. Are you trying to serve these tiles to another
> application? Or are you trying to make a digital/print map?

Serve them from local storage to a viewer application.

> And SVG is probably excessive for static tiles. It won’t have the size
> reduction of a raster tile nor the benefit of a true vector solution.

Ah, is there another vector format for tiles other than SVG? Or are you saying
that I should just generate a bunch of compressed rasters?

~~~
bransonf
Ah, I haven’t really got into that territory with mapnik. But to the second
point, yes you should just generate a bunch of Raster tiles. And before doing
this, ask yourself if you really need to.

If this isn’t a huge project, Mapbox is an easy packed solution. Otherwise,
there are dozens of really good tile providers already.

------
anonymousCmntr
I use this project for serving vector tiles with PostGIS and Django REST.

[https://github.com/corteva/djangorestframework-
mvt](https://github.com/corteva/djangorestframework-mvt)

