
New pathfinding algorithm - jsnell
https://factorio.com/blog/post/fff-317
======
payasr
There's a _lot_ of work on hierarchical route planning in games. I don't mean
to put down the developers or anything, and I'm so glad all of this works, but
there are plenty of ideas in the academic-world that I think developers should
know more about.

Botea, Adi, Martin Müller, and Jonathan Schaeffer. 2004. “Near Optimal
Hierarchical Path-Finding.” In Journal of Game Development.
[http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.112....](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.112.314).

Rahmani, Vahid, and Nuria Pelechano. 2017. “Improvements to Hierarchical
Pathfinding for Navigation Meshes.” In Proceedings of the Tenth International
Conference on Motion in Games, 8:1–8:6. MIG ’17. New York, NY, USA: ACM.

Toll, Wouter van, Roy Triesscheijn, Marcelo Kallmann, Ramon Oliva, Nuria
Pelechano, Julien Pettré, and Roland Geraerts. 2016. “A Comparative Study of
Navigation Meshes.” In Proceedings of the 9th International Conference on
Motion in Games, 91–100. ACM.

And route planning in graphs (with static edge weights) has been studied to
death.

[https://arxiv.org/pdf/1504.05140.pdf](https://arxiv.org/pdf/1504.05140.pdf)

~~~
payasr
I've been thinking about writing a blog series on route planning algorithms if
there is enough interest.. let me know if anyone wants it lol?

~~~
mehrdadn
Long shot, but if by any chance you could explain Highway Dimension [1] like
I'm 5, and how it leads to faster routing algorithms, I would definitely be
interested! I've watched some of Goldberg's videos (e.g. [2]), so I have some
vague intuition, but not nearly as much as I feel I could/should. (Though it's
entirely possible it's just something that can't be meaningfully ELI5'd in a
usable manner... I don't know.)

[1] [https://www.microsoft.com/en-us/research/wp-
content/uploads/...](https://www.microsoft.com/en-us/research/wp-
content/uploads/2010/01/soda10.pdf)

[2]
[https://www.youtube.com/watch?v=ArO1ZviOMhI](https://www.youtube.com/watch?v=ArO1ZviOMhI)

~~~
payasr
Sure! That is a classic paper, I didn't know there was a video too. Let me
try:

Highway Dimension- Say we are given a bidirectional graph G and a radius r.
Now take a vertex v, and find the set of all shortest paths from v of length >
r and <= 4r. Let us call this set P(v, r). Iterate for all vertices in the
graph and real radii, and obtain similar sets. Then, highway dimension is the
size of the smallest set (H) of vertices such that all sets collected in the
previous step have at least one vertex in H.

Why is this useful- Empirically, we know that road networks have small highway
dimensions. This is interesting because it implies that there are only a
handful of vertices from which you can take the shortest paths to all the
vertices in the network. Intuitively, it makes sense too. If you want to
travel long distances, it's best to take the highway as early as possible, and
travel along it for as long as you can, then descend to the local roads as you
reach closer to the destination.

Highway dimension basically gives us a theoretical way of capturing the
inherent 'hierarchy' of the road network edges and explains why Contraction
Hierarchies works so well on road networks (as opposed to random graphs, where
CH can be easily beaten). HTH!

~~~
mehrdadn
Oh wow! To confirm I understand this, is the following correct? _" The
radius-r highway dimension of a graph G is the(/a?) minimal set of vertices
that must be visited when traveling a distance ∈ (r, 4r]."_ I'm actually a
little unclear on r—I assumed HD is a function of r, but you said you iterate
over all real radii to obtain HD, which would mean HD is independent of r?

Yes this is helpful, thank you so much! :)

~~~
payasr
HD is independent of r. And it's not the minimal set of vertices that must be
visited.. it is the smallest set of which at least one must be visited.
Another good explanation of HD is in section 1.3 of the skeleton dimension
paper
([https://arxiv.org/pdf/1609.00512.pdf](https://arxiv.org/pdf/1609.00512.pdf)).

~~~
mehrdadn
Oh, sorry, I was vague about the visiting—I meant the _set_ must be visited,
i.e. something in it must be visited. I'll take a look at that one too,
thanks!

------
nsilvestri
I've always loved the Factorio devs for seeming to make their game not only
fun but also a well-designed piece of software. The gamedev industry,
particularly by indie devs, is notorious for doing whatever it takes to make
it work, code health be damned.

~~~
eindiran
I think a big part of that is that historically most games are updated
relatively little after they are released, so maintainability of the code base
is secondary to not letting the release date slip. But Factorio is a game that
has been getting updates for a while now post-release, so the devs have to be
happy revisiting the code, which increases the incentives for keeping the code
maintainable and the code health high. It seems like more games are moving
towards this model (lots of updates post-release), so I'd expect to see the
game industry start to adopt some of the practices used by the software
industry in general to keep code maintainable.

~~~
roenxi
That is a chicken-and-egg problem though. One major reason that most games
don't get updates is surely because the code is unmaintainable.

~~~
surye
I think rather it's a revenue issue, until relatively recently there was no
way to profit from updated games, so you're better off moving on to the
sequel, or next year's edition of the game. Also a lot of the video game
development culture likely stems from console releases, which again until
"recently" could not be updated (only new versions of disks/carts for future
customers) so it was not feasible to plan for future maintenance, beyond what
was necessary for derived codebases.

------
shoo
A small tweak to this approach would be to precompute and cache the entire
chunk-graph distance matrix, so the chunk-distance between any two chunks can
be computed by a lookup, instead of searching. Use a bit of memory and make
the time to generate the world a bit slower, but could be viable if the chunk-
graph pathfinding calls are still slowing things down a bit at runtime.

This might make sense if a bunch of assumptions hold: the chunk-graph does not
change during gameplay, map sizes aren't much bigger than the one shared in
the post so the number of vertices in the chunk graph is pretty small, land
units share the same costs for traversing terrain (i.e. same chunk graph with
same edge cost shared by all units).

Rough back of the envelope: number of vertices, V = 20 chunks high * 20 chunks
wide = 400. Chunk graph is sparse, each chunk connected to 8 neighbours by
undirected edges, so number edges, E = (1/2) * 8 * V .

Memory required to store chunk-graph distance matrix: V^2 * sizeof(distance) =
400^2 * sizeof(float) = 640,000 bytes. Could use +inf for disconnected chunks
due to islands.

Time to precompute full chunk-graph distance matrix by repeatedly dijkstra-ing
from each chunk to all chunks in the chunk graph: V * T(dijkstra(E, V)) = V *
O(E + V log V) = O(V E + V^2 log(V)) = O(V^2 + V^2 log(V)) = O(V^2 log(V)) ,
which won't scale due to the quadratic-ish V^2 log(V) term, but probably fine
if V <= 400 .

~~~
alecbenzer
> the chunk-graph does not change during gameplay

We mean the chunk _component_ graph, right?

That does technically change, because you can use
[https://wiki.factorio.com/Landfill](https://wiki.factorio.com/Landfill) to
fill up water and modify which positions a chunk component takes up (and also
potentially merge two chunk components), but I think that happens rarely
compared to pathfinding.

~~~
shoo
> chunk component graph

yes. thanks for helping to clarify.

> > the chunk-graph does not change during gameplay

> that does technically change, because you can use landfill to fill up water
> and modify which positions a chunk component takes up

Yep, that might be a dealbreaker.

If the terrain only changes from impassable -> passable, not the other way
around (e.g. flooding disconnecting bits), then the distances computed for the
old map should be valid lower bounds for the distances on the new modified (by
landfill) map, so it might be possible to use the old distances as an
admissible a* heuristic to guide search to recompute new distances on the new
modified map, but that's getting a bit fiddly.

This caching is likely the kind of thing that is probably not worth adding to
the codebase unless you get a significant performance boost, as it adds a
number of constraints from the required assumptions that impede you from
changing the game rules in future

~~~
username90
You can add walls which biters try to move around. I think they add an extra
cost since they don't move around them perfectly, and often they try to attack
the thinner parts of the walls.

------
social_quotient
Neat!

Unrelated: Here are some cool visualizations and simulations for pathfinding
we have used in the past for trade shows and mall wayfinding.

[https://qiao.github.io/PathFinding.js/visual/](https://qiao.github.io/PathFinding.js/visual/)

[https://clementmihailescu.github.io/Pathfinding-
Visualizer/](https://clementmihailescu.github.io/Pathfinding-Visualizer/)

[https://www.growingwiththeweb.com/projects/pathfinding-
visua...](https://www.growingwiththeweb.com/projects/pathfinding-visualiser/)

[https://www.reddit.com/r/dataisbeautiful/comments/78of3s/pat...](https://www.reddit.com/r/dataisbeautiful/comments/78of3s/pathfinding_algorithms_visualized_oc/)

------
veselin
I am surprised that the text mentions the word contraction and yet doesn't
really do
[https://en.m.wikipedia.org/wiki/Contraction_hierarchies](https://en.m.wikipedia.org/wiki/Contraction_hierarchies)

Using something like this will on average get pathfinding for something like
O(LogN) time. May sound strange or impossible at first - this is less than the
number of squares the path goes through, but works and does not need huge
space overhead. This is because the CH algorithm selectively adds shortcuts to
the graph essentially making it completely skip many squares. Also this is not
a heuristic, but a provably correct algorithm (no provable time complexity for
arbitrary graph, but I think logN or maybe log^2(N) can be established for the
simple graph in this pathfinding).

------
donio
[https://anuke.itch.io/mindustry](https://anuke.itch.io/mindustry) is an open
source game with some similarities to Factorio

Here is the pathfinder code:

[https://github.com/Anuken/Mindustry/blob/master/core/src/io/...](https://github.com/Anuken/Mindustry/blob/master/core/src/io/anuke/mindustry/ai/Pathfinder.java)

~~~
kzrdude
Thanks for the link.

Do you know any similar games that _don 't_ focus on tower defense and
military (like factorio does)? I love the resource flows, transport problems
and factory building.

~~~
misnome
I'd really hesitate to say that factorio focuses on tower defense/military.
There is defense, but unless you are ramping up the difficulty the bugs aren't
really that much of a problem - more of a low level pressure unless you expand
deep into their areas - and, you have the option to turn them off completely
when starting a new game, and you wouldn't be losing a major part of the
appeal (like minecraft with creepers off).

~~~
kzrdude
I have played factorio a lot, and at this point I'm looking for a different
game, or something where the design does not include bugs in the pointless way
factorio does. It's an otherwise fantastic game. Maybe it's not even wrong to
have adversaries, but the way it's done in factorio is so simplified (infinite
bugs from outer space) it seems like satire.

------
rl3
Note that this post is #317. The Factorio team has the best development blog
on the planet in my opinion.

I'm just awestruck every time I stop and think about the fact they've written
over three hundred of these in-depth articles, complete with annotated
screenshots/video in most cases. The quality, depth and sheer consistency of
the write-ups given their team size is simply incredible.

Can't really say I know of anything that comes close, but I'd love to hear
about anything that does.

~~~
mschuetz
Closely followed by the Dolphin blog.

------
Animats
Pathfinding isn't hard if you have full information and enough memory to store
it. Most full info grid pathfinding algorithms, like A-star, are basically
variations on flood fill.

I had to develop a new pathfinding algorithm recently.[1] For fun, I've been
working on adding NPCs to Second Life. These have to be coded under these
constraints:

\- Each program is limited to 64KB, for stack, heap, and code. But you can
have lots of programs that intercommunicate via messages.

\- Finding out about the world is done by ray casting. You get about 100 ray
casts per second.

It was interesting to code a maze solver under those constraints. What I wrote
is basically "head for goal; if obstructed, wall follow." I do wall following
in both directions simultaneously, so the shortest path wins. This finds paths
around simple obstacles with minimal time wasted examining cells not of
interest. Then, after finding a path, I tighten it up, removing inside corners
where possible. This is not absolutely optimal, like A* is, but it's usually
pretty good, and requires far fewer ray casts.

[1] [https://github.com/John-
Nagle/lslutils/blob/master/npc/mazes...](https://github.com/John-
Nagle/lslutils/blob/master/npc/mazesolver.lsl)

~~~
Havoc
>\- Finding out about the world is done by ray casting. You get about 100 ray
casts per second.

That's how secondlife does things? That sounds computationally expensive

~~~
Animats
It's about like real world robotics from 15 years ago, where you had a LIDAR
or two with a modest data rate.

------
krisrm
The visualizations in this article are really well done. I've read a bit about
hierarchical pathfinding before, but this is the first time I feel like I
actually "get" it.

~~~
willis936
The factorio devs have weekly writeups of this caliber. They’re great reads if
you like the idea of making games.

------
akersten
Great optimization. I wonder if the second video is also real-time? The first
video is captioned that it hasn't been slowed down, but the 2nd video, I'm not
sure. If it is real time, it still seems like it takes a few seconds to
generate the path.

I wonder if it would be sufficient to generate the chunk-level path, get the
entities moving, and then do local A* to navigate through the individual
chunks as-needed. Right now it looks like the algorithm generates the complete
A* all at once (just after the chunk-level pathing where most of the bad paths
are thrown out).

~~~
sand500
>the pathfinder was slowed down significantly to make this video, to show how
it works. At normal speed, the entire search takes only a few ticks.

few ticks basically means instantly.

~~~
dragontamer
The 2nd picture:

> This video shows how fast the algorithm works in reality; it hasn’t been
> slowed down.

~~~
sand500
My parent mentions

>The first video is captioned that it hasn't been slowed down, but the 2nd
video, I'm not sure.

I assume they meant [https://cdn.factorio.com/assets/img/blog/fff-317-long-pf-
bef...](https://cdn.factorio.com/assets/img/blog/fff-317-long-pf-before.mp4)

so the one they were asking about is
[https://cdn.factorio.com/assets/img/blog/fff-317-long-pf-
aft...](https://cdn.factorio.com/assets/img/blog/fff-317-long-pf-after.mp4)

------
brunoTbear
The best part is the minifactory at the bottom of the page. Completely nuts to
launch a rocket in that small of a space in that little time. Everyone should
scroll down and check it out if you haven't.

~~~
willis936
Most of my enjoyment of factorio has been watching the factories of other
people. I have been playing industrial games for ten years and I still stumble
through the game that’s primarily based around discovery. Seeing what
thousands of hours can accomplish is inspirational.

~~~
NeedMoreTea
DaveMcW regularly does amazing and unbelievable things in Factorio, like the
in-game video screen[1]. Whereas I am happy to muddle half a dozen combinators
together with my lazy head on - to control a level display, delivery buffer,
or turn off specific features etc.

I just hope he doesn't decide to turn his hand to self-aware general AI. :)

[1]
[https://www.youtube.com/watch?v=mgfwwqwxdxY](https://www.youtube.com/watch?v=mgfwwqwxdxY)

~~~
ShamelessC
How in the... That's amazing. How is the video itself even "stored"? Does this
require a mod?

~~~
BlueTemplar
Just like memory cells are made IRL electronics using latches, you can make
memory cells in Factorio using combinators :

[https://wiki.factorio.com/Tutorial:Combinator_tutorial#Memor...](https://wiki.factorio.com/Tutorial:Combinator_tutorial#Memory_cells)

------
dgant
Neat!

My StarCraft-playing bot uses a similar approach for fast pathfinding. The
similarity is that we're both trying to use a better heuristic than straight-
line distance. In the case of my bot, it's done by breaking the map into
roughly-convex regions, then precalculating the distance from the center of
those regions to every other tile by flood-filling. That lets units outside
the region use (distance to boundary of goal region + straight-line distance
from boundary to goal tile) as a pretty good heuristic.

------
bduerst
Huh. I've been playing Factorio this week on a rail-world and cheesing the
biter bases with artillery, the way they're describing. Some of the bugs
manage to make it around the lakes from pollution spread, but cleaning them
out with artillery makes them wander and disappear off screen.

------
rowanG077
Since I don't think a true optimal route is necessary here I would have
checked out RRT instead of a*. There are variants of it that are much much
faster that can find almost optimal routes.

If you guys want to explore this further you could look into integrating
ompl[1]. It has a variety of planners that could be used. I believe you could
get an order of magnitude improvement using something like that.

[1] [https://github.com/ompl/ompl](https://github.com/ompl/ompl)

------
strbean
Was hoping for flow field pathfinding:
[https://www.youtube.com/watch?v=bovlsENv1g4](https://www.youtube.com/watch?v=bovlsENv1g4)

[https://leifnode.com/2013/12/flow-field-
pathfinding/](https://leifnode.com/2013/12/flow-field-pathfinding/)

------
Havoc
I like the authors writing style. Makes it easy to follow his thought process

------
afettere
Not exactly new - [https://www.computer.org/csdl/proceedings-
article/ictai/1997...](https://www.computer.org/csdl/proceedings-
article/ictai/1997/82030084/12OmNAolH2Q)

------
BlueTemplar
rare pic of biter phalanx (circa 32 BC, colorized) :

[https://www.reddit.com/r/factorio/comments/8q5q3c/rare_pic_o...](https://www.reddit.com/r/factorio/comments/8q5q3c/rare_pic_of_biter_phalanx_circa_32_bc_colorized/)

------
ronilan
It’s A*. I wrote one encapsulated as an ES6 JavaScript Class a while back.
Just dropping it here if anyone finds it useful.
[https://github.com/ronilan/a-mazing-
thing](https://github.com/ronilan/a-mazing-thing) Use as you wish.

~~~
dragontamer
The first picture and second pictures are A* , which is then demonstrated to
be insufficient for the game.

The 3rd+ pictures on are hierarchical pathfinding, which seem to be much, much
faster than textbook A* .

~~~
ronilan
As far as I understand they go back, goal to start with A* and use heuristic
scoring that is smarter than the simple Taxicab geometry. That is at least
what I understood from what the text explains. I might be wrong though.

~~~
username90
The entire blog is about how to create said heuristic, you can't just brush it
off.

~~~
ronilan
Who brushed?

