
Show HN: Wave function collapse algorithm - ExUtumno
https://github.com/mxgmn/WaveFunctionCollapse
======
fitzwatermellow
Great work! The fact that it captures "long-range order" seemingly perfectly
is something not many have been able to do before! And the "collapse"
visualization is great fun to watch.

But is your algorithm really qualitatively all that different from previous
search methods (e.g. Efros and Leung), if you are still (uniform random?)
sampling over the input distribution of patches?

I notice also your input textures tend to have sharp boundaries (as is common
in pixel art). It would be interesting to see the results when the input has a
lot of "noise", such as high def images of rocks or clouds ;)

While I still prefer search methods because they are easy to implement (and
give better results), "deep" methods are definitely gaining some ground:

Deep Textures

[http://bethgelab.org/deeptextures/](http://bethgelab.org/deeptextures/)

Combining Markov Random Fields and Convolutional Neural Networks for Image
Synthesis

[https://arxiv.org/abs/1601.04589](https://arxiv.org/abs/1601.04589)

~~~
ExUtumno
Thanks!

Efros' and Leung's method doesn't satisfy the (C1) condition. The closest
previous work is Paul Merrel's model synthesis.

WFC and texture synthesis serve similar purposes: they produce images similar
to the input image. However, the definition of what is "similar" is different
in each case. If you have a high def input with noise (like realistic rocks
and clouds) then you really want to to use texture synthesis methods. If you
have an indexed image with few colors and you want to capture... something
like the inner rules of that image and long range correlations (if you have an
output of a cellular automata, for example, or a dungeon), then you want to
use WFC-like methods.

Btw, I have classic texture synthesis algos in a different repo:
[https://github.com/mxgmn/SynTex](https://github.com/mxgmn/SynTex)

~~~
derefr
> something like the inner rules of that image and long range correlations

I assume that if you feed WFC a large input image, it just thinks of that as a
very complex set of rules that are harder to satisfy than those of a small
input?

Is there a way, then, to instead train the WFC algorithm on a large corpus of
small, similar samples, such that it can try to derive the rules _common_ to
all the inputs in the corpus, and produce one image that "really" fits the
rules, rather than just the ephemeral quirks from an individual sample?

Would there be, for example, a way to train WFC to produce outputs matching
the level-design "aesthetic" of a given game, rather than just "continuing" a
particular level?

~~~
ExUtumno
About harder and easier to satisfy, the question of how the rate at which the
algorithm runs into contradictions depends on the input is not easy at all.
There is no simple correlations between the contradiction rate and the size of
the input.

But the first thing you'll notice if you feed it an image with a lot of
patterns, is that it will work very slowly.

Yeah, the corpus thing can be done if we cut out rare patterns and leave only
frequent ones. I haven't tried it though.

~~~
mathgenius
I wonder if it would be interesting to purposely search for tilesets that
maximize contradiction rate. What would those things look like?

~~~
ExUtumno
A very good question! The opposite of it is also important, can we follow some
heuristics while creating tilesets to minimize contradiction rates, but not
making tilesets easy? I don't know. If someone knows please tell me.

~~~
mathgenius
Hmm, this possibly relates to sheaf theory. Robert Ghrist has a good book
about this stuff (applied topology) if you want to check it out.

------
etatoby
This is great!

It would be interesting to apply this concept to wavelets (instead of pixels
or voxels) in order to work on real-life pictures.

Also, 3 dimensions as in X, Y and time, to work on animated GIFs. Think about
an infinite, never repeating Blue Ball Machine!
[http://m.imgur.com/5Flh68G](http://m.imgur.com/5Flh68G)

~~~
chronolitus
This is crazy, and I think it hints at the possibility of universe creation:

you start from a finite pattern, which becomes the 'rules' of your created
universe. Then, using this wave function collapse algorithm you expand it into
an infinity where the possibilities are endless within the constraints of
those generator rules

~~~
borgel
If you enjoy pondering on that, you may enjoy reading Permutation City [1] by
Greg Egan.

[1]
[http://www.goodreads.com/book/show/156784.Permutation_City](http://www.goodreads.com/book/show/156784.Permutation_City)

~~~
brianpgordon
One of my absolute favorite sci-fi novels, and it changed my thoughts on
philosophy of mind more than any other book.

------
Balgair
Fantastic stuff. I love it! As with most meachine learning stuff and these
very cool ideas (of which I am very hazy on, esp their categorizations) I
immediately want to use it for our lab's brain work.

We have a LOT of 3-D images (per voxel is ~200nmx200nmx~600nm for a lot of
1028x1028 .tiff images all stitched together) and would love to feed these
images BACKWARDS into this. IE we have the 'far field', and we want the
'elements' that make it up. Say we have a large amount of data on the
Pituitary gland, and we want to compare the 'elements' of the Pituitary to the
'elements' of the hippocampus. We know a lot of these differences, but there
may be 'something else' in there that we humans are not seeing. This may be of
great use to use for disease pathology like Lewy Body Disease precursors.

~~~
4bpp
Sounds like an altogether easier problem, assuming you don't care to
compensate for noise in your data somehow - supposing you want to identify all
distinct m×n×k-sized "elements", simply use some appropriate rolling hash[1]
(i.e. a hash of a window that you can update in constant time as you slide the
window) as a key mapping to a list of "elements" you have seen so far with
that hash, and only do pointwise comparison (to see if you have seen that
exact pattern before) if the hashes match. Assuming you don't have too many
distinct elements, this should give you performance close to linear in the
size of your data.

[1]
[https://en.wikipedia.org/wiki/Rolling_hash](https://en.wikipedia.org/wiki/Rolling_hash)

~~~
Balgair
Whew, now that was a wikipedia binge. Thanks for the leaping off point!
Unfortunately, our data is VERY noisy. We can do some techniques to smooth the
data, but the unfortunate part is that the things that matter in biology are
under the diffraction limit of light. Inherently, what we want out of the data
will then be noisy. Gestalt structures are less noisy and these techniques can
help with that (think using a fiber-optic read-out as you go into the brain
for surgery so you know what region you are in), at least I think. Also, the
'elements' of our data set are unknown, but likely very large. Maybe in the
100s to 1000s, not 5 or 10. It turns out the brain is complicated, who knew?!

Still, thanks a ton for this info! I think it can really help with some
computational bio stuff another lab is working on here (viral similarity in
genes in your DNA)

------
phodo
Nicely done! Your 3d extensions could be further extended (performance
notwithstanding) to create some pretty cool procedurally generated VR
experiences. Thanks for sharing.

------
throwaway000002
Brilliant!

Don't really understand the technique, but would like your thoughts on if it's
possible to give a Penrose tile set as a seed and see if aperiodic order is
generated.

Lovin' it!

~~~
ExUtumno
Thanks!

I'm not sure, but I think that Penrose tilesets are what I call "easy": you
can't run into a situation where you can't place a new tile. It would be great
if someone here could confirm or deny this.

So if this is the case, then Penrose tilesets are not interesting to WFC,
because you can produce arbitrary tilings with much simpler algorithms.

Right now though WFC is only working with square tiles, but it's not hard to
generalize it to arbitrary shapes. Paul F. Harrison made a tiling program that
supports hex tiles: [http://logarithmic.net/pfh/ghost-
diagrams](http://logarithmic.net/pfh/ghost-diagrams) See also the relevant
paragraph in the readme (just search the word "easy").

~~~
DennisP
Maybe it could be interesting to place Penrose tiles with the simple
algorithm, but color them with your algorithm just like you're currently
coloring squares.

~~~
ExUtumno
So basically make a not-easy tileset with the shapes of Penrose tiles. Yes,
this could be interesting.

------
abecedarius
As I understand it: this treats image generation as constraint satisfaction.
The constraints are that each NxN patch appears in the source image. The
satisfaction method is arc consistency
[https://en.wikipedia.org/wiki/AC-3_algorithm](https://en.wikipedia.org/wiki/AC-3_algorithm),
except, when that settles down prematurely, pick the least-constrained patch
and make a random valid choice, then continue. (If this leads to getting
stuck, then give up instead of backtracking.)

Is that the idea? The description wasn't completely clear to me.

~~~
ExUtumno
Yes, (C1) is a constraint problem. But we also want to satisfy (C2) as close
as possible, otherwise we could have just colored some outputs in a single
color.

~~~
abecedarius
Yes, I took that to mean, when you're making a random valid choice, the
weights come from the input distribution.

I don't understand about "a single color" unless you mean setting all output
pixels to the same color, which would only satisfy the constraints if the
input has an NxN patch all one color.

I hope my comment didn't seem to imply that the work seemed unoriginal. I
don't think that; I wanted to check my reading.

~~~
ExUtumno
Most of the examples in the repo have those NxN all one color patches. Or,
without (C2) the algorithm would have generated completely empty integrated
circuits, or completely grass terrain, which is really boring.

You understood right, it's constraints + probabilities.

Btw, I have different algorithm that satisfies (C2) perfectly, but not (C1):
[https://github.com/mxgmn/ConvChain](https://github.com/mxgmn/ConvChain)

~~~
abecedarius
Thanks!

I might try this approach to generate formal poetry -- it's something I've
done by backtracking before, and I'd considered doing something like your
ConvChain.

~~~
ExUtumno
At first I thought that my methods don't offer anything new to text generation
besides the Markov chain, but several people already proposed ideas that sound
sensible, so let me know if you make anything!

------
twfarland
Mindblowing... could this technique also be applied to music?

~~~
ExUtumno
I doubt it, because music is 1-dimensional and for 1-dimensional arrays WFC is
just a Markov chain.

~~~
barrkel
There are three-dimensional views of music, e.g. time-frequency-amplitude. See
[https://en.wikipedia.org/wiki/Spectrogram](https://en.wikipedia.org/wiki/Spectrogram)

~~~
ssalazar
Spectrogram is 2D (plot of amplitude given time and frequency).

Its interesting to think about it for a spectrogram because "similarity" is
different in each dimension (freq vs. time). Frequency is also perceived
logarithmically, so you would probably want to convert to e.g. Mel scale
before applying this algorithm (a 2000-2100Hz change is much subtler than a
200-300Hz change).

~~~
jmilloy
Isn't that 3 dimensions (amplitude, time, and frequency)? The plot of course
fills 2 spatial dimensions and uses color to represent the 3rd dimension. But
I don't know very much about this.

~~~
ssalazar
I don't have a mathematically rigorous understanding of it but the number of
dimensions is basically the number of freely varying inputs to the
corresponding functional representation. In a 2d image, x position and y
position are mapped to a color, e.g. I(x,y) = C. In a spectrogram, freq and
time are mapped to a color (amplitude) e.g. S(f,t) = A. In neither case can
you just pick an arbitrary color or amplitude and in general produce a
singular x/y or f/t from that.

------
xixixao
Amazing. Make sure to scroll through the whole README to see voxel models.
Rendered using
[http://ephtracy.github.io/index.html](http://ephtracy.github.io/index.html)
which is in itself pretty cool too.

------
matthewwiese
Excellent! I especially like the example third from the top (black/white
"rooms"). This would be a great tool to generate maps for roguelikes in a more
"organic" fashion. Not only that, but the bitmap source you use for input
would provide wildly different results.

Given that and the simplicity of just providing a bitmap as input, this could
be adapted well to provide customization to the player as well.

------
bduerst
I don't have much to contribute to the conversation on this other than that
it's incredibly cool.

~~~
sushisource
My thoughts exactly. I will definitely read through this to understand it by
my first impression was "This is so fucking neat!"

------
ianai
I'm missing a lot here. How does this equate to wave function collapse?

~~~
vinchuco
>with the help of ideas from quantum mechanics. >so it doesn't do the actual
quantum mechanics, but it was inspired by QM.

~~~
netsec_burn
Yea, my only disagreement here is the name.

~~~
ianai
That'd explain why when I couldn't find an explanation of what wave function
collapse meant.

------
phaedrus
This requires no more up front work than a Markov-based technique, but
produces results on par with an L-grammar. It may be generally applicable to
the problem domains of both!

------
amelius
This reminds me of the quite powerful PatchMatch algorithm [1]. Is it related?

Also note how PatchMatch can work with constraints.

[1]
[http://vis.berkeley.edu/courses/cs294-69-fa11/wiki/images/1/...](http://vis.berkeley.edu/courses/cs294-69-fa11/wiki/images/1/18/05-PatchMatch.pdf)

~~~
ExUtumno
PatchMatch is an algorithm to quickly... match similar patches in an image, it
is used in a lot of texture synthesis algos. See my answer to fitzwatermellow
for the difference between texture synthesis and WFC. So yes, it's related.

Photoshop's implementation of PatchMatch handles constraints perfectly, yes.

------
valine
You should add some gifs with a lower frame rate. I had to download the one
you had and walk through frame by frame. Absolutely beautiful.

~~~
TheRealPomax
lower framerate, or lower playback speed?

~~~
vinchuco
Probably meant playback speed. Probably you knew this.

------
wodencafe
That's incredible!

This could be great for generating maps in a sprite based strategy game!

~~~
khalilravanna
As a person currently rewriting the mapgen algorithm for their Dwarf-Fortress-
like game, I'm very excited to experiment with this this weekend =D

~~~
Sire
Cool, got anything to show? :)

~~~
khalilravanna
As far as the game? Sure I got a blog over here =>
[http://ripplega.me/development/month-development-
feb-08/](http://ripplega.me/development/month-development-feb-08/) If you
meant the map specifically, at a high level I haven't changed the algorithm
(yet) so you can check out an overview of my current one here:
[http://ripplega.me/development/month-development-
may-10/](http://ripplega.me/development/month-development-may-10/)

The blog itself hasn't been updated it in a while as I've been in the midst of
some pretty serious rewrites/refactors over the past several months. Since
that post I've rewritten the game in TypeScript, rewritten the core engine to
use the ECS (Entity Component System) pattern, and right now am actually in
the midst of a rewrite of pretty much the whole game. Hence I mentioned the
rewrite of the mapgen code, the first version of which was probably written 2
years ago hehe. I had a huge amount of features built up over 2 years
(idealogically similar to DF in that regard heh) but realized I needed to pare
down the feature set to get an alpha out so people could play. It's also
allowing me to address some fundamental flaws that the engine built up over
the past year or two. So that's the point where I'm at now.

Feel free to sub to the mailing list if you'd like to know about any updates
and when the game comes to fruition you'll be notified ;)

------
Sharlin
This could be extra awesome if it could be made small and fast enough to be
able to execute as a fragment shader.

~~~
ExUtumno
Well, right now it is not fast at all. :) But I plan to think about the
problem of generating pixel shaders form examples in the future.

------
tgb
This is great. Someone should make a Minecraft plugin.

~~~
mc42
The way I see it, this would be the perfect way to produce semi-altered
terrain sprites automatically. Right now,the textures are pseudo-randomly
rotated for things like dirt blocks, grass tops, and stone. [1] However, with
this you could create a multitude of different textures based off a single
source.

Plus, dungeons could be a lot more interesting. It'd be interesting to see
what could be done with it...

[1] - [http://i.imgur.com/UJeLy.png](http://i.imgur.com/UJeLy.png)

------
michrassena
I don't have much to contribute other than to say this is really amazing, and
I want to throw all kinds of things at it and see what happens. Pardon my
ignorance of how this works, does this algorithm have anything to do with
symmetry breaking?

~~~
ExUtumno
Thanks!

No, not really. ConvChain though is related to symmetry breaking, the same way
as MCMC simulation of the Ising model is
[https://github.com/mxgmn/ConvChain](https://github.com/mxgmn/ConvChain)

~~~
michrassena
ok thanks. The ConvChain project is also very interesting.

------
vinchuco
Can you feed it something other than bitmaps? Like its own source code?

~~~
ExUtumno
Source code is a 1-dimensional array. For 1-dimensional arrays WFC is just a
Markov chain. 2 and higher dimensional arrays are much more interesting
because they have cycles, and there is no _canonical_ way to generalize Markov
chains to higher dimensions.

~~~
toxik
This will be abstract, but you seem to know your abstract algebra -- is it
possible to do this kind of thing with graphs? It should be, right? And we all
know code can be constructed with graphs, so… voila, you can generate code,
no?

~~~
ExUtumno
What do you mean by "code can be constructed with graphs"?

~~~
krylon
I suppose the GP refers to compilers/interpreters parsing source code into
abstract syntax trees (ASTs).

~~~
palunon
Or maybe to things like PureData, Unreal Engine blueprints, etc

------
Myrmornis
The README says it uses real-valued probability amplitudes, not complex-valued
wavefunction. Could the simulation be done with complex numbers and if so how
would the results differ?

~~~
ExUtumno
We need to interpret those coefficients somehow. Real coefficients can be
interpreted as mixing of colors, but for complex ones I don't see a good
interpretation.

------
hood_syntax
Very cool, might try do something like this in a game I've been thinking about
writing.

------
mirekrusin
Can this approach be used to generate "missing parts" in unfinished song? If
composer has few good parts but is too lazy to finish the whole piece, for
example? Or to "extend" Moonlight for example?

~~~
ExUtumno
There are special approaches to generating music. The best for ratio of
quality/complexity that I know of are Markov constraints
[https://www.youtube.com/watch?v=buXqNqBFd6E](https://www.youtube.com/watch?v=buXqNqBFd6E)
and WaveNet. I don't think WFC offers something useful and new for generation
of music.

------
antirez
Very cool algorithm, and impressive visual results. Thanks for sharing.

------
alexmorenodev
Extremely nice to create procedural maps and other stuff. Really nice.

------
ejcx
I love that one of the files is "Stuff.cs".

Super neat visuals and I can't wait to play with this, as someone who knows
nothing about this kind of "stuff".

------
smilekzs
The circuit board example looks alarmingly convincing despite it's obvious
we're not looking at an actual circuit...

------
artursapek
Amazing. If this is fast enough, I imagine it could do wonders for games
(terrain and background generation).

------
WhitneyLand
Very impressive. I have no use for your work whatsoever yet would love to dive
into it further.

------
partycoder
WOW, this is amazing. Really impressed seeing how it can adapt to multiple
patterns.

------
psyc
This is amazing, and could have very cool applications in procedural
generation.

------
chrischen
Cool. I wonder what the results are on a high resolution image.

~~~
vinchuco
Google maps could maybe be interesting. Create alternate Earth maps to use...
somehow... logistics? climate?

------
xer0x
I don't quite get it, but damn it's awesome!

------
CiPHPerCoder
This is pretty awesome, but there is no LICENSE file so I'm assuming no one is
allowed to use it in their own projects.

~~~
zitterbewegung
The source files say its distributed under the MIT license.

------
isoprophlex
Thanks for posting this! Really exciting work!

------
natch
What language is this?

How can I compile / run it?

~~~
nhatbui
It's C#. I'm not familiar with it which is why I was surprised that after
downloading Visual Studio Tools, I opened Developer Command Prompt, I did a
`csc *.cs` and I was left with one nice executable, Main.exe.

------
angelftbcn
Wow!

